When I was a computer science undergrad in university, I received a coffee mug for Christmas. You may have seen this mug, it has a number of tongue-in-cheek expressions and quotes about software development and software programmers on it. Here's a random sampling:
You get the idea . . . . My mug is copyright 1980 by Art 101 Ltd., and I find it interesting that most of the sayings on this mug have stood the test of time, apart from one that mentions "job control cards" and an antiquated list of programming languages such as JCL, Algol, and PL/1. However, one saying on this mug still stands out to me today:
I freely confess that this particular saying really gets my ire up. Perhaps I'm overly sensitized. I've had people quote it to me on a number of occasions; in particular, a former boss who felt that software developers lacked professionalism (he of course was a professional engineer and had the ring to prove it). Certainly what Weinberg says has a kernel of truth to itthat software developers can be a sloppy lot. But, by stating it as a law crafted as a negative statement, it implies that programmers can't write good software and don't pay attention to architecture, and that builders do. This is a slight to professional programmers, on the one hand, while being overly generous to builders on the other.
I am a big believer in the power of metaphor, especially in the context of technology. Sometimes, when faced with a highly technical term or feature to develop, it is incredibly valuable to use a metaphor that ties the term or feature into the real world through a simple comparison that everyone can understand. This is used in Extreme Programming, not to mention in many other technical fields such as usability analysis.
My real objection to Weinberg's Law is that it implies that a valid metaphor for software development is architectural engineering (i.e., building a building). It is astonishing how many people, from users to managers to software developers, believe this is a valid metaphor for software development when asked. Certainly, it is tempting to think that software is like building a building, and frankly what software developer, team leader, or manager wouldn't want to use the inspiration of Frank Lloyd Wright or the Sears tower in Chicago as a metaphor for his or her work? Well-architected buildings are certainly interesting to look at, and exceptional buildings help define cities.
Figure I-1. These buildings show different architectural styles and through their style and aesthetics also demonstrate the temptation to use buildings as a metaphor for software development. Left image ©Kathy McCallum, right image ©Steve Lovegrove. Images from BigStockPhoto.com.
Why isn't building a building a valid metaphor for software development? The short answer to this question is that buildings are largely static and software is not. Buildings are very expensive to modify after they have been built, and indeed the entire process used to design and construct the building is intended to maximize the chances that the building will meet the customer's requirements and will, within some margin of error, stay within the desired budget. It's easy to identify when a building is complete, but successful software never is. Software is cheap to modify, and any successful piece of software is virtually guaranteed to change over its lifetime. Actually, the software industry has trained its users into believing that major modifications are possible, if not inevitable and expected.
Good software developers realize that their programs can be modified to add just about any unanticipated featureof course, there are limits. Changing a web browser into a database or a scientific visualization application into a word processor might involve huge rewrites, but the key thing is that good software programs with good developers can do just about anything they want. "It's only software" is a common mantra.
Buildings, on the other hand, are another matter. Ask an architect to make an unanticipated change and chances are she'll reply that the only recourse is to tear the building down and start over. Ask an architect to design a building that will handle all the likely configurations and he'll refuse the project. This is because there are only a limited number of possible changes the architect and customers can make to the building over time. In software, unanticipated features are highly likely because software is developed in a highly complex, hard to understand, technical, and constantly evolving environment. After five years of use and modification, the source code for a successful software program is often completely unrecognizable from its original form, while a successful building after five years is virtually untouched.
These differences alone should be enough to convince you that software is not like "building a building" at all. And yet, the methodology taught in school to software developers, the methodology used in most software companies, and the methodology described in the bulk of today's software literature is exactly the same basic methodology, that is, used in building construction and other forms of engineering; the so-called "waterfall" method, which advocates thoroughly understanding user requirements before designing the software, followed by coding, and then testing.
I don't mean to imply that the waterfall method is plain wrong. Great works of engineering such as buildings, airplanes, bridges, and ships have been built with this method. And many of the practices introduced in the waterfall method are good. The problem as I see it is that too few people question the methods they are taught in school and the workplace. This is a human trait more than anything else and not a reflection on software developers in particular. It's just that probably the majority of projects would be better off using a more appropriate metaphor for software development.
If building a building is not a valid metaphor for software development, then what is? My favorite metaphor for software development is a coral reef. I think good software like Adobe's Photoshop, Oracle, SAP, Linux, and Microsoft Windows creates an entire ecosystem of customers, plug-in developers, distributors, VARs (Value Added Resellers), service and consulting firms, hardware manufacturers, and even competitors. This software ecosystem is like a coral reef.
A successful software product plays the role of the underlying coral in the reef. The coral is what makes the entire ecosystem possible. None of the other organisms can survive without it. And just like software, a coral reef is continually evolving, growing, and changing (though, of course, at a different pace!). It's not just the reef itself that is changing, it's the entire ecosystem with all its dependent fish, invertebrates (sponges, fans, etc.) that change and evolve together. The reef ecosystem is incredibly complex, yet also fragile and vulnerable, just as a software product's ecosystem is prone to disruptive technologies or competition. This level of complexity, growth, constant change, evolution, and vulnerability I believe accurately portrays the complex world that today's software products inhabit.
If you are a scuba diver, you should be able to appreciate this metaphor. Coral ecosystems are truly a wonder. You can view them from a distance or up close, noticing schools of fish, large fans, sponges, and hugely interesting and varied coral formations. And if you are fortunate to have visited the same reef over the course of a few years, you will also know how much a reef changes from year to year and season to season, despite the common misconception that coral is more like rock than a living organism. And the complexity of the ecosystem is obvious because it is impossible to understand the entire ecosystem at almost any scale, both when you can see the entire reef, and when you are watching only a small portion of it.
Figure I-2. Nature provides many useful metaphors for software and software architecture that accurately reflect the nature of software development. Successful software is like a coral reef because it inhabits a complex and continually evolving ecosystem of both complementary and competing technologies and software. Image ©Jason Sah. Image from BigStockPhoto.com.
Why does the metaphor we use to think about software development matter? Metaphors help us analyze a problem from a unique and common perspective. Good metaphors help groups of people grasp a complex concept or problem easily and clearly to understand what they are trying to build. In the software industry, we have been using the wrong metaphor, largely unconsciously, and have adopted practices from engineering that reinforce this metaphor.
Too many development teams develop software products as if they were building a building. They place too much emphasis on features of the building and trying to completely understand its requirements before it is built. The temptation is to finish the project or product and then to stand back and admire the creation. Inevitably, however, unanticipated changes come along in the form of user requests, bug fixes, and changes to other software or hardware that the program is dependent upon. The result is that over time, doors get added in odd places, walls get thinner and are moved around, cables get run through elevator shafts, and holes get added between floors or rooms. A mess accumulates that becomes increasingly hard to manage, and the greater the mess, the easier the temptation to add to it. Add in competitive pressures and a constantly changing ecosystem of underlying technology (operating systems, web browsers, databases, UI toolkits, etc.) and chaos ensues.
These projects are on the path to unsustainability. At some point the whole edifice will come tumbling down and become unmanageable. One of the main features will be an increasingly high cost of change where every change, no matter how minor, introduces a risk that something completely unrelated will break. The programmers will blame management, and management will blame everyone else. If the company can manage it, the application will be rewritten, only to start the clock ticking on the unsustainability cycle again. These projects aren't any fun, yet they are distressingly common in the software industry. The software industry is not slow-paced after all; laggards do not have high odds of survival.
This book, as you've gathered from the title, is about sustainable software development. The coral reef metaphor is an important one because the software industry needs to recognize that its principles and practices must support free-flowing, adaptive, and opportunistic evolution with the goal of surviving in the short term and thriving in the long term. Software projects are typically approached to meet one of short or long term, but not both. Either the project is a fragile edifice where the capability to meet customer needs over the long term is in doubt, or the project team spends endless analysis time that compromises its ability to ship, if indeed it ever happens. To deal with the short- versus long-term paradox, we need to embrace change, not be afraid of it, and we must recognize that software is complex, not pretend that it is simple. No software product stands completely on its own, and the development team must be able to interact with and foster its ecosystem. Good software products, companies, and the open source movement clearly exhibit these traits.
The principles and practices we use for project management, software architecture, and even those that help us effectively work together are hugely important in defining the richness of the ecosystem we want our product to inhabit and/or create. Linux, for example, would not be as successful as it is today if it weren't managed as an open source project, didn't have a modular architecture that allows multiple programmers to modify it at once, and have a license that allows businesses to freely use it.
Sustainable development is a pace that can be maintained indefinitely. This does not mean slower developmentquite the opposite. Sustainability is not attained by artificially limiting work weeks to 40 hours as some books claim; the number of hours worked is an outcome of being in control of the project and having software with a consistently low cost of change. Sustainable development requires a singular focus on a form of technical excellence that regularly provides useful software to customers while keeping the cost of change low. Achieving this is not easy because the more common forms of technical excellence concentrate on designing and writing perfect software while being change adverse.
This book describes the principles and practices required for change-embracing technical excellence; these principles and practices promote sustainability and result in faster development with less effort through having a consistently low cost of change. For teams who are used to unsustainable development, the experience is best characterized as liberating because they are able to deal with change, not be afraid of it or view it as a risk.
Sustainable development and technical excellence are vital parts of agile development. The agile development movement is challenging the traditional approaches to software development. As a result, I believe the software industry is on a path to creating a widely accepted new way to develop good software that uses the best practices from both traditional and agile practices. However, this is going to take time because agile development is so new and is only now starting to be noticeably employed in a widespread manner. It's going to take time for the mindset and practices to sink in to the industry, to be accepted, and to eventually be incorporated into the education given to new software developers. In short, I think the current debates in the software industry are healthy.
What I find particularly enlightening (and heartening) is that software developers aren't the only ones challenging the standard waterfall development practices. Huge and successful manufacturing companies are using lean manufacturing practices to get their products to market faster, and even traditional construction companies are beginning to embrace lean construction as a way to reduce costs and increase efficiency. Underlying all these new approaches, even though they are completely different industries, is the need to connect with customers to ensure that the resulting work meets the needs of customers, is built in the least possible amount of time with optimal efficiency, and essentially embraces the chaos of the short term to maximize the chances of long-term survival.
Balancing the short and long term requires projects to be more driven by a vision than tracked against a plan. Just as important, though harder to describe, is the conscious tradeoff being made in these approaches toward good enough analysis and documentation of the results of the analysis as opposed to an emphasis on the documents as the guarantor of success. By its nature, this relies on the people doing the work to be self-disciplined and professional as opposed to thinking that a process with its mandatory documentation applied to that group will provide the necessary results. This is a roundabout way of stating that agile development and lean methods emphasize flexibility and collaboration over stifling bureaucracy to minimize wasted effort. And this is why agile and lean practices are as much about culture change as they are about practices.