Software Engineering Reconstructed

I have twice reported on the mismatch between our use of the term software engineering and the practice of engineering.^[2] As part of and following those reports, I have been researching the question of what "engineering" consists of in practice, where we went wrong with our interpretation of the phrase software engineering, and how we can put software engineering back on a solid footing. This section contains what I have learned so far.

^[2] The first edition of this book and "The End of Software Engineering and the Start of Cooperative Gaming" (Cockburn 2004b).

Where the Term Came From

The term "software engineering" was coined in 1968, at the NATO Conference on Software Engineering. It is fascinating to read the verbatim reports of that conference and compare what those attendees were saying with what we have created since then.

The first thing to note is that the conference organizers did not deduce that software development was like engineering. Rather, they were deliberately being provocative in coining the term "software engineering." We read in the introductory note, "BACKGROUND OF CONFERENCE" (Naur 1968, p. 8):

"In the Autumn of 1967 the Science Committee established a Study Group on Computer Science. The Study Group was given the task of assessing the entire field of computer science, and in particular, elaborating the suggestions of the Science Committee.

"The Study Group concentrated on possible actions which would merit an international, rather than a national effort. In particular it focused its attentions on the problems of software. In late 1967 the Study Group recommended the holding of a working conference on Software Engineering. The phrase 'software engineering' was deliberately chosen as being provocative, in implying the need for software manufacture to be based on the types of theoretical foundations and practical disciplines, that are traditional in the established branches of engineering."

Where We Went Wrong

The second interesting thing to note in those proceedings is the mismatch between what they claimed engineering consists of and how they described their own methods and recommendations for developing software. For example, they make the following false comparison (Naur 1968, p. 12):

"We build systems like the Wright brothers built airplanesbuild the whole thing, push it off the cliff, let it crash, and start over again."

In fact, the Wright Brothers practiced engineering in its best possible sense: They used theory, experimentation, guesses, and feedback. The Wrights' account (Wright 1953, pp.1518) reads as follows:

"In order to satisfy our minds as to whether the failure of the 1900 machine to lift according to our calculations was due to the shape of the wings or to an error in the Lilienthal tables, we undertook a number of experiments to determine the comparative lifting qualities of planes as compared with curved surfaces and the relative value of curved surfaces having different depths of curvature . . . In September we set up a small wind tunnel in which we made a number of measurements . . . but still they were not entirely satisfactory. We immediately set about designing and constructing another apparatus from which we hoped to secure much more accurate measurements . . . we made thousands of measurements of the lift, and the ratio of the lift to the drift with these two instruments."

But engineering is more than just calculating tables. It consists of interacting with a material (air, in the Wright Brothers' case). The activity, "doing engineering," is learning the qualities of the material and devising (inventing) ways to accomplish what one wants with the material. There are almost always conflicts, so a perfect solution is seldom available. Finding a solution that works adequately in a particular situation requires reframing the problem over and over as more is learned about the material, the problem, and the set of possible solutions.

Donald Schön, a professor at MIT, refers to this constant reframing as having a "reflective conversation with the situation." In The Reflective Practitioner (Schön 1983), he catalogs, among others, chemical engineers tackling a new problem. These chemical engineering graduate students were asked to create a manufacturing process to replicate a particular patina on china that came from the use of cow bone, which would soon become unavailable. The students progressed through three stages in their work:

• They tried using chemical theory to work out the reactions that were happening with the bone so that they could replicate them. They got completely bogged down using this approach.

• Abandoning theory as too difficult, they next experimented with processes in whatever form they could think up. This also got them nowhere.

• Finally, their professor coached them not to replicate the bone process but to devise a process that resulted in a similar effect. They combined bits of theory with experiments to make incremental progress, getting one category of improvement at a time and reducing the search space for future work.

Their third stage shows their reflective conversation with the situation. As they learned more about the problem and the solution, they were able to make better moves in their game.

Schön's account meshes perfectly with Pelle Ehn's description of software design as a language game (see Appendix B):

"In the conversation with the materials of the situation, the designer can never make a move that has only intended implications. The design material is continually talking back to him. This causes him to apprehend unanticipated problems and potentials, which become the basis for further moves."

Engineering, then, proceeds as a cooperative game of invention and communication, in which the moves come from theory or inspiration and are checked with experiments that reveal more about the situation.

How, then, did engineering become so misunderstood? Schön relates:

"Harvey Brooks, the dean of the Harvard engineering program, was among the first to point out the weakness of an image of engineering based exclusively on engineering science. In his 1967 article, "Dilemmas of Engineering Education," he described the predicament of the practicing engineer who is expected to bridge the gap between a rapidly changing body of knowledge and the rapidly changing expectations of society. The resulting demand for adaptability, Brooks thought, required an art of engineering. The scientizing of the engineering schools had been intended to move engineering from art to science.

"Aided by the enormous public support for science in the period 19531967, the engineering schools had placed their bets on an engineering science oriented to "the possibility of the new" rather than to the "design capability" of making something useful . . . Practicing engineers are no longer powerful role models when the professors of highest status are engineering scientists . . . by 1967 engineering design had virtually disappeared from the curriculum, and the question of the relationship between science and art was no longer alive . . ." (Schön 1983, pp. 171172)

People mistakenly think that since engineers use mathematics in their craft, their predictions about how long a project will take (or how much it will cost) will be similarly accurate. However, civil engineers fail in the same way as the average software developer when put in similar situations. As just one example, the project to build a highway under the city of Boston was estimated in 1983 to cost $2.2 billion and be completed in 1995. It was still incomplete in 2003, with a cost estimate of $14.6 billion (Cerasoli 2001, Chase URL). The 600% cost overrun was attributed to the project being larger than previous projects of this sort and to its use of new and untried technologies.

After studying this and other civil engineering projects, Martin Fowler quipped in a public talk, "Compared to civil engineers, software developers are rank amateurs at cost overruns."

Popular expectations about engineering are faulty because popular understanding of engineering as an activity is faulty. Once we understand engineering as an economic-cooperative game, the difficulty of accurately predicting the trajectory of an engineering project becomes understandable.

Reconstructing Software Engineering

If we reconsider engineering as a cooperative game of invention and communication and as reflective conversation with a situation, can we create a better understanding of the term software engineering? I believe so, and I believe that we can fit software development in the proper engineering tradition by building on the foundations of craft, the cooperative game of invention and communication, and the lessons we have learned from lean manufacturing.

The craft model captures individual behavior. The cooperative game model captures team behavior. Lean manufacturing theory captures the economics of structures and strategies.

The key in transferring lessons from manufacturing to engineering is to recognize that the "unvalidated decision" is the unit of inventory in engineering (or cooperative games of invention and communication in general).

Here are the components of the restructuring.

Craft

In The Reflective Practitioner, Schön follows the working habits of architects and engineers. In that book, we see people making predictions as to what their situation will support; then they do some of their work and watch what the situation actually produces, and from that, they alter their understanding. They update their understanding of the problem and the proposed solution according to the changed situation and their new learning. This "reflective conversation with the situation" is just what the Wright Brothers did in designing their airfoil.

Craft implies skill in working in a medium, mental or physical. There are many materials in an engineering project and therefore many skills or crafts to become adept at. Some people need people management skills, others need mathematical skills, others need visual design skills, and so on.

Each skill and material brings its own specialized vocabulary. Ceramicists "wedge" clay and study the shrink rate of different glazes compared to clays. Civil engineers work with the tensile strength, fatigue rates, and thermal expansion of their materials. Writers look for flow and development in their texts. Programmers look at cohesion and coupling, testability, clarity of expression, and computational complexity in their algorithms. Testers work with test coverage and probabilities of occurrences. User interface (UI) designers work with cognitive-motor task switching times, recognition times, color scales, and user work patterns. Project managers work with people and are concerned with what builds or tears down trust, community, and initiative.

In each case, craft practice requires practitioners to have a reflective conversation with the material, using the specialized vocabulary. They work with the constraints offered by the materials, the vocabulary, and the project.

On a software project, the UI designer is constrained by the two-dimensional screen, the operating system's rules, and the UI component set that has been selected for use on the project. The UI designer will learn what is possible and not possible in the framework (and maybe also what is not supposed to be possible but can be done anyway). As work progresses, using the evolved understanding of what can and cannot be done, the UI designer may propose a different workflow to the users from what they originally requested.

The materials for a system architect are architectural standards, application program interfaces (APIs), and signal propagation technologies. The architect balances performance degradation with ease of change as he or she adds APIs and separates the system across different boxes or locations. This work is partly a mathematical activity and partly heuristic.

The materials for a programmer are the programming language, the operating system, and the components or APIs the programmer is obliged to use. Some programming languages are very malleable, such as LISP, APL, Smalltalk, Python, and Ruby. Other languages are less malleable, such as Ada, C++, and the other strongly typed languages. Programmers sometimes compare working in the former languages to working with clay and working in the latter languages to working with marble. Wonderful sculptures can be made from either clay or marble, but the necessary skills and the approach differ significantly. So, too, for the different programming languages.

In software engineering, the crafts include at least

• Project management

• "Requirements" elicitation (in quotes because the development team can usually propose alternatives, which meant they weren't really requirements in the first place)

• User interface design

• System architecture

• Program design and programming

• Database design

• Algorithm design

• Testing

• Communicating

To develop professionals in our field means we should recognize the various crafts needed to carry out a project. It means training newcomers in the basics of those crafts. It requires professionals in the field to continue developing skills as a natural part of being in a craft profession. It means honoring the different crafts required to complete a project.

The crafts define the specific moves that the people make in the cooperative game.

Cooperative Games

Engineering is seldom done alone. Engineers exchange their thoughts and proposals as they proceed along their intertwined paths. They communicate with sponsors and the users: The sponsors must find ways to fund the work, and the users must respond to the unexpected ways in which the system will change their lives.

The following is a description of engineering work at Lockheed's Skunk Works facility. In this extract, it is easy to see the principles of the cooperative game in action and the cross-coupling of the crafts. This is an outstanding example of a group holding a reflective conversation with the situation as a team.

Skunk Works Rooms

"Kelly kept those of us working on his airplane jammed together in one corner of our [building] . . . My three-man thermodynamics and propulsion group now shared space with the performance and stability-control people. Through a connecting door was the eight-man structures group . . . Henry and I could have reached through the doorway and shaken hands.

". . . I was separated by a connecting doorway from the office of four structures guys, who configured the strength, loads, and weight of the airplane from preliminary design sketches . . . [t]he aerodynamics group in my office began talking through the open door to the structures bunch about calculations on the center of pressures on the fuselage, when suddenly I got the idea of unhinging the door between us, laying the door between a couple of desks, tacking onto it a long sheet of paper, and having all of us join in designing the optimum final design . . . It took us a day and a half . . .

"All that mattered to him was our proximity to the production floor: A stone's throw was too far away; he wanted us only steps away from the shop workers, to make quick structural or parts changes or answer any of their questions." (Rich 1994)

Lean Manufacturing

Lean manufacturing may seem a peculiar foundation for software engineering. Surely, if there is one difference between manufacturing and engineering, it is that manufacturing is about making the same item many times, and engineering is about approaching situations that are unique each time.^[3]

^[3] However, see Mike Collins' description of lean manufacturing in a mass customization environment, on page 323.

Replace manufacturing's flow of materials with engineering's flow of decisions, however, and the rules governing the two are remarkably similar.

One remaining difference is that the flow of decisions in an engineering project has loops (requests for validation and correction), which designers of manufacturing lines try hard to avoid. These loops complicate the optimal strategies for the team but do not change the rules of the situation.

Figure 1.1-2 shows the flow of decisions in a software development project. The top image highlights the decisions being made; the bottom image highlights the feedback loops. Let us look at the decisions:

• The users and sponsors make decisions about what they want. They feed those decisions to UI designers and business analysts (BAs).

• The UI designers and BAs, collaboratively with the users or on their own, make detailed decisions about the appearances, behaviors, and data to be put into the system. They feed those decisions to the programmers.

  Figure 1.1-2. Top: Flow of decisions on a software development project, showing types of decisions made. Bottom: Flow of decisions on a software development project, highlighting feedback loops.

  

• The programmers make decisions about the program. Those decisions probably change some of the users' requests and the decisions made by the UI designers and BAs, but let's skip that topic for the moment. However they make their decisions, those decisions go to the testers.

• The testers make decisions about where the foregoing people have made mistakes.

The testers can't tell whether the program is correct until they get running code; the programmers can't know what to code until the UI designers and BAs make their detailed choices; and the UI designers and BAs can't decide what to request until the sponsors and users make up their minds.

As with manufacturing, there is "inventory," there are "work-in-progress" queues between workstations, and there is the problem of selecting batch sizes for those handoffs.

The engineering team's inventory is made up of unvalidated decisions. The more decisions saved up and passed from one person to another at one time, the larger the batch size involved in that handoff.

Texts on lean manufacturing (for example, Reinertsen 1997) stress that increasing the batch size generates almost uniformly negative consequences on the system's overall behavior. Small, natural variations in work speed produce unpredictable delays at the handoffs. In the end, it is hard to predict the flow through the system, even for a well-designed system. The cure is to reduce batch size at the handoff points and then choose from a collection of techniques discussed in the lean manufacturing literature to deal with the inevitable bubbles of inventory that pop up from place to place within the system.^[4]

^[4] The simplest method is to cross-train people to be able to do the work of those adjacent to them in the flow. When inventory starts to build up at a handoff point, the upstream worker joins the downstream worker in clearing it.

In software development, reducing batch sizes corresponds to designing, coding, testing, and integrating smaller amounts of functionality at one time. It is hard to imagine what just "one" piece of functionality is, but it is easy to see that on most software projects, many unvali-dated decisions are batched up in the requirements document, many unvali-dated decisions are batched up in the UI design, and many unvalidated decisions are batched up in the assembled code base before the testers are given anything to test. Reducing the functionality requested in one delivery increment is the first step in reducing batch sizes.

As with manufacturing, a cost is associated with reducing batch sizes. In manufacturing it is the cost of switching from one job to another. Lean manufacturing companies spend a lot of money reducing job switching times so that they can gain the benefits of smaller batch sizes. (Important note: They don't make excuses that they can't have small batch sizes because they have high job switching costs; they redesign their job switching procedures to lower that cost so that they can use smaller batch sizes. This difference is significant.)

Engineering groups have not only the task-switching costs but also the cost of injecting new work into the system through the feedback loops and the cost of revising the designs as the requests change. Unresolved arguments still rage in the software development community about how small the batch sizes can be made in UI and database design.

Figure 1.1-2 shows one other important characteristic of a development team: There is a different number of specialists at each point in the flow, so they have different capacities for clearing their decisions. Alert managers do two things when studying these decision-flow capacities:

• They work to balance the relative capacities at the respective stations. It does little good to have lots of BAs and not enough programmers, or lots of programmers and not enough testers.

• They decide what to do with the specialty groups where there is excess capacity. If they can't simply send the extra people home, they have to decide how to use those people. This is where Principle #7 from Chapter 4 comes in ("Efficiency is expendable at non-bottleneck stations"). That principle points to a wide set of management strategies that can be used in different circumstances (Cockburn 2005e).

Now to deal with the feedback (Figure 1.1-2, bottom). When the testers find a mistake, they send a report back to the programmer or the UI team. These defect reports constitute additional inventory for the programmer or UI team to handle. A loop exists from the UI designers to the sponsors and users. The UI designers make a mock-up of the user interface, show it to the users, and come away with a stack of rework decisions. A third loop may exist from the programmers to the users and sponsors, creating more decisions being fed to the UI designers, BAs, and programmers.

The same results apply to batch sizes at the handoff points and also to the use of creative strategies at non-bottleneck stations.

All these dependencies exist even when the people are sitting next to each other in the same room.

I suggest that those three foundations provide a model that is both sound and practical. It applies in general to cooperative games and in particular to both engineering and software development (or software engineering, as it is now safe to write).

The model helps us gain insight into both education and project management. On the education side, people should be trained

• In the vocabularies of their respective crafts, whether they are civil engineering, user interface design, requirements negotiation, programming, or testing

• In the principles and techniques of invention and communication

• In the mathematics and lessons of lean manufacturing systems

• In strategies for dealing with dependency flows and loops

On the management side, managers should

• Learn how to work from queue size and flow to improve their project planning strategies.

• Pay close attention to the quality of the communication, morale, and community present in their teams.

• Help their teams to become better skilled at all facets of their respective crafts.

• See that people get cross-trained in other specialties so that they can help each other smooth the inevitable bubbles that accrue randomly at the hand-off points.

Collaboration in Other Engineerings

Dr. Stephen C-Y. Lu, David Packard Chair in Manufacturing Engineering, University of Southern California, writes that "human behavior dynamics impact technical decisions that cause societal changes, which, in turn, shape human and social dynamics to influence future technical decisions. Such adaptive, cyclic socio-technical interactions represent the true fabric of modern engineering."

In 2003, the International Academy for Production Engineering (CIRP) started a working group to study engineering as collaborative negotiation.^[5] They felt the "need of broadening the engineering foundation from 'Engineering is Applied Science' at the present to 'Engineering as Collaborative Negotiation' (ECN) in the future. This ECN paradigm encourages engineers to use various negotiation approaches and techniques when making joint engineering decisions."

^[5] See http://wisdom.usc.edu/ecn

"Engineering as Collaborative Negotiation (ECN) is a decision-making paradigm that we propose as the foundation for collaborative engineering research. As such, this new paradigm offers a different perspective towards the basic questions of what is collaborative engineering, how engineers should engage in this activity, and how it can be better supported in engineering practice."

The following sidebar comes from Dr. Lu, Chairman of CIRP's Working Group on Engineering as Collaborative Negotiation.

Collaborative Engineering and ECN

by Stephen C-Y. Lu Professor

Collaborative engineering is rooted upon both natural sciences (the science of the nature based on physics) and social sciences (the science of the artificial based on preferences). It is made up of technical decision-making by individuals, plus the social endeavor of "teamwork" by a team of individuals.

Collaborative engineering is fundamentally different from traditionally understood engineering. The traditional engineering supposes that individuals' decision perspective is static and unaffected by the dynamic social interactions that always take place when working with other individuals. It avoids the fact that social interactions during teamwork can influence individuals' perspectives, which in turn, will alter their preferences in making decisions. It overlooks the significant benefits of collaboratively constructing a "shared mind" through participative joint decisions by multiple engineers.

Collaborative engineering is an emerging field. We are looking into particular types of collaborative activities in engineering, hypothesizing how these collaborative activities are accomplished in practice, and validating these hypotheses to develop a theory with a framework of collaborative engineering.

We present the Engineering Collaboration via Negotiation (ECN) hypothesis to guide our research in developing a theory of collaborative engineering:

"Engineering Collaboration via Negotiation (ECN) is a type of collaborative engineering activity. It is a socio-technical group decision-making activity, whereby a team of engineers collaborate with each other to resolve conflicts, bargain for individual or collective advantages, agree upon courses of action, and/or craft joint decisions that serve their mutual interests, on some technical matters."

In this description, the word social refers to collaborative behaviors that take the interests of others into account as well as the collaborative characteristics between multiple engineers. At the same time, we also use social to represent common stakeholder characteristics (such as his/her background, objective, interests, and criteria), which can influence collaborative team dynamics during social interactions of collaborative engineering. Those characteristics, initially brought into the collaborative teamwork by the participating engineers, are continuously co-constructed and dynamically evolve during the social interaction process.

Therefore, the term "socio-technical" in our ECN hypothesis signifies the mutual consideration of and the true integration/synergy between the social (teamwork) and technical (task-work) aspects of engineering activities. The ECN hypothesis explicitly acknowledges collaborative engineering as a dynamic interface between individual decisions and group interactions, and as an assimilation of social and technical activities operating in parallel over different time, space, and discipline scales in an engineering team.

The inability to model the social and technical interactions as an integral part of engineering decisions has been a major roadblock for collaborative engineering research. Social scientists have long understood that human behavior dynamics impact technical decisions that cause societal changes, which in turn, shape human and social dynamics to influence future technical decisions. Such adaptive, cyclic socio-technical interactions represent the true fabric of modern engineering.

Thus, the ECN hypothesis explicitly recognizes the fact that collaborative engineering is a socially mediated technical activity.

The ECN-based study treats collaborative engineering further as a human activity within a socio-technical environment, a conscious human endeavor driven by goals and motives. Psychology theory suggests that interactive human activities include three types: coordination (plan each others' activities for sequential dependency), cooperation (taking each others' decisions into consideration for reciprocal reliance), and construction (develop new understanding and solutions for shared knowledge). We believe that collaborative engineering is more about the construction type of human interactions, and the group construction (or co-construction) activity in engineering is best achieved by collaborative negotiation based on our ECN hypothesis.

Thus, ECN-based research advocates that a negotiation activity is best carried out by the co-construction process, where everyone is dynamically and simultaneously influencing (or co-constructing) everyone else's decision perspectives (or value functions). Human perspectives are always dynamically changeable, due to social interactions with others. We use the term "co-construction" to indicate that such perspective changes are always reciprocal in collaborative engineering negotiation.

Finally, the scope of negotiation in the ECN-based study is extended from traditional dispute resolutions to also include a new viewpoint of collaborative deal-making and the possibility of collective innovation in engineering. As such, the result of collaborative engineering is more than just a single agreement that meets stakeholders' competing preferences, but also includes an expansive set of options (or a totally new option space) from which win-win outcomes and/or new innovations, which are impossible to attain individually, can be obtained collectively for much improved consensus.

In other words, in addition to conflict resolutions, the ECN study also has the goals of promoting collectivity via mutual knowledge probing (jointly explore understanding and application of knowledge) and shared value creation (develop common problem definition, value proposition, and solution formation) in engineering decisions.