Building the Third System | Linux and the Unix Philosophy

3.8 Building the Third System

The goal then is to build the Third System, for it is that which gives everyone the biggest return for the effort expanded. Containing the ideal mix of features, resource consumption, and performance, it includes only those features that people actually use. It strikes a proper balance between disk space, memory, and CPU cycles versus the level of performance delivered. Users recommend it, and customers buy it year in and year out.

How do you build the Third System? You build the other two systems first. There is no other way. Any attempt to change the order merely guarantees that you will build more First or Second Systems than would otherwise be necessary.

There are some shortcuts, though. The secret is to progress from the First to the Third System as quickly as possible. The more time spent on building the first two systems, the longer it will take to achieve the Third System's optimum balance. If you keep the cycles involved in building the First and Second Systems short and iterative, you will arrive at the Third System faster.

Although producing the Third System is the goal of both Unix developers and those following "traditional" software engineering methodologies alike, the Unix developers take a radical approach. First, consider the way most software is written:

Think about the system design.
Build a prototype to test your assumptions.
Write detailed functional and design specifications.
Write the code.
Test the software.
Fix bugs and design flaws found during the field test, updating the specifications as you go.

The traditional software engineering methodology was invented by those who believe that it is possible to get it right the first time. "If you're going to build a system," they say, "you may as well build the Third System." The problem is, no one knows what the Third System is until it's been built.

The traditionalists like to see everything written down, as if having written specifications to the nth detail assures them that they have considered all design factors. The notion that "90 percent of the design should be complete before you write the first line of code" comes from the belief that excellent designs require forethought. They believe that by fully documenting the design, you guarantee that you have investigated all viable possibilities, and by having complete specifications, you keep things organized and ultimately increase your efficiency.

The traditional methodology results in very good specifications early on and poor specifications later. As the project gets underway, schedule pressures force the engineers to spend progressively more time on the software and less time on the specifications. Such a shift in priorities causes the specs to become out of sync with the actual product. Given a choice between a product with little documentation versus ample design specifications without a product, software developers always choose the former. People will not pay them for specifications without a product behind them.

Unix developers take an alternative view toward detailed functional and design specifications. Although their intent is similar to that of the traditionalists, the order of events differs:

Write a short functional specification.
Write the software.
Use an iterative test/rewrite process until you get it right.
Write detailed documentation if necessary.

A short functional specification here usually means three to four pages or fewer. The rationale behind this is that (1) no one really knows what is wanted, and (2) it's hard to write about something that doesn't exist. While a traditionalist may spend weeks or even months writing functional and design specifications, the Unix programmer jots down what is known about the goal at the start and spends the rest of the time building the system.

How does the Unix programmer know if he's proceeding in the right direction? He doesn't. Neither does the traditionalist. Eventually the design must be shown to the prospective end user. The difference is that whereas the traditionalist presents to the user a massive tome containing a boring description of what the system is going to be like, the Unix programmer shows the user a functional application.

The traditionalist wonders whether the final product will meet the user's needs. He cannot be sure that the user has communicated his requirements effectively, nor can he be certain that the implementation will match the specifications.

One the other hand, the Unix approach provides the user with a functional First System that he can see and touch. He begins to get a feel for how the final product will operate. If he likes what he sees, fine. If not, it's far easier to make major changes now instead of later.

Remember, though, that a characteristic of the First System is that it displays a concept that ignites others' imaginations. Viewing a live implementation of the First System sets off a creative spark in the user's mind. He starts to imagine what he might do with the product. This spark feeds on itself, and he begins to think of new uses, some of which may not have been thought of by the original designers.

At this point the Unix approach leapfrogs over the traditional engineering method. While the traditionalist's user wonders how the product will look, the Unix developer's user is already thinking of what to do with the working prototype.

For the Unix user, the iterative design process has begun. He and the developers are proceeding toward the Third System. Once the developers receive the initial reactions from the users, they know whether they are on the right track. Occasionally, the user may inform them that what he wanted is not what he received, resulting in a complete redesign. More often than not, the user will like part of the design and will provide useful commentary on what must be changed. Such cooperation between the developers and the end user is tantamount to producing a Third System that meets the user's needs in most respects.

The best time to write a detailed design specification is after the iterative design process is complete. Of course, by then a lengthy spec may not be necessary, since it is mainly intended for the developers and interested third parties to read and review. The users and the developers have been reviewing the application throughout the iterative design process, so writing a detailed spec at this point may not serve any useful purpose. If one is still required for some reason, it is far easier to document an existing application.

Some people have expressed concern that the Unix approach to software development, although quite suitable for small systems, does not work for larger ones. They argue that beginning the coding phase without a thorough design is engineering suicide. They claim that lack of adequate forethought can only lead to disaster.

We're not saying that one should plunge into coding a large system without an adequate design, however. Some amount of deliberation is necessary in order to define proper goals. It's just not very useful to document every detail before proceeding because the details are likely to change. The key here is to identify how the system might be broken into smaller components. Then each component can be architected within a much smaller design domain.

Also consider that while Unix has had the benefit of very little up-front design, it has evolved into a system capable of handling tasks once relegated to larger systems. A system of its stature traditionally would have required countless tomes of functional specifications and detailed design documents. Plenty of Unix documentation exists today through the efforts of a few enterprising technical publishers, but most of it describes designs that already exist. Original specifications are practically nonexistent.

Unlike most systems planned in the traditional way, Unix evolved from a prototype. It grew out of a series of design iterations that have transformed it from a limited-use laboratory system into one called upon to tackle the most arduous tasks. It is living proof of a design philosophy that, although unorthodox to some, produces excellent results.