Chapter 11: A Cathedral? How Bizarre

First they ignore you. Then they ridicule you. Then they fight you. And then you win.

Gandhi

When the Rolling Stones released their Forty Licks CD on October 1, 2002, everyone thought it would go straight to the top of the charts over the following week. It almost did. It made it to the number-two spot. The Stones confirmed that, indeed, you can't always get what you want. This quintessential greatest-hits collection by the Greatest Rock Band of All Time surrendered the number-one spot because the listening public couldn't help falling in love with another collection of greatest hits by no one other than Elvis Aaron Presley, released on September 24, 2002, and modestly titled ELV1S 30 #1 Hits^[1].

Suspicious minds wanted to know how an old hound dog like Elvis could prevent the Stones from getting their satisfaction. How was it that they couldn't get him off of their cloud? Why were the Stones under Elvis's thumb, instead of the other way around?

The answer is simple. The Stones' music was done in a cathedral, while Elvis borrowed his music from the bazaar of American music. Huh? Before you go having your nineteenth nervous breakdown over this one, let me explain. Maybe you'll see why the Stones, instead of Elvis, were crying in the chapel.

Since the early days-back when time was on their side-Mick Jagger and Keith Richards spent many nights together, penning hit after hit that blew our noses and our minds like a, well, you get the idea. It was only rock and roll, but we liked it. Jagger and Richards seemed to be incapable of anything else. They did one thing, and they did it well.

For all their creative genius, however, their work was a closed-source, proprietary kind of effort. Rarely, if ever, did you hear songs by the Stones that came about as a result of collaboration with third parties. Wild horses couldn't pull them out of the cathedral of their minds and into the maelstrom of opensource cooperation. They were victims of their own NIH.

Meanwhile, the wonder of Elvis was that he operated in a bazaar fashion. He liked things all shook up, and his musical styles ran the gamut from rock and roll to blues, country, gospel, ballads, and show tunes, depending on what his latest flame was at the time. His good-luck charm was his versatility. It didn't matter whether it was on the big screen, in front of a studio microphone, or on the famous Ed Sullivan television show-Elvis had a burning love to perform … everything.

Elvis was also a master of reuse. While Mick Jagger and Keith Richards were busy writing good songs, Elvis was busy borrowing them-borrowing them and turning them into hits, that is. While he could have written many of the songs he performed, he chose instead to leverage the work of other song-writers in addition to his own. That allowed him to have a much greater impact on the entertainment world. Yes, he became very wealthy from the sales of his music and merchandise. But many others also shared in his good fortune. That's what happens in a true open-source collaborative kind of environment. Everyone benefits.

The King is dead. Long live the King.

Some might say that this comparison of the Rolling Stones and Elvis to Eric Raymond's The Cathedral and the Bazaar^[2] is a bit of a stretch. After all, the closest Elvis ever got to a bazaar was probably the Memphis Flea Market out on Maxwell Boulevard. And the least likely place to find the producers of Goat's Head Soup would be a cathedral. So let's take a look at a real cathedral, or rather a language once found in cathedrals: Latin.

Latin once held the place that English holds today. Through the conquests of Rome, Latin usage had steadily grown from about 250 B.C. until the 6th century. Around that time, the Roman Catholic Church pronounced that Latin was the language for scientific, philosophical, and religious writing. So, from that point forward, if you wanted to be a cool priest or scholar, you had to speak Latin. However, with the gradual decline of the Roman Empire, invading barbarians, who, because of their war-like nature had little interest in being intellectually stylish, usually modified Latin to their liking. In adding their own idioms for kill, maim, and plunder, they felt it was their God-given right to subdue the language as well. Meanwhile, the priests and the scholars decried this pollution of their beloved Latin, further continuing to promulgate its importance to an ever-shrinking religious minority, until such time as the only ones who could speak and write Latin did so in the quiet seclusion of cathedrals.

On the other hand, English grew to become the international language not because it was pure or holy, but because it was adaptive. It would admit the entrance of practically any foreign word or concept into its everyday usage. It was able to interface with other cultures better than any other language to date. And interface it did. In the wild diversity of the bazaar we call life, Eng lish found its place among the largest number of nations primarily because of its ability to connect to and with anything.

This is precisely why open-source software (OSS) flourishes. It can be used by anyone to connect anything to anything else. It does this because of its openness: open standards, open protocols, open file formats, open everything. Nothing is hidden in the OSS world. This openness makes it possible for software developers to see how their predecessors did things. If they like what was done before them and can use it, that's great. They have found a useful tool and saved themselves some time. If not, they're at least free to view past mistakes and improve them.

Meanwhile, those software developers who toil away in the cathedrals developing "superior" software eventually find their markets overtaken by works of the open-source developers. Why? Because cathedral-style software developers only develop programs for others who live in the same cathedral. Whether this cathedral is a single company, a single community, or a single nation, sooner or later the software must interface with software outside the cathedral, or else it will surely die.

Adherents of the Unix philosophy and especially today's Linux developers are acutely aware of this. For example, Linux systems sharing a mixed environment with Windows systems often run Samba, a Windows-compatible file-sharing application that is compatible with Windows SMB networking. The irony is that, according to one benchmark, Linux actually outperforms Windows 2000 running Microsoft's own SMB networking.^[3]

Most Linux distributions come with the ability to access Windows file systems on other disk partitions. Can Windows perform the reverse? Can Windows access Linux file systems on other partitions? No. Windows is a closed-source, proprietary system that, like the perennial ostrich with its head in the sand, wants to pretend that other file systems don't exist.

By now you probably perceive some similarity between OSS and software that adheres to the Unix philosophy. By themselves, many works of OSS look like small potatoes compared to the larger "industrial-strength" (as some believe) closed-source proprietary software applications. After all, these works are typically produced by lone individuals, at least in the early stages. As a result, these early efforts are typically small projects, lacking in marketing-style flash, but full of substance. But that's okay. Remember that in the Unix world, small is beautiful.

Eric Raymond referred to these works as software that "scratches a programmer's personal itch." What appears to be a common goal of many OSS developers is that they just want to get it written. Their backs are often against the wall because of their daily job pressures, and they don't have much time for frills. So, they usually skip most of the fancy glittery stuff that the marketing droids love and, instead, do one thing well, which is to produce a lean, mean application that solves a personal need. In the case of the successful ones, these bare-bones solutions strike the matches that set others' imaginations on fire. That is how the hit OSS applications are born.

Once the fire is lit, other members of the OSS development community at large begin to contribute to the program. At first, the program will attract a small, but loyal, following. Then, eventually the software takes on a life of its own, and it grows beyond the scope of its original intent. During this time, dozens of programmers evolve this first system into a larger, more encompassing second system. As of this writing, Linux itself is in the second system phase. A merry time is being had by all.

Obviously, if one wants to strike the match that turns a one-person OSS project into a software phenomenon, one must build a prototype as soon as possible. It's not enough simply to talk about a great idea for a piece of software. In the open-source world, those who gain the most respect are those who write the initial version of the software or contribute something of significance later on.^[4]

In order for an OSS project to entice the greatest number of users, it must be available on the largest number of platforms. So good OSS developers choose portability over efficiency. A program that takes advantage of specific hardware features on a particular machine will not develop as large a following as one that can be run by anyone on his or her favorite platform. By moving away from closed, proprietary architectures and toward open ones, a program can maximize its market potential by addressing the largest number of target platforms.

Consider Microsoft's Internet Explorer Web browser and its competitor from the open-source world, Mozilla. As of this writing, Internet Explorer has a larger market share than Mozilla, mainly because it is the dominant browser on Microsoft Windows. Windows currently owns the largest share of the desktop market. Mozilla, on the other hand, runs on Windows as well as virtually every Linux and Unix platform out there. As users begin to cut through the Internet Explorer marketing hype and realize what a fine browser Mozilla is, more and more of them on all platforms will switch to Mozilla. Meanwhile, Internet Explorer has barely grown beyond the Windows-installed user base because it lacks the portability to make porting it to other systems inexpensive. If Mozilla succeeds at capturing even half of the Windows-installed base, then it will have won the competition. Any portable browser good enough to capture the hearts and minds of half of the Windows user base would likely capture most of the Linux user base as well.^[5]

In order for OSS programs and interfaces to be ported to other platforms easily, it helps to store data in flat text files. This kind of openness exposes the file formats to the rest of the world and encourages healthy debate over their contents. This debate ultimately results in open standards for data transmission and storage that are far more ubiquitous than closed-source, proprietary standards.

A good example of this approach is that taken by the folks at OpenOffice.org. They opted for XML as a file format, an excellent choice. XML is a superset of HTML, the language used in Web browsers to display text, tables, and such. XML files are flat text files in that they can be edited with a regular text editor (not a word processor) such as vi or Windows Notepad. As an industry standard, XML may possibly be the best choice for structured interoperable data, not only in flat text files, but also in flat network messages as well. Undoubtedly, the format will evolve over time. As more and more eyes view the data files, the errors and omissions in the format will become evident. This will strengthen and refine the format, as it eventually becomes better than any closed-source format for document storage currently available.

OpenOffice.org's use of flat text files is nothing new, of course. Office suites in the real world have always used flat text. Business correspondence is normally carried out on 20-pound white paper transmitted in Number 10 business envelopes. These documents are

• Portable: The envelope can be carried down the hall or across the world. Scaleable mechanisms exist for moving documents from place to place, individually and in large groups.

• Accessible: Anyone can open the envelope. You don't need a special proprietary tool purchased at great expense from a single vendor to view the contents.

• Searchable: Documents can be indexed and placed in file cabinets according to whatever scheme the recipient desires. They can be retrieved in random-access or sequential fashion. The entire text can be searched, albeit slowly.

• Resilient: Documents can be stored and retrieved thousands of times, today and in the future. One does not need either to upgrade one's office suite to read the newest documents or to install an older version in order to view a ten-year-old document.

No, we're not suggesting that you abandon your computer and go back to using paper. Paper is still a death certificate for your data, for reasons menioned elsewhere in this book. However, there is another kind of death that threatens your data: closed-source, proprietary file formats. When your data is stored in non-human-readable file formats owned by a single vendor, then your data is at enormous risk, as Peruvian congressman Dr. Edgar David Villanueva Nuñez pointed out accurately in a letter to Microsoft^[6]:

The use of proprietary systems and formats will make the State ever more dependent on specific suppliers. Once a policy of using free software has been established (which certainly, does imply some cost) then on the contrary migration from one system to another becomes very simple, since all data is stored in open formats.

Open-source programs demand open data formats.

As OSS developers release their software for public consumption, they use software leverage to their advantage. This leverage works both ways. First, the developers can use bits and pieces of other OSS projects in their projects. This saves overall development time and helps keep development costs low. Second, the release of good OSS into the world usually attracts other developers, who can leverage that work for their own purposes. Some of this work may eventually find its way back into the original work. Give unto others, and it shall be given unto you.^[7]

One way to improve the chances of success within an OSS project is to use scripts to increase leverage and portability. While there are plenty of hard-core low-level coders out there, there is a growing number of people who don't have the time or the chutzpah to work at the lowest levels. By providing an interface to a popular scripting language or two, it is possible to enhance the usefulness of any piece of software dramatically.

If these interfaces gain enough popularity, a significant subcommunity within the OSS community will form. These subcommunities are passionately devoted to experimenting with and extending such interfaces. They will hold user-group meetings and conferences, post to online mailing lists, and generally become the focal point for the greater part of activity surrounding a popular interface.

A couple of examples of this are the communities surrounding Perl and the Concurrent Versions System (CVS) for software version control. CVS has attracted quite a following within the OSS community. Because virtually all of the CVS commands can be used within scripts, numerous tools and addons have sprung up to combine them in new and interesting ways. The original authors of CVS would certainly not have predicted many of these uses.

CVS, by design, avoids CUIs. Every CVS command can be run from the command line by passing in the appropriate parameters. This has helped make it easier for CVS add-ons to spring up. The basic CVS commands, such as checkout, update, and commit, form the basis of many of the integrated tools built to enhance CVS. The various add-on tools created using these commands often have CUIs. But the basic CVS commands do not. This is one of the main reasons that CVS has flourished.

Perl has gained massive notoriety as an open-source scripting tool. More powerful than the common Unix shells, such as the Bourne and Korn shells, it excels at interfacing with other software. Nearly all Perl programs act as filters. In fact, its full name, Practical Extraction Reporting Language, attests that Perl was designed for that purpose. When a program extracts information from someplace, modifies it, and reports it (i.e., "outputs" it), it is fulfilling the role of a filter.

Perl's acceptance also grew because of its extensibility, too. Hackers (in the good sense of the word) found a veritable playground in this "kitchen sink" of languages and proceeded to add libraries to adapt it to a wide variety of situations. Today you can find a Perl module to handle anything from complex mathematics to database development to CGI scripting for the World Wide Web.

Not surprisingly, Perl is a shining example of the "worse is better" lesser tenet of the Unix philosophy-and stunning proof that that which is worse (e.g., Perl) usually has better survivability characteristics than that which is better (e.g., Smalltalk). Few people would contest the notion that Perl is a quirky language with a hideous syntax. And fewer still would suggest that Perl is anything less than rampantly successful.

It should be evident to the reader at this point that OSS more often than not adheres to the Unix philosophy. OSS lends itself to the "bazaar" style of development, a style that Unix developers in the early days coddled and today's Linux developers enthusiastically embrace. But while the Unix philosophy focuses on the design approach, much of the rhetoric coming from the opensource community highlights the software development process. The Unix philosophy combined with open-source development processes is a marriage made in heaven, to use an old cliché. They augment and complement each other nicely.

One area in which the open-source community has made significant advances is marketing. In today's fiercely competitive computing world, it is not enough to produce high-quality software in accordance with sound design principles. One must tell people that one has done so. You can have the best frazzlefloozle in the world, but if no one knows about your frazzlefloozle, it won't matter how good it is. It will fade into obscurity or languish as an undiscovered relic for years.

Many excellent Unix programs have disappeared due to a lack of solid marketing. I could name several of them, if I could only remember them. Get the point?

Microsoft has well understood the value of solid marketing, perhaps better than any other company to date. This is the primary reason why the company has been so successful. Microsoft has produced a number of hit applications, but its strength lies mainly in its ability to convince everyone that its software is the finest on the planet, regardless of quality.

The main issue I have with Microsoft software is that it is developed cathedral-fashion in a closed-source environment. Microsoft has some very talented people who have written some very good software. But Microsoft has not cornered the globe on technological innovation. There are plenty of talented people in the rest of the world who do not work for Microsoft and who write some very good software as well. Because of their cathedral style of development, the Microsoft developers have isolated themselves from the others. They cannot fully benefit from software leverage because they are not sharing anything that others can augment and give back to them. So, most Windows operating system innovation is undertaken by the Microsoft developers and is thereby limited by their vision. Open-source developers, on the other hand, have the much larger open-source development community to draw from for ideas. In terms of sheer size, the open-source development community dwarfs Microsoft's internal development community. And that-Microsoft's superb marketing efforts notwithstanding-will ultimately be Microsoft's downfall.

At the time of this writing, the Linux development community (rumored to be more than a million developers strong) began to invade the desktop environment, long thought to be Microsoft's stronghold and considered impenetrable by any outsider. In as little as 18 months, the developers of Gnome, a Windows-like application environment popular on Linux, had taken Linux from a Unix command-line user interface to one having nearly as much capability as Windows 95. It took Microsoft more than five years to accomplish the equivalent in going from MS-DOS to Windows 3.1 to Windows 95.

It's hard to see how Microsoft can keep up with this rate of innovation. It is simply outgunned. The KDE community, which may be even larger than the Gnome community, has already been shown to be quite capable of producing commercial-quality GUI software for the desktop environment. Unless Microsoft reinvents itself to meet this challenge, it is only a matter of time before Microsoft's Windows operating system becomes the next OpenVMS (i.e., great software made by a great little team).

Microsoft had ignored Linux and the open-source community for nearly a decade. Then, they ridiculed Linux, saying that it was "only a server system" and that it was not suitable for the desktop. Now Microsoft has an all-out desktop war on its hands, with KDE's minions leading the charge.

Was Gandhi right? The market will decide.

Before closing this chapter, it's important that we address one aspect of living and computing in the postmodern era that has been on everyone's mind since the September 11, 2001, attacks on the World Trade Center and the Pentagon: security.

Security has been on everyone's mind in one way or another as a result of that day. This collective security consciousness has also been on the minds of computer scientists as well. And perhaps, in a strange way, the Unix philosophy and open source hold the key to the solution.

In the world of information security and encryption technology, it has long been thought by some people that in order to provide the highest level of security, one must hide everything and batten down the hatches, as it were. The early Unix developers had a different approach. Rather than hide the password data and the algorithm used to encrypt the passwords, they chose instead to make the encrypted passwords and the encryption algorithm plainly visible to anyone and everyone. Anyone was free to attempt to crack the Unix encryption algorithm and was given all the tools to do so. In the spirit of cooperation, those who found ways to crack the Unix password file submitted their solutions to the original developers who in turn incorporated those solutions in future versions of the encryption algorithms. Over the years, after repeated attempts to defeat the Unix password mechanism, some of which were successful, Unix grew to be a far securer system than the original developers were capable of producing without the help of others.

Meanwhile, attempts to achieve security through obscurity in closed-source systems have failed miserably. Security holes in systems such as Windows NT can only be patched as fast as Microsoft, which maintains a tight grip on the source code, can plug them.

In one sense, security is a numbers game, and the numbers don't look very good for closed-source systems. For every malicious hacker (one of the bad guys) out there, there are probably 100 times or even 1,000 times as many good guys. The problem is, the closed-source company can only afford to hire 10 of the good guys to look at their proprietary source code. Meanwhile, in open-source land, 1,000 times as many good guys than the bad guys can look at the source code to resolve security problems. The closed-source companies, no matter how large they are and how much they're willing to spend, simply cannot keep up with the number of good guys in the open-source community at large.

The implications for open-source software versus closed-source proprietary software here are significant. With many more eyes looking at the security mechanisms in OSS, OSS will eventually prove the more secure of the two. Furthermore, as individuals, companies, and nations become increasingly aware of the need to audit the software used for information security, it will become evident that the only security software that can be trusted is one for which you have the source code.

At some point, everyone must decide whether to trust the producer of the software used to protect information. Everyone has to make his or her own moral and ethical judgment of the software's provider. Can that provider truly be trusted? When your finances, reputation, or physical or national security are at stake, the only acceptable solution is one that lets you verify the software down to the very last line of code.

In the world in which we live, there are both open and closed societies. The closed societies operate much like the cathedral development environment, where the source code is kept under wraps by a privileged few who decide what to leave in and what to leave out. New ideas spread slowly in those societies. And the development of effective security mechanisms that keep pace with the introduction of new threats can take years.

The open societies of this world, like the OSS community, are seemingly chaotic, but also brashly imaginative. Creativity abounds. Ideas for new technology, especially security technology, emerge at a frenetic pace. Open societies' ability to respond rapidly with innovative ideas ensures that no threat can remain a threat for long. Like the developers of the Unix password algorithm, their willingness to remain vulnerable will ultimately lead them to the greatest levels of security.

^[1]No, "ELV1S" is not a typo. The fourth letter really is a "1" in the CD name.

^[2]Raymond, Eric S., The Cathedral and the Bazaar: Musings on Linux and Open Source by an Accidental Revolutionary. Sebastopol, CA: O'Reilly & Associates, Inc., 2001.

^[3]John Terpstra, "Using the SNIA CIFS Benchmark Client to Test Samba Performance", CIFS 2002 Conference Proceedings, Santa Clara, CA

^[4]This is not meant to discredit those visionaries who act as catalysts in the open-source community. Good ideas have to come from somewhere, and some of these individuals are veritable fountains of originality.

^[5]See http://www.planettribes.com/allyourbase/

^[6]http://www.opensource.org/docs/peru_and_ms.php

^[7]Luke 6:38: "Give, and it will be given to you: a good measure, pressed down, shaken together, running over, will be poured into your lap. For the measure you use will be the measure you receive."