Section 1.1. Founding of the Mozilla Organization: Obvious for Developers, a Bold Step for Management

1.1. Founding of the Mozilla Organization: Obvious for Developers, a Bold Step for Management

The Mozilla project originally grew out of Netscape Communications Corporation and its Netscape Communicator product. In early 1998, the Netscape management team made the decision to continue development of Netscape's flagship product, Netscape Communicator, through an open source development model. At the time, Netscape Communicator and Microsoft's Internet Explorer browser were locked in a fierce competitive battle often referred to as the "browser wars." Netscape's goal was to seed a broad-based development effort within the software development community to produce future browser products as a shared resource.

At its inception, the Mozilla project faced some paradoxes. First, the only people familiar enough with the code to participate actively in its development were Netscape employees. Those employees were still expected to work within the management system and practices that Netscape had developed in its proprietary days. There was no volunteer community. And yet, even at that early time, it was clear that the long-term success of the project required a broad constituency of people and companies working jointly on the project. It was not enough to have open source code (code available under an open source license). The project needed an open development process, and this required authority over the code's development to be based on technical merit and distributed outside Netscape. The question was how to get there from here.

One thing was clear: the success of the project depended on it being a real open source project. In other words, the project needed to have technical legitimacy and development decisions would need to be guided by technical considerations. This was intuitively clear to the group of Netscape employees who were familiar with open source, eager to help move the Mozilla code into the open source world and who ultimately became the founding members of the Mozilla Organization. This group made the need clear to Netscape management, which was receptive to trying to do the right thing.

When the Mozilla project was officially launched, Netscape executive management therefore took some bold steps. First, they officially anointed "mozilla.org" as the steward of the codebase and leader of the project. I say officially because it's quite possible that a group like mozilla.org would have developed even if Netscape hadn't officially helped to create one. But this step was important, as it allowed mozilla.org to focus on building the project rather than on proving the necessity of its role.

The creation of mozilla.org was a significant step that set the tenor of the project's development. It meant that the development of the Mozilla codebase was to be guided by something other than Netscape's own product and revenue plans, and also that Netscape management would need to give up control. This may seem like an obvious statement in an open source world, but it is one of the most difficult problems in moving from a proprietary to an open system. It is particularly difficult when the commercial vendor is actively trying to ship a product and the code has not yet reached a good, solid state that can serve as the basis of that product.

Some have said that the Mozilla project was not a true open source project during this time because Netscape employees contributed so much to the project and Netscape management was so involved for so long. It's possible this is true. But I believe that Netscape management lived in an intensely uncomfortable setting as control of the project moved from their hands into those of mozilla.org. And since I personally was the fulcrum for stresses between the project leadership and the Netscape management team, I'll warrant that Netscape management felt it was living in a real open source project. In 1999, Netscape Communications Corp. was acquired by America Online (AOL). This resulted in many changes, but the relationship between mozilla.org and the Netscape browser development group continued as before. For quite a while, we used the term Netscape/AOL to describe the Netscape browser development group after the AOL acquisition, and I'll use that phrase for the rest of this chapter.

The members of the Mozilla Organization are known as "mozilla.org staff." The original members of the 1998 launch were Netscape employees who had a vision for an open source Mozilla project and a determination to see it succeed. The most media-genic of these founders was Jamie Zawinski, who left the project after a year. But the most consistent and long-term contributor has been Brendan Eich, who was a founding member of mozilla.org and remains the technical and philosophical leader of the project to this day. Over time, the percentage of mozilla.org staff employed by Netscape decreased steadily. Today, the mozilla.org staff does not have any Netscape/AOL employees.

After the Mozilla project was launched, mozilla.org staff members began the process of changing development styles from a proprietary to an open source model. The early steps were logistical: establish public communications channels such as mailing lists and newsgroups; establish a public system for viewing and tracking bugs. A harder task was changing habits. For example, the existence of public communications channels was not enough. Old habits die hard, and there was a tendency for people to use the methods they had always used. This was complicated by the fact that Netscape as a company still had confidential data about itself and its business partners that couldn't go into public forums. So, it was not possible to eliminate all private channels. Eventually we changed the names of any remaining private mailing lists to something long and awkward that required conscious thought to use. This gave Netscape/AOL employees a way to disseminate confidential data when necessary, but made public disclosure the easiest path.

Even as basic a step as public communications in an open source project can be difficult for some management teams to accept. In a system that is public by default, everyone needs to learn what information must remain confidential, and to remember this while working. At first it's a big effort to work in public and some people see it as overhead. Then as the project progresses and the public interaction provides increasing value, the need to keep something private is seen as a burden. This is certainly the case for the Mozilla project today, where the only private development information we solicit are bugs which could have an impact on the security features of our products. We've set up a system for treating these bugs privately, and the system has overhead. We bear it in the security context because security is critical, but we avoid it in other contexts.

We also set up a public bug and issue-tracking system. This is commonplace today, but was innovative at the time. We made the bug-tracking system an open source project under the Mozilla umbrella, and today Bugzilla is a successful project in its own right (http://www.bugzilla.org). We also set up a continuous build system and web frontend (http://tinderbox.mozilla.org/showbuilds.cgi). This means that we have an automated process that builds and rebuilds the software continuously on multiple platforms to see if and when a new piece of code causes the software not to build. Then came a period of learning to "work in the fishbowl." Some people adapt easily to having all their work visible, and others struggle. Many simply walk down the hallway to talk with a buddy, and then forget to tell everyone else. This period takes some time and effort.

1.1.1. Updating the Codebase

About six months into the project, it became clear that the codebase was in need of updating. By late 1998, the inherited code was several generations old, had been patched over and over, and actually hindered ongoing innovation. Old and fragile, it looked backward toward the beginning of the Web, rather than forward to the new technologies a modern browser would need to support. So, in late 1998, a painful decision was made to rewrite the layout engine, a critical and complex core component.

This decision changed the scope of the project dramatically. The initial project was an incremental upgrade from the Netscape Communicator 4.x product line to a proposed 5.x product line. Moving to the new layout engine (known as Gecko) meant that the incremental model was gone; the Mozilla project would need to develop a complex new layout engine and then build a new browser application on top of it. And things got harder from there. As the new layout engine began to mature, it became clear that other significant parts of the codebase would also need rewriting. The development process turned out to be a lot like a remodeling project where fixing one problem leads to another. Then came the long, slow grind to producing something useful (Mozilla 1.0 in June 2002) and finally something great (Mozilla Firefox in November 2004).

During this time, many proclaimed us dead, a failure. What those people didn't see was the passion and commitment of the contributors to the project, including the Netscape employees. The contributors knew that they were developing good technology. They knew they had a shot at building a great browser and mail client. And they knew it mattered. The Web matters. Browsers matter. Much of the world decided that the days of browser innovation were over. Some mourned the loss of choice, and many didn't realize the dangers of accessing the World Wide Web only through a single access point. But the contributors to the Mozilla project realized both the danger and the potential for something innovative. They persevered. The prominence of Netscape often obscured the efforts and dedication of the individuals themselves. Yes, many contributors were paid by Netscape. Of these, many contributed far beyond the requirements of a job, doing extra work to make the product "theirs" and to make sure they were proud of it. Meanwhile, the individual volunteers provided critical expertise and contributions across the codebase.

During this period, almost all the code in the Mozilla browser and email client was rewritten. The focus was a modern layout engine, and a set of technologies designed to make the promise of cross-platform development a reality for the Web. We created a cross-platform component model (XPCOM),a cross-platform XML-based UI language known as XUL (pronounced zool), a new toolkit using XUL, and a set of cross-platform applications themselves. Developing these was a long process, but we felt that it was important to have technology that would help us move forward. The power of these technologies has been demonstrated through our new products: Mozilla Firefox and Mozilla Thunderbird, in which we were able to build award-winning cross-platform applications quickly on top of mature, preexisting infrastructure.

1.1.2. A Disciplined Methodology

Along the way, the Mozilla projects developed a highly disciplined method of distributed software development. Many people think that open source development is necessarily chaotic. Or they wonder about the quality of the code because anyone can create a patch and offer it for inclusion in the source base. Open source need not be chaotic, and the Mozilla project is not. For every piece of code checked into the Mozilla products, we track:

• Who checked it in

• When it was checked in (to the minute)

• What problem it was trying to address

• The complete history of the issue (bug) the code was trying to address

• Who did the code review (often two levels of review)

• Whether the next build of the software was broken on any of our main platforms

• Whether the code affected our performance metrics, by platform

• Build and optional log comments

• A comparison with the previous version of the code.

This information is available at any time; it does not require an expert to find or assemble the data. It is available online, in real time, through a web interface; all one needs is a web browser (http://tinderbox.mozilla.org/showbuilds.cgi). We do this so that many contributors can work on the same codebase simultaneously and know what's going on. We do this so that the source code "tree" stays healthy, and we address problems before more new code makes them worse. We also have policies determining who gets access to the CVS tree, what's required before code can be checked in, what to do when the tree doesn't compile and run, how authority is delegated to those with expertise, and so on.

1.1.3. Building an Open Source Project

The process of building software proceeded simultaneously with building a vibrant open source project. The creation of mozilla.org had gone hand in hand with the recognition that Netscape management would need to give up a great deal of control over the development process. Now it was time to figure out how to make the transfer. I describe in this section two of the most significant topics we addressedcontrol over the source code repository and control of the designated releasesin some detail, for I believe this shows how the Mozilla project came of age.

Implementing a transfer of control from Netscape management to mozilla.org caused a number of strains. Mozilla.org staff could have proceeded in opposition to Netscape. Indeed, we thought about it many times. However, Netscape was a large and valued contributor, whose involvement and work product remained very important for the project. So, we spent a great deal of energy figuring out techniques that addressed the concerns a commercial entity like Netscape brings to a project and simultaneously building a strong open source community.

Having control of the source code repository may seem like an obvious requirement for an open source project, but there were many sensitivities in our case. For example, the Mozilla project has processes to make sure that code is of good quality before it is checked in. This is often the case in open source projects, and many contributors understand that their code needs to meet project standards. However, this is not always the case in commercial settings. Often the employment decision is what matters. If someone is an employee, his code goes into the project he was hired to work on. Instituting code review for everyone, even people employed to contribute code, can be a surprise for new employees and for their managers. Suddenly the decision of what code goes into the source repository is not made by managers for a particular person through the employment process. Instead, the decision is made by engineers who review the code itself rather than the person's credibility or employment status. And the code review is not optional or auxiliary; code cannot be checked into the tree until an appropriate reviewer has given formal approval.

We instituted code review as a prerequisite to check-in relatively early in the process without too much controversy. Sometime later we implemented a second layer of review which we ended up calling "super-review." Super-review is an "integration review" or "plumbing review." Does the code conform to coding guidelines? Does it use the somewhat tricky XPCOM in appropriate places and not elsewhere? Does it conform to the overall architectural goals? Does it avoid needlessly diminishing modularity?

We instituted the super-review requirement because we felt we had to. The Mozilla codebase is large and complex, and we were worried about overall code quality. We needed to increase our confidence that the code we were accepting into the tree would solve the immediate problem, and still leave us enough flexibility for future development. Implementing the super-review was very painful for everyone involved. It was painful for the identified super-reviewers. Being a super-reviewer is actually a lousy jobwho wants to review yet more code instead of writing it oneself? And of course, the super-reviewers were well regarded and had plenty of work to do themselves. They took on the super-review job not because they wanted to, but because they believed it was important to making Mozilla the project we wanted it to be. Super-review was also painful for the contributing engineers. Adding another layer of review to the pre-check-in requirements was seen as "yet more bureaucracy" by some.

There were many requests for absolute, guaranteed turnaround times for super-review. Engineers and managers complained they could not schedule work accurately due to the unknown lags caused by waiting for super-review. And yet, the super-reviewers had to balance review of code with writing their own. And since they were well regarded, their code was acutely needed. I was unwilling to agree to rigid turnaround times for super-review, believing that doing so would put the work of key contributors at risk. We eventually agreed on a timeframe for an initial response from the super-reviewer, which would contain some estimated time for full review.

Another issue was determining who has "write-access" to the source code repository. This was extremely sensitive. In a commercial setting, it is often the case that when people are hired they are given access to the part of the source code repository to which they are expected to contribute. In open source settings, one typically earns access by making valuable contributions. Moving the open source standard of "earn the right to touch the tree" to employees who need to contribute code to do their job can be difficult. In particular, the rationale for why this is necessary can be hard to explain in a commercial setting when all the code an employee writes must be reviewed and super-reviewed before it can be checked in. The question comes up: assume Employee X is hired. She writes code, and it passes review and super-review. Why on earth can that employee not go through the mechanical task of actually checking the code in? I was never able to provide a complete answer to this question. I know that open source projects regularly vet people before allowing access to the source code repository. And I know it would be very odd for a management chain in a company to make the decision about CVS access. So it's clear that this "is just not done." But I was not able to explain clearly how someone could do damage by having CVS access if all of her code was reviewed and approved before check-in, even though the technical leadership of the project felt very strongly about this. The idea of automatic access for employees had an emotional response because open source projects rely on peer review and technical leadership, and I shared this. But the key engineers were adamant that the quality of the code would suffer through automatic access, even though our pre-check-in code review requirements are quite stringent.

It was an awkward setting to institute a policy for which I couldn't give a crisp reply to the various management teams affected. (I've been responsible for policy for mozilla.org since 1999, so it was my job to write the policy, describe it to management, and address the concerns and complaints that might come up.) And the possibility existed that management groups would be surprised, distressed, or outraged that such a policy would be instituted without a clear answer as to why the code quality wasn't adequately protected. Nevertheless, we instituted this policy in 2001.

A second area where shifting control to mozilla.org was highly sensitive concerned control of the milestone releases. By "owning the releases," I mean several things: first, defining and implementing a planned milestone schedule; second, defining and implementing a process for getting a release into shipping condition; and finally, identifying and shipping the release.

When the Mozilla project was launched, the planning and release schedule was determined and implemented by Netscape employees contributing to the project, but not directly by mozilla.org staff. We worked on changing this for quite some time before we had proven ourselves trustworthy enough for Netscape to give up control. This may seem laughablehow can a so-called "open source" project not control its own releases? But it's important to remember that Netscape was under enormous pressure to release a product, and giving up control of the process by which releases are made is extremely uncomfortable. Mozilla.org was new and unproven. And of course, managing a software project with hundreds of people working on it is not easy in any setting.

Control of these aspects moved formally from the Netscape management to the mozilla.org staff following the release of the Netscape 6 browser. This shift had been discussed for some time but still involved a leap of faith for those who had previously exercised decision-making power.

This was a tempestuous time; mozilla.org staff had influence, but not control. We thought many times about whether we needed to create a fork in order to affect such a shift in control. Each time, the mozilla.org staff decided that Netscape's contributions were far too important to the project and outweighed the desire for open source purity or credibility. I suspect that the Netscape management team must have had similar discussions, weighing a fork and the ability to manage their releases as they felt best with the value gained from the open source project. In the end, we all hung in and mozilla.org staff became the official keeper of our releases after Netscape 6.

Owning our releases did not mean that we ignored Netscape. Netscape remained the largest single contributor, and its pool of talented and dedicated engineers was boggling. Netscape was also the largest single distributor of Mozilla-based products. Managing the project without taking Netscape's needs into account would have been stupid. By this time, Netscape didn't exercise control as it once had, but its leadership role in the project was greater than some might have liked. Mozilla.org staff continued to hear from the community that we were Netscape stooges and there wasn't a "real" open source project. However, to my mind, Netscape's role was now determined by classic open source principles: leadership and influence through the quality of one's contribution. Of course, this involves an acceptance of the role of a corporate entity in an open source project, which made some uncomfortable.

Mozilla.org was able to own our releases well, in part because we had developed an active, effective quality assurance (QA) community. Over the years, I've learned how few people intuitively grasp the importance of the QA effort (and I suspect, how important QA will be to other projects of similar scope). Both of our major effortsweb browsing and emaillive in an extremely complex world. The Web is very diverse, and people use our software in a boggling array of environments and in wildly different ways. Hiring a QA team as full-time testers is part of a solution, but it is not the complete answer. I'm becoming more and more convinced that it may well be impossible to hire a QA team big enough and diverse enough to do thorough testing of a product like a web browser.

By 1999, it had become apparent that an active community of people was interested in contributing to our testing and QA effort. Christine Beagle joined us to lead an effort at mozilla.org to make this group cohesive, figure out ways to give the group some authority, and encourage them to step forward. The response to a bit of attention and appreciation directed at this nascent community was astounding. One mark of success is that shortly thereafter, we hired one of the more active and organized of these folks as our community QA lead. This person was Asa Dotzler, who has been a key figure in the project ever since, still coordinates QA activities, and is extremely active in managing our release process. With Asa's coordination, we began to see a set of people doing more organized testing of our products. This provided enormous value. The testers also did things such as look at all the bugs assigned to a particular engineer, verify that the bugs were legitimate, check to see if the bug existed on all platforms, create test cases, and then verify fixes across our main platforms. These efforts saved enormous amounts of time for the engineers trying to write the code to fix the bugs. I've learned that this type of work often gets little respect, but we value it highly and find these contributors to be extremely important to our project. Many of these efforts were coordinated through mozillaZine, an independent webzine dedicated to the Mozilla project that was founded by Chris Nelson in September 1998 (http://www.mozillazine.org).

Massive community testing remains important today. We provide Release Candidates for our major product releases precisely so we can get 50,000-100,000 downloads from our key community and get a good reach on quality.