The Key to All Schedules: Milestones

Milestones are "pinch points" in your project where any outside observer can observe the progress of the development team and compare it with a list of milestone acceptance criteria. Depending on the nature of the milestone, certain development practices change to ensure the team is on track. Every milestone is defined in the following way:

• A title: Sometimes a keyword or short phrase describing the major accomplishment of the milestone.

• A short description: Usually a bullet list that exposes one level of detail about the tasks on the milestone.

• A due date: Usually on Wednesdays, to give the development and test teams a few days to tweak the build before the weekend, and spaced no less than five weeks apart and no more than nine weeks apart. They are never on holidays, either.

• Production dates: These are the dates the team focuses on production, as opposed to acceptance. Usually the production time is 80% of the acceptance time.

• A non-subjective test for acceptance: This is sometimes called the "Acceptance Criteria." Non-subjective means that the test is worded in such a manner that there is minimal room for opinion or interpretation regarding acceptance. This is the meat and potatoes of the definition of the milestone, and therefore your schedule. I'll have much more to say about this subject shortly.

• A risk assessment: A few words on the risk level of the milestone and what steps are being taken to mitigate that risk.

• Deliverables: What exactly is going to be delivered to the test team (documents only, documents and a working prototype, a release candidate, and so on).

• Contributors: A list of everyone expected to contribute to the milestone and a short description of their contribution (only one liners—a detailed individual milestone document will be created too).

The collection of all the milestones in one document is usually an integral part of any publishing contract, and is a serious commitment from the development team to the publisher.

Best Practice

Creating an internal milestone document for the team's eyes only and another document for external publication is a wise approach. Lies and deceit? Of course not! The internal milestone document will be 15% to 20% more aggressive than the external document, and may include things that the team wants in addition to the contractual deliverables, or simply has some tasks scheduled to complete sooner than they'll be expected by the publisher. This allows a development team a little wiggle room when unplanned events affect your game plan. And honestly, don't believe the publisher doesn't know you're doing this to begin with!

Writing a good milestone description is as difficult as writing an iron clad contract. The best way to prove this statement is to show an example of the same milestone description written poorly and written well.

Example A:

Physics System Integrated: The new physics library from Havok has been fully integrated into the game allowing objects to move and react under the effects of gravity, friction, collisions, and finally lie at rest.

Example B:

Physics System Integrated: The following can be demonstrated: The new physics library from Havok can be compiled and linked into the game with no warnings or errors. Havok's movement and orientation data are correctly translated to our custom renderer "SqueakMaster," our 3DStudio Max plugins work with Havok's exporter to provide a seamless export pipeline, a test script can "throw" all our dynamic objects into various map level areas with natural looking gravity, collisions, different coefficients of friction, and objects can lie at rest in a pile of other objects. All of this is done within the performance characteristics required in the technical specification and the object densities expected in the design specification.

The milestone description in Example A looked pretty good at first because it described what was supposed to be accomplished. The problem with it was that it was still vague. The phrase "fully integrated into the game pipeline" doesn't explicitly say anything about making the artists jobs easier by writing a 3DStudio Max plug-in, does it?

If you've ever heard anyone say, "I didn't know I was supposed to do that" or perhaps you've even said it yourself I'll bet my bottom dollar that the source of the communications problem was a bad milestone description. Two people read something that sounded a lot like the first example and read two completely different things. What usually follows is a clarification of the task very late in the milestone and the programmer is the one that gets nailed for weekend time.

A Tale from the Pixel Mines

After Martian Dreams shipped in March of 1991, I was asked to spend some time working on Ultima VII. I took a look at my assignment, which was explained as "User Interface," and scheduled it for six weeks. The infamous "Gump" system, as it was called, had already been written by Zack Simpson so it wasn't unreasonable to think that I could knock out a series of dialog boxes for the game in six weeks. I didn't finish my work until nearly twelve months later. The problem was that much of the code that manipulated the data structures or actions controlled by the buttons and so on hadn't been written or even considered. In truth, the task was much larger than everyone expected.

Let's look at one more example:

Example A:

Special Rendering Effects—Sun Effect & Light Glare: The code for the sun is capable of showing a fake sun in the sky that looks real when the camera view is pointed towards the sun during the day. At night or twilight all lights have a glare that approximates camera lens effects.

Example B:

Special Rendering Effects—Sun Effect & Light Glare: The API for the sun effects must be able to recreate the actual sun location in the sky given a date, time, latitude and longitude, temperature of sun surface and relative size when compared to Sol. The setting sun must change color appropriately as specified in the design spec. The effects expected when the camera is pointed in the sun's direction is a general brightening of the view until pointed directly at the sun, which should wash everything out. If the sun is partially occluded, as when standing the shadow of a building, the light from the sun must appear to peek around the edges of the building. Light glares must be created by the artist using 3DStudio Max user data and exported along with models that have lights attached, such as street lamps or vehicles.

Once again you can see the difference between the two examples. The second example does a much better job explaining what the technology is supposed to accomplish, where the data comes from, and how it is created or interacts with production tools.

Best Practice

It's fun to be creative when designing technology. That creativity can lead you down the wrong path, though, if someone hands you a task description that looks more like the shorter examples that leave too much room for interpretation. One of the differences between good programmers and great programmers is that great programmers always clarify the nature of the work before they begin, and they always predict their expected effort with a high accuracy. Make sure that every milestone description on your project is as detailed as it can be, and leaves little room for error.

Stock Milestones

There are some milestones that are "stock," that is they happen for every game project whether you plan for them or not! Go ahead and admit they exist, plan for them properly, and you'll be better for it. Here they are:

• Project Kickoff

• Design and Functional Specification Draft

• First Production Milestone, Second Production Milestone, and so on

• Code Complete

• Content Complete

• Zero Bugs

• Release to Manufacturing (or Web)

Project Kickoff

The project kickoff represents the first day anyone is officially assigned to work on project tasks. Generally, an awful lot of planning and conference calls have already happened and everyone is confident enough to start spending real money on the project. The following things should be deliverables for the project kickoff:

• A summary of the functional specification and scope of the work.

• A list of primary contributors (team management, tech leads, and so on).

• A good faith estimate of the timeline and budget.

• A list of project constraints and priorities (time, budget, features, competition, and so on).

• For external developers, a first draft of the contract. (No one wants to see a deal breaker pop up in a contract after significant work has been accomplished.)

Gotcha

Don't do deals on a handshake. Don't give in because someone's legal department is backlogged. I've seen too many projects (one of my own, too) get two or three milestones under their belt before anyone realized there was a deal breaker clause in the contract. No one wins in that scenario; at least get an LOI (letter of intent) with reasonable termination language that protects you for any expenditures you'll incur while the contract is winding it's way through the legal department. In other words, "lawyer up" and cover your butt—there's simply too much money at stake.

During this milestone the team is getting its feet wet and getting their head around the plan. Any outstanding research projects (crazy rendering techniques or bleeding edge AI) should be underway and everyone should have high confidence in either the outcome of the research or the viability of the backup plan. Lots of specification writing is accomplished and the final milestones, schedule, and budget are finished.

"But wait a damn minute," you say. How can you finish the milestones, schedule, and budget before the final specifications are complete? More lies and deceit? Again, of course not. The general parameters of any project (time, money, and features) are known to some degree before the project begins. Producers are given budgets, manufacturing slots, and marketing deals that constrain the time window when a project can be released, and your feature set must compare well with your competition. The trick is to assemble the right team and the right solution strategy to succeed within these constraints. The project schedule and milestones are proof that this difficult task is actually possible.

Design and Functional Specification Draft

The main deliverable for this milestone is documentation. The entire product specification (user interface, features, level design, character design, missions, and so on) is described in all its glory. This documentation accomplishes a few important goals:

• The test team will use it to construct the test plan.

• The development and production groups use it as a bible to answer the question: Are we making the product we agreed to make?

• Since every detail of the final product is described, it can be cross referenced against the inventory of scheduled tasks and therefore increase everyone's confidence in the schedule.

Hold on though, how can you create a complete and detailed product specification for a game? Games are closer to art than science and you can't really insert a three week task assigned to Roofus McDoofus that says "Increase the Fun to 100%." That's very true: You can't specify fun as a task. So what do you do? Mostly, you make sure the development team is experienced enough to detect the fun factor in the design instead of attempting to retrofit it into the game late in the schedule.

The deliverables for this milestone include:

• The entire design and functional specification.

• Logistics document (coding practices, asset management plan, team meetings, disaster recovery plans, and so on).

• Buildable "Hello World" project with the Source Code Repository.

• An inventory of all external licensed materials: technology, fonts, sounds, and so on.

An important thing to note about the documents you create in this milestone is that they are drafts, to be further modified and corrected until the delivery of the first production milestone, usually six to eight weeks later. You should expect to change no more than 15% of the scope of the documents or your plan as the corrections are made. Mostly, you'll simply reduce the scope of the product since the timeline and budget will be difficult or impossible to change.

First Production Milestone, Second Production Milestone...

You and your team implement your entire development plan during these milestones. You'll want to be more aggressive in your planning at the beginning to kick-start the team into their best intensity. Don't forget that each of these milestones will take extra time—about one week for every four weeks of solid development—for the entire team to fix bugs and get the product to meet each and every item on the acceptance criteria. When you plan for these milestones, make sure that each team member has something to contribute. It's a risky thing to allow anyone on the team to go more than one milestone in between rigorous and public checks on their progress. Annoying middle manager types (sadly I consider myself exactly that) can push all they want, but nothing motivates someone like the prospect of letting their team down.

It's also a good idea to keep these milestones coming at a reasonable pace. Five weeks should be an absolute minimum. This leaves four weeks for development and one week for acceptance. Four weeks is a pretty short time for a development team to accomplish something truly worthy of spending the testing group's time on another acceptance pass. You shouldn't plan any milestone longer than nine weeks, either. Any longer and you run a significant risk of missing opportunities to make a course correction in your schedule.

Best Practice

You should expect acceptance on any milestone to take about 20% of the actual development time.

A Tale from the Pixel Mines

The second Microsoft Casino revision was planned with actual milestone dates only four weeks apart with no special time set aside for acceptance. This was a real mistake because by the third production milestone the team couldn't keep performing on both their task load and the steadily climbing bug count. Microsoft execs were getting reports of a project out of control and skyrocketing bug counts, and they were very worried about our ability to pull it off. We did make our ship date, but after way more overtime and lost weekends than I had planned on. Also, I believe the quality of the product suffered because there were some low priority bugs that were postponed that would have made a noticeable difference in the product.

The deliverables for each production milestone should be:

• Two builds of the product: a release build and a debug build.

• Complete source code tree that can reproduce the build.

• Release notes and test data for the test team.

• The most recent product documentation, including the milestone acceptance criteria agreed upon by the test and development team.

Getting a release build working early in the project might seem onerous, especially since it takes valuable time away from implementing. Debugging release builds can be a real pain, further reducing any motivation that might exist. One thing is sure: The time and effort it takes to get a release build running again once it goes into limbo increases linearly with time. It's therefore a good idea to make every milestone dependant on a release build passing acceptance tests.

Whether or not you supply a complete "rebuildable" source code tree with your milestone submission is up to you, your publisher, and sometimes the lawyers. All the same, it's a good idea to create it and test it as a part of your milestone acceptance. Think of it as an insurance policy against a catastrophic failure of your network or some other similar disaster.

A Tale from the Pixel Mines

While I was at Compulsive Development we were working hard on two projects for Microsoft. For those of you who live in or around Austin, Texas you're already familiar with the violent springtime storms that have tornados, golf ball sized hail, and tons of rain. One afternoon a particularly large storm hit our area and dumped six to seven inches of rain in just a few hours, flooding a creekbed just down the street from our office. One of our employees, Ellen Hobbs, noticed a flowing stream of water entering her office and shortly thereafter the server room next door! We stopped everything to safeguard our computers, furniture, servers, and toys by moving them to other offices. By the end of the day water had seeped into almost every office, and we could float little boats in Ellen's swamp. Nothing had been damaged except for some waterlogged books (sorry about that Donavon!), but it could have been much worse. Well, the smell couldn't have been worse! Hot Texas weather and wet carpet don't make for the best software development environment.

Best Practice

If you include a rebuildable source code tree with each build, make sure that your network security limits access to the build machine. If anyone can take a blank CD and burn a copy of your entire build, game libraries and everything else it might walk right out of your door and you'll never be the wiser. I hate to think of the fact that people might be enticed to steal everything your team worked so hard to create, but face it, your game source code is a valuable thing and you should protect it.

Code Complete

The code complete milestone is the most important milestone beside the last one. Once this milestone passes the acceptance tests the programmers shift their effort from writing and engineering new code to fixing code. This might sound earth shattering but it is the first time in the projects life cycle that you can accurately predict the ship date of your product.

Microsoft has found that on average a programmer can reduce their bug debt between three and four bugs per day. That may not sound like a lot, but remember that while a programmer may fix eight to ten bugs per day, the test team is writing up new issues at a similar rate. After code complete the production staff will begin to postpone minor issues in favor of fixing more important ones.

The product should pass the hardware and performance tests as well. This might sound unreasonable at first but take a moment to consider carefully what "Code Complete" really means. If the product fails significant portions of the hardware or performance tests, it is quite likely that a significant body of code is missing.

The deliverables for the code complete milestone should be exactly the same as the production milestones, with the addition of the following:

• Proof that the product passes hardware and performance tests, especially on the minimum machine configuration.

• A permanent and complete archive of everything: source code repository, art and sound sources, and so on.

• A report of every change made to the source code since the last submitted build.

• The install program.

The permanent archive of code and media assets is a good idea at this point. That includes original sound WAV files, models and scenes from Max or Maya, and anything artists used to create anything in the game. The closer you get to the end of the project, it's a good idea to spend some time and effort on ensuring that even terrible events have a minimal effect on your ship date.

Regarding the detailed change report for every build, most source code repositories have command line mechanisms for creating the change report. Every Microsoft project I worked on required this report, and at first glance this requirement seems a little draconian. There's a good reason behind it, though. Late in the development cycle the development team is constantly tempted to make little last minute changes without involving the production or testing group. Hidden changes drastically increase the risk of introducing problems because they don't get tested properly. If the team commits to creating the change report for every submitted build, it will make sure that every change goes through the right channels: development, production, and test.

There's another reason the code complete milestone is important. It marks the first major change in not only what the programmers are doing, but how they should do it. The changes in their work habits are all geared to reducing risk in the schedule:

• Nothing changes in source code without an approved bug entered into the bug database.

• Every change is bracketed with comments that identify the bug number, the rationale for the change, and anything else that may be needed to answer questions about the change.

• Every change is code reviewed before it is checked in.

These additional work habits serve to double and triple check every change and solution strategy implemented by each member of the team. Doing this late in the project is important for everyone on the team because the work hours and stress have peaked. It's quite easy for someone to believe that a particular solution or change is the right way to go when it might actually set the project back.

Best Practice

The oddest thing about programmers working in a team is that the intelligence of any programmer is instantly tripled when a respected colleague is silently looking over their shoulder. The colleague in question is usually concentrating on their shopping list or some other inane thing while the person at the keyboard is explaining their problem in detail and at the same time solving it with a sudden unexplainable streak of brilliant thinking. I've been on both sides of this phenomenon. Countless times I've run up against some coding issue and I called a friend of mine into my office to have them help. As I explained the problem the solution presented itself. Laughing, they usually leave my office telling me how they are glad they could help.

The point of all this is to realize that the reviews and reports aren't some kind of Orwellian scheme to stifle the thinking and creativeness of programmers. This is a common misconception and is a normal first reaction to these ideas. Instead, consider the possibility that when individuals begin to rely on the strength of the team everyone benefits from reducing errors and in turn makes a better product.

Best Practice

Rely on the strength of the entire team by participating and encouraging code reviews, and by following rigorous development practices that somehow seem more at home at NASA. These development practices are useful for any kind of software, games included. If more teams relied on good habits, as well as raw talent, more games would ship on time and they'd be more fun.

Content Complete

Content complete is really a milestone that is pretty special to game development. This stage focuses on completing and installing all the other game data: sound effects, speech, music, map levels, help, and anything else that isn't a part of the executable. Content complete usually follows four to eight weeks after the code complete milestone, but it could take much longer depending on your project. Most projects I've worked on spent about 50% of the overall development effort on programming, 40% on art, and the rest on sound or other assets like help. Of course, your mileage will vary.

You can image a project like Myst spent their development time very differently. The technology effort might have been as little as 20% of the man-hours, meaning that their development team might have reached code complete very early. The artists were probably working on vast modeling, lighting, and rendering tasks long after the programmers finished the display and user interface systems. In that kind of project you would expect the code complete and content complete milestones to be separated by one or more implementation milestones.

The deliverables for the content complete milestone should be exactly the same as the code complete milestones, with the addition of the following:

• Proof that the full scope of the media fits onto the CD-ROM with enough room for extra material required by the publisher.

• Signed license agreements for any third-party materials such as fonts, sound effects libraries, licensed technologies such as renderers and physics engines.

• Proof that every piece of media (art, sounds, and speech) used in the product is either in the public domain, created from scratch, or is accompanied by the appropriate permissions from the owner.

One of the things that happen during content complete is a steady stream of media assets are installed on a daily basis. The size of the product will grow quickly during this phase, and should therefore be monitored closely. Your team shouldn't wait to find out about a bulky product from the test team, because solving it at that point is distracting task for a programmer.

Gotcha

Don't assume that you'll get 100% of the CD-ROM space for your product. Publishers will usually require 50Mb or more for the install program, the latest version of DirectX, and perhaps some marketing materials for other products. This is a good thing to clear up before you finalize the contract, and any costs you incur due to a change in this plan should be borne by the publisher.

Another big issue that's forgotten about regarding intellectual property in games these days is the proper use of fonts. Arguably, it is against the letter and spirit of copyright law to duplicate a font exactly into a bitmap form and use it in your game without licensing it. Some font companies will require you to pay huge license fees for using their fonts interactively, which would include the display of letters in an edit control, for example. Have someone check the terms of a font license to be sure.

Legally you also can't use a model of a Ferrari in a game without permission. This is especially true for company logos, as it may be seen by the company in question to be an illegal use of their intellectual property. Someone who knows the intricacies of the law should check the question of what falls into fair use of trademarked or copyrighted material. The last thing you need is to have your game pulled off the shelves because one of your more colorful characters wore a "Have a Coke and a Smile" t-shirt.

Gotcha

Programmers and artists are free spirits, unencumbered by little things like the law. It can be fun to put a picture of Mickey Mouse in your game somewhere, perhaps as a joke or even as a serious design idea. Different companies have different tolerances for this kind of thing, and Disney is well known for their relentless pursuit of anyone using their intellectual property without their permission. Bottom line: Be careful what goes into your game, and unless you work for Disney don't even think of using Mickey.

Zero Bugs

From the moment you have a code complete build, the programmers are working 100% on fixing approved bugs. After content complete the entire team is working towards this goal. The milestone is reached the moment the team submits a build to the test team with every active bug addressed. Depending on the size of your product, development team, and bug count, you can expect the timeline from content complete to zero bugs to be no less than four weeks. A good rule of thumb would be two weeks for every three months of development time.

It is technically possible that this build could pass every test that the test team has in its arsenal. This means that your zero bug build is actually the first release candidate. Clearly this means that as the team gets close, less than five bugs per developer, the development practices get extremely tight. Every change made to fix a bug risks breaking three other things. There's no escaping the fact that at least once during the march to zero bugs, someone on the team will make an innocent change that will have unexpected and serious results.

A Tale from the Pixel Mines

On Ultima VII I was asked to make a small change in a text string. I can't honestly remember the exact change but I'm pretty sure it had something to do with Sherry the Mouse. I made the change myself and submitted the code for the next build. Ultima VII didn't have a robust way for me to jump ahead to the point in the game where the change could be tested, so I had to hope for the best. The change overran a static buffer used to compose conversation strings, and the game crashed after QA had spent six hours playtesting the game. They weren't very happy about that. The next time I made the change it was witnessed by no fewer than five programmers and Richard "Lord British" Garriott himself. It worked the second time.

The deliverables for this milestone are the same as the content complete build, and remain the same until the final build.

Release to Manufacturing (or Web)

Every build from here on out has zero active bugs. A build is sent to test and a number of issues will show up, most of which will get postponed or the team will decide to leave it, since the risk of fixing it is much worse than letting the problem remain in the game. At some point, only one bug will pass muster and it will be the only change made before a new build is sent to test. You should plan on spending at least four weeks on this final milestone.

The test team will be hammered in this milestone, and it isn't unreasonable for the development team to help out. The team should have very little work to do and can even spend some time thinking about the next project. They could design changes to their game engine for the next version (surely you wouldn't consider actually making core changes to the engine code now), or work short hour days to catch up on some laundry or reintroduce themselves to their family.

Sometimes you might consider making a simple change without performing the entire build process. A good example of this might include a small text change to a help file or the credits. I'm steadfastly against this practice since it introduces a variable in the development process for what could be the final build. Every time I've seen this happen something important is forgotten.

Instead, go ahead and make the change as you would normally, perform the entire build process, and compare the results to the in situ change file by file. If they end up exactly the same, good for you. If they are different you should figure out why the difference exists and question the wisdom of creating a version outside of your normal build process.

When the last bug is fixed, get the whole team to look at it. Make an event out of it, even though you might make this event happen two or three times before your release candidate is accepted. In addition to the network archive of the source tree and all media assets, which you've been doing for every submitted build since content complete, you should also archive the hard drives of the build machine. This will record the exact build environment, operating system version, and service packs included for later use. It isn't uncommon for publishers to request all of the backup tapes when the final build is submitted. Of course, don't forget to keep a copy for yourself.

A Tale from the Pixel Mines

Compulsive Development, at the time a 12-person company, had just finished our Bicycle Casino project. We sent the backup tapes to Microsoft made on a brand new Dell multitape backup system. Microsoft has a central group responsible for testing and archiving every tape backup sent to them by a third party. It turned out they didn't have the exact make and model of backup tape drives and archive software we used at Compulsive.

We got a phone call from our main contact there that we had to make backup tapes that were compatible with their system before our last milestone would be accepted! Since we purchased our server hardware from Dell, we weren't entirely motivated to spend thousands of dollars on new backup hardware. We got in touch with the archival group at Microsoft and convinced them that Microsoft would probably benefit from buying the hardware and software themselves, especially since the systems we were talking about weren't exactly cobbled together from spare parts lying about. Just because you are a small developer doesn't mean you can't reason with a multibillion dollar company!