Finishing Issues

If your project is going well, you'll likely only need a few tweaks here and there to make sure you "stick the landing," so to speak. You can recognize this on your project by looking for a few telltale signs:

• Your bug count is under control. Your developers have less than 4 active bugs to fix per day until the "zero bugs" date.

• Everyone is in good spirits.

• The game is fun to play, performs well, and has only polishing issues remaining.

If this describes your project, congratulations! Don't get too cocky, because there are some easy missteps you can make, even at this late stage.

Quality

Perhaps the two biggest questions you and everyone else on the team asks at this point are likely to be, "Is the game good enough? Is it fun?" If a bug comes out of the testing group, it's because they want something changed to make the game better. Anyone on the development team can motivate a change as well, and they should if they think the game will become better because of it.

The closer the project gets to the scheduled zero bugs milestone, the less likely a bug will actually get fixed. This rule of thumb is directly related to the fact that any change in content or code induces some risk. I've seen a single bug fix create multiple new bugs. This implies that any high-risk change should either happen much earlier in the schedule, or there has to be some incredibly compelling reason, like there's no other choice, and the project is in jeopardy if the change isn't made.

Everyone on a project has his or her pet feature, something he or she really wants to see in the game. The best time to install these features is before the code complete milestone. There are a few good reasons for this. First, it gives the team a huge burst of energy. Everyone is working on their top tier wish lists, and tons of important features make it into the game at a time where the risk of these changes is pretty tolerable. Second, it gives the team a message: either put your change in now or forever hold your peace. After code complete, nothing new code-wise should be installed into the game. For artists and other content folks, this rule is the same, just the milestone is different. They use the content complete milestone as their drop-dead date for pet features.

One more note about programmers and artists adding anything: If the game isn't reaching target performance goals, it's a bad idea to add anything. More code and more art is unlikely to add any speed to the game. Make sure the performance issues are completely taken care of before code complete, and monitor it closely until the project ships.

A Tale from the Pixel Mines

It's a common practice to put inside jokes or "Easter Eggs" into a game. On Ultima VII, the team installed a special way to kill Lord British, especially since Richard Garriott wanted Lord British to be completely invincible. You need a little background first:

Origin was in an office building in the west Austin hill country, and the building had those glass front doors secured with powerful magnets at the top of the door. One day, Richard and some other folks were headed out to lunch, and when Richard opened the door the large block of metal that formed a part of the magnetic lock somehow became detached from the glass and fell right on Richard's head. Lord British must truly be unkillable, because that metal block weighed something like 10 pounds and had sharp edges.…

The team decided to use that event as an inside way to kill the monarch or Britannia. At noon, the Lord British character's schedule took him into the courtyard of the castle. He would pause briefly under the doorway, right under a brass plaque that read, "Lord British's Throne Room." If you double-clicked the sign, it would fall on his head, and kill him straightaway.

Perhaps the weirdest thing about this story is that a few weeks later the same metal block on the same door fell on Richard a second time, again with no permanent damage. The guy is truly protected by some supernatural force, but he did buy a hard-shell construction helmet, and he wasn't so speedy to be the first person to open the door anymore.

By the time the team is working solidly to zero bugs, all the code and content is installed and there is nothing to do but fix bugs. It's a good idea to add a few steps to the bug fixing protocol. Here's the usual way bugs get assigned and fixed:

1. A bug is written up in test, and assigned to a team member to fix.

2. The bug is fixed, and is sent back to test for verification.

3. The bug is closed when someone in test gets a new version and observes the game behaving properly.

Close to the zero bug date, a bit of sanity checking is in order. This sanity checking puts some safety valves on the scope of any change. By this time in the project it usually takes two overworked human brains to equal the thinking power of a normal brain:

4. A bug is written up in test, and discussed in a small group—usually the team leads.

5. If the bug is serious enough, it is assigned to someone on the team to investigate a solution.

6. Someone investigates a potential solution. If a solution seems too risky they pull the plug then and there and report back that the bug should remain in its natural habitat.

7. The solution is coded, and checked on the programmer's machine by a colleague.

8. The solution is presented to the leads, and a final decision to check in the code or abandon the change is made.

9. The bug is sent back to test for verification.

10. The bug is closed when someone in test gets a new version and observes the game behaving properly.

If you think that the bureaucracy is a little out of control I'd understand. It does seem out of control, but it's out of control for a reason. Most bugs, about 70% to 80%, never make it out of step #1. Of those that remain, one-half to three-quarters of those are deemed too risky to fix, and never make it out of step #4.

A Tale from the Pixel Mines

My first experience with bugs in games was on Martian Dreams at Origin Systems. The whole team would gather in the conference room and each new bug from test was read aloud to the whole team. Sometimes the bugs would be so funny the whole room was paralyzed with laughter, and while it wasn't the most efficient way to run a meeting it sure took the edge off the day.

On Ultima VII, Ultima VIII, and Ultima Online, the teams were simply too big, and the bugs too numerous to read aloud in a team meeting. Between the inevitable laughter and complaining about fixing the object lists again we'd probably still be working on those games.

Even on smaller projects, like Bicycle Casino and even Magnadoodle we held bug meetings with the team leads. It turned out that the rest of the developers would rather spend their time making the game better and fixing as many bugs as they could, rather than sitting in meetings. Outside of that, time away from the computer and sleep was a great diversion.

Of course, everything hinges on your active bug count. If you are two months away from your scheduled zero bug date, and you are already sitting at zero bugs (yeah, right!) then you have more options than a team skidding into their zero bug date with a high bug count. I hope you find yourself in that situation someday. I've never seen it myself.

The only hard and fast rule is how many bugs your team can fix per day—somewhere between three and four bugs per person. Of course, your mileage may vary, but probably not by much. If your team is over this limit, or even a single person is over this limit, start getting conservative on approving bug fixes. If things are looking good, loosen the screws a little and make your game better while you can.

Code

At the end of every game project, the programmers are the ones who are hammered the most. Artists, level builders, and audio are hit especially hard during the content complete milestone, but after that their work levels off, mostly because it is usually more predictable. If you don't believe me, just ask an artist how long it will take him to tweak the lighting on a model. Or ask a level designer how long it will take to place a few more power-ups in a level, and he will not only give you a solid answer, he will be right about it.

Ask a programmer how long it will take to find the random memory trasher bug and he will shrug and say something like, "I don't, know—a few hours maybe?" You may find that same programmer, 48 hours later, bashing his head against the same bug, no closer to fixing it than when he started.

These setbacks happen all the time, and there's not much to do about it but get as much caffeine into their bloodstream as they can stand, get the other programmers to take up the slack in the bug debt, and maybe lend a few more neurons to the problem. Don't forget about the advice earlier in the book: Any two programmers looking at the same problem are at least three times as smart as a lone programmer.

When the bug is eventually found, there is sometimes a decision that has to be made about the nature of the solution. Sometimes a simple hack will suffice, but a "real" solution exists that will touch a lot of code and perhaps induce more risk. At the very late stages of a project, I suggest hacking. Wonton, unabashed, hacking.

Some of you may be reeling at this sacrilege, but I'm sure as many of you are cheering. The fact is that a well thought out hack can be the best choice, especially if you can guarantee the safety and correctness of the change. "Hack" is probably a bad word to use to fully describe what I'm talking about, because is has somewhat negative connotations. Let me try to be specific in my definition:

• Hack - n. A piece of code written to solve a specific corner case of a specific problem, as opposed to code written to solve a problem in the general case.

Let me put this in a different light: Everyone should be familiar with searching algorithms, where the choice of a particular search can achieve a "first solution" or a "best solution" criteria. At the beginning of a project, coding almost always follows the "best solution" path, because there is sufficient time to code a more complicated, albeit more general algorithm. At the end of the project, it is frequently the case that the best solution would lead a programmer down a complete reorganization of an entire subsystem, if not the entire code base.

Instead, games have a "get out of jail free" card, because the players don't generate the game data. Since the game inputs are predictable, or even static, the problem domain is reduced to a manageable level. A programmer can be relatively sure that a specific bit of code can be written to solve a specific problem, on a specific map level, with specific character attributes. It seems ugly, and to be honest it is ugly. As a friend of mine at Microsoft taught me, shipping your game is its most important feature.

The hack doesn't have to live in the code base forever, although it frequently does. If your game is even mildly successful, and you get the chance to do a sequel, you might have time to rip out the hacks and install an upgraded algorithm. You'll then be able to sleep at night.

A Tale from the Pixel Mines

At Origin it was common practice for programmers to add an appropriate comment if they had to install a hack to fix a bug. A couple of programmers were discussing which game had the most hacks—Ultima VII or Strike Commander. There was a certain pride in hacking in those days, since we were all young, somewhat arrogant, and enjoyed a good hack from time to time. The issue was settled with grep—a text file search utility. The Strike Commander team was the clear winner, with well over 500 hacks in their code.

Ultima VII wasn't without some great comments, though. My favorite one was something like, "This hack must be removed before the game ships." It never was. What's more I think the same hack made it into Ultima VIII.

Commenting your code changes is a fantastic idea, especially late in the project. After the code complete milestone the changes come so fast and furious that it's easy to lose track of what code changed, who changed it, and why. It's not uncommon for two programmers to make mutually exclusive changes to a piece of code, each change causing a bug in the other's code. You'll recognize this pretty fast, usually because you'll go into piece of code and fix a bug, only to have the same bug reappear a few versions later. When you pop back into the code you fixed, you'll see the code has mysteriously reverted to the buggy version. This might not be a case of source code control gone haywire, as you would first suspect. It could be another programmer reverting your change because it caused another bug.

That situation is not nearly as rare as you think, but there is a more common scenario. Sometimes I'll attempt a bug fix, only to have the testers throw it back to me saying that the bug still lives. By the time it comes back I may have forgotten why I chose the solution, or what the original code looked like. Even better, I may look at the same block of code months later, and not have a clue what the fix is attempting to fix, or what test case exposed the bug.

The solution to the problem of short term programmer memories is comments, as always, but comments in the late stages of development need some extra information to be especially useful. Here's an example of a late stage comment structure we used on the Microsoft projects:

 if (CDisplay::m_iNumModals == 0) {      // ET - 04/10/02 - Begin      // Jokerz #2107 - Close() here causes some errors,      // instead use Quit() as it allows the app to shutdown      // gracefully      Quit(); // Close();      // ET - 04/10/02 - End } 

The comment starts with the initials of the programmer, and the date of the change. The entire change is bracketed with the same thing, the only difference between the two being a "begin" and "end" keyword. If the change is a trivial one-liner with an ultra short explanation, the comment can sit on the previous line or out to the right.

The explanation of the change is preceded with the code name for the project, and the bug number that motivated the change. Codenames are important because the bug might exist in code shared between multiple projects, which might be in parallel development or as a sequel. The explanation of the change follows, and where it makes sense the old code is left in, commented out.

Most programmers will instantly observe that the source code repository should be the designated keeper of all this trivia, and the code should be left clean. I respectfully disagree. I think it belongs in both places. Code reads like a story, and if you are constantly flipping from one application to another to find out what is going on, it is quite likely you'll miss the meaning of the change.

Best Practice

At the end of the project, it's a good idea, although somewhat draconian, to convince the team to attach an approved bug number with every change made to the code. This measure might seem extreme but I've seen changes "snuck" into the code base at the last minute without any involvement from the rest of the team. The decision to do that shouldn't be made by a programmer at 3 a.m. on Sunday morning. If every change is required to have a bug number, it becomes a trivial matter to hunt down and revert any midnight changes made by well meaning but errant programmers.

There are plenty of software companies that employ some form of code review in their process. The terms "code review" and "game software" don't seem to belong in the same universe let alone the same book. This false impression comes from programmers that don't understand how a good code review process can turn a loose collection of individual programmers into a well oiled team of coding machines.

When most programmers think of code reviews they picture themselves standing in front of a bunch of people who laugh at every line of code they present. They think it will cramp their special programming style. Worst of all, they fear that a bad code review will kill their chances at a lead position or a raise.

I've been working with code reviews in a very informal sense for years, and while it probably won't stand up to NASA standards I think it performs well in creative software, especially games. It turns out there are two primary points of process that make code reviews for games work well: who initiates the review, and who performs the review.

The person who writes the code that needs review should actually initiate the review. This has a few beneficial side effects. First, the code will definitely be ready to review, since the person needing it won't ask otherwise. Programmers hate surprises of the "someone just walked in my office and wants to see my code" kind. Because the code is ready, the programmer will be in a great state of mind to explain it. After all, they should take a little pride in their work, right? Even programmers are capable of craftsmanship, and there's not nearly enough opportunity to show it off. A code review should be one of those opportunities.

The person performing the review isn't the person you think it should be. Most of you reading this would probably say, "the lead programmer." This is especially true if you are the lead programmer. Well, you're wrong. Any programmer on the team should be able to perform a code review. Something that is a lot of fun is to have a junior programmer perform code reviews on the lead programmer's code. It's a great chance for everyone to share his or her tricks, experience, and double check things that are critical to your project.

This implies that the programmers all trust each other, respect each other, and seek to learn more about their craft. I've had the privilege of working on a programming team that is exactly like that, and the hell of being on the other side as well. I'll choose the former, thank you very much. Find me a team that enjoys code reviews, and performs them often, and I'll show you a programming team that will ship their games on time.

When I worked on the Microsoft casual games, the programmers performed code reviews for serious issues all throughout the project, but they were done constantly after content complete, for each change, no matter how minor. Most of the time a programmer would work all day on five or six bugs, and call someone who happened to be on their way back from the bathroom to do a quick code review before they checked everything in. This was pretty efficient, since the programmer doing the review was already away from their computer. Studies have shown that a programmer doesn't get back into the "zone" until 30 minutes after an interruption. I believe it, too.

Bottom line: The closer you get to zero bugs the more checking and double checking you do, on every semicolon. You even double check the need to type a semicolon. This checking installs a governor on the number, the scope, and quality of every code change, and the governor is slowly throttled down to zero until the last bug is fixed. After that, the game is ready to ship.

Content

Programmers aren't immune to the inevitable discussions, usually late at night, about adding some extra content into the game at the eleventh hour. It could be something as innocuous as a few extra names in the credits, or it could be a completely new terrain system. You think I'm kidding, don't you?

Whether it is code, art, sounds, models, map levels, weapons, or whatever makes your game fun you've got to be serious about finishing your game. You can't finish it if you keep screwing with it! If you are really lucky, you'll wind up at a company like Bungie or id, who can pretty much release games when they're damn good and ready. The rest of us have to ship games when we get hungry, and sometimes the desire to make the best game can actually supercede basic survival. At some point, no matter how much you tweak it, your game is what it is, and even superhuman effort will only achieve a tiny amount of quality improvement. If you've ever heard of something called the "theory of diminishing returns" you know what I'm talking about. When this happens, you've already gone too far. Pack your game up and ship it, and hope it sells well enough for you to get a second try.

The problem most people have is recognizing when this happens—it's brutally difficult. If you're like me you get pretty passionate about games, and sometimes you get so close to a project you can't tell when it's time to throw in the towel.

Best Practice

Microsoft employs late stage beta testers. These people work in other parts of Microsoft but play their latest games. Beta testers are different from play testers because they don't play the game every day. They are always just distant enough and dispassionate enough to make a good judgment about when the game is fun, or when it's not. If you don't have Microsoft footing your development bills, find ad-hoc testers from just about anywhere. You don't need professional testing feedback. You just need to know if people would be willing to plunk down $60 for your game, and keep it forever.

A Tale from the Pixel Mines

As you've seen many times in this book, there are balancing issues. When I worked on the Ultima series, it wasn't uncommon for truly interesting things to be possible, code-wise, at a very late stage of development. On Ultima VIII, a particular magic spell had a bug that caused a huge wall of fire that destroyed everything in its path. It was so cool we decided to leave it in the game, and replace one of the lamer spells. It wasn't exactly a low risk move, completely replacing a known spell with a bug turned feature, but it was an awesome effect and we all felt the game was better for it.

I'm trying my very best to give you some solid advice instead of some wishy-washy pabulum. The truth is there's no right answer regarding last minute changes to your game. The only thing you can count on is 20-20 hindsight, and the only people that write the history books are the winners. In other words, when you are faced with a decision to make a big change late in the game, trust your experience, try to be at least a little bit conservative and responsible in your choices, and hope like hell that you are right.