Chapter 14: Everything (You Hate) to Know About Testing

When you've finished reading this chapter you'll have some great tools to create or improve your testing process. You'll learn about creating and using test plans, automated testing, and when to perform certain kinds of tests. You'll also learn who does the testing and what makes them tick, how to keep the development team and the test team from brawling, and most importantly, how to predict your ship date from statistical analysis of your bug database.

Why Are Games Buggy?

Most avid game players complain about the quality of software in the games industry. Even the very best games have bugs you can find easily for a laugh. Some bugs are so catastrophic, and yet simple to reproduce that it makes you wonder how in the hell they made it through the testing lab. Are the programmers and testers just a bunch of losers that have no idea what they are doing? The answer is absolutely not.

Every programmer and tester I've worked with were passionate about the quality of their products, and worked extremely hard to make the very best games they could. So what gives? If everyone is so excited about quality why do our games crash?

Reason 1: The Team Never Saw the Bug Reproduce

This has nothing to do with bugs getting it on with cheesy saxophone music in the background. Reproducing a bug involves identifying the exact procedure a tester uses to get the bug to raise its head in your game. If the team never saw a bug reproduce in their test lab, the bug will become an undocumented feature, much to the happiness of the competition. With significant application of time, money, and legions of testers it is possible to get rid of this problem and create a completely bug free game. In some industries such as health care, aerospace, and finance, this approach had better work because lives or money can be at stake.

A Tale from the Pixel Mines

I know a programmer who worked at a company that develops software for real time financial transactions, who I will call Scott—not his real name of course. He was a new guy and he'd just finished upgrading one of the older modules and it went online at 9:00 a.m. the next day. After a few hours, one of the auditing subsystems caught an enormous error and shut the offending module down, alerting everyone and their dog that something heinous was afoot. After the dust settled, Scott was called into conference with all the honchos. Scott was notified that in the three hours the module was live, the company lost enough money to pay his annual salary. Needless to say, Scott wasn't the only person called into conference that day with the expected string of management hierarchy, because any number of people should have caught this bug. Scott may have typed in the proper sequence of characters that created this horrible flaw in the code, but it was the lack of process that lost the company Scott's annual salary. The upside is that both the company and Scott learned a valuable lesson and everyone was motivated to change the process and give a higher priority to testing.

When it comes to games, no one will likely lose money, their life, or their property playing a game. Well, with the possible exception of all those folks paying out reams of cash for time on EverQuest—they're a special case. However, more and more gamers are expecting their games to be error free. A few years ago Disney had a horrible time with huge complaints from parents who had purchased one of its Lion King games. This decidedly non-technical audience reminds us that not every purchaser of a game is going to tolerate patches, and other configuration troubleshooting sessions to make things work.

Game testing procedures can be extremely tight or extremely loose, depending on the company and the people. It goes without saying that loose testing procedures will cause bugs, sometimes really embarrassing ones, to fall through the cracks.

Reason 2: The Team Decided to Leave the Bug in Its Natural Habitat

Sacrilege, you say! Why would any self-respecting team actually leave a known flaw in any application? Believe it or not, there are good reasons. There are also some excuses. The reason almost always involves time. I would argue that almost any bug can be addressed with enough time and effort applied to it, but time doesn't come free of charge. Time tends to drain the company coffers, adds shelf life to your competitors' products, or worse, moves your manufacturing slot to January!

Teams decide to leave bugs where they lie because they compared the time and costs required to fix them with the costs or risks associated with leaving the bugs. Here's an example of a bug like this:

"Screen was in 1280x1024 resolution, started the game and Alt-Tabbed back to desktop, desktop was in 1024x768. Machine: 833 Mhz IBM Thinkpad 384 MB RAM DX 8.1 onboard S3 graphics chipset attached to external monitor Sony G500. The true bug is that game doesn't launch in this resolution on my Dell."

This problem isn't exactly catastrophic, and it only affected machines using the S3 graphics chipset. Sometimes bugs fall into the corner case category, which means that they are highly unlikely for someone using the software to experience the bug because of the odd nature of the reproducible steps. Here's an example:

1. Open the game in full screen mode and start your last saved mission.

2. Open game options dialog.

3. Click the OK button of the game options dialog, and immediately afterward press Alt-Tab to minimize the game.

4. Restore the game window back to full screen mode.

5. Notice that most of the game textures have severe artwork corruption.

It turns out that the user had to press Alt-Tab in an extremely short amount of time after hitting the OK button—something like 150ms. Anyone who saw this bug was specifically looking for it, because it didn't fit into the common interface usage profile of people playing the game normally. The team decided that there were plenty of other bugs that deserved attention. In a perfect world there would be enough time to fix everything, but in truth time simply does not exist in sufficient quantity to accomplish that task.

Any modern piece of software is extremely complicated and has hundreds of closely interacting parts. This easily creates the possibility that an observed flaw in the application is actually hiding a truly nasty bug. The team made a choice between the lesser of two evils.

Now that you've seen some unfortunate but reasonable conditions under which known flaws can remain in computer software, compare them to their lesser cousins: the excuses. A classic excuse is not testing the application on the most recent operating system release. The opposite end of that spectrum is failure to bench test the game on the minimum machine configuration or common hardware configurations. Perhaps the saddest of all is a lack of communication between the designers and the testers. If the designers don't tell the testers about every feature of the game, how can they test it? If you create a rigorous testing process and consistently strive to improve it you will minimize the existence of these problems.

Gotcha

Testing is typically the bastard stepchild for many developers. It's like the paper you don't write until the last second and then you decide to wing it. This of course turns the end process into an organization and scheduling nightmare. So if there is any lesson to this chapter it's teaching all game developers how to integrate the testing process into every aspect of your development process. You will hate having to do this, but you will love the professionalism, headache savings, and time you gain by doing things the way I learned to from my experiences with Microsoft, Mattel, and at Origin. Testing is among a handful of processes that separate successful developers from the ones who never make it.