Dealing with Big Trouble
Murphy is alive and well in the computer game industry, and I'm sure he's been an invisible team member on most of my projects—some more than others, but most especially at Origin Systems, where Murphy had a corner office. I think his office was nicer than mine!
Big trouble on game projects comes in a few flavors: too much work and too little time, human beings under too much pressure, competing products in the target market, and dead-ends. There aren't necessarily standard solutions for these problems, but I can tell you what has been tried and how well it worked, or didn't work, as the case may be.
Projects Seriously Behind Schedule
Microsoft has a great way of describing a project behind schedule. They say it's "coming in hot and steep." I know because the first Microsoft Casino project was exactly like that. We had too much work to do but too little time to do it in. There are a few solutions to this problem such as working more overtime or throwing bodies at the problem. Each solution can work, but it can also have a dark side.
The Dreaded Crunch Mode—Working More Hours
It amazes me how much project managers choose to work their teams to death when the project falls behind schedule.
| A Tale from the Pixel Mines | On my very first day at Origin Systems, October 22, 1990, I walked by a whiteboard with an ominous message written in block letters: "84 Hour Workweeks - MANDATORY." With a simple division I realized that 84 divided by 7 is 12. Twelve hours per day, seven days per week, was Origin's solution for shipping Savage Empire for the Christmas, 1990 season. To the Savage Empire team's credit, they shipped the game a few tortured weeks later, and this "success" translated into more mandatory overtime to solve problems. We were all young, mostly in our late 20s, and the amount of overtime that was worked was bragged about. There was a company award called the "100 Club," which was awarded to anyone who worked more than 100 hours in a single workweek. At Origin, this club wasn't very exclusive. |
Humans are resilient creatures, and under extraordinary circumstances can go long stretches with very little sleep or a break from work. Winston Churchill, during World War II, was famous for taking little cat naps in the Cabinet War Rooms lasting just a cumulative few hours per day, and he did this for years. Mr. Churchill had good reason to do this. He was trying to lead England in a war against Nazi Germany, and the cost of failure would have been catastrophic for his country and the entire world.
Game companies consistently ask for a similar commitment on the part of their employees—to work long hours for months, even years on end. What a crime! It's one thing to save a people from real tyranny, it's quite another to make a computer game. This is especially true when the culprit is bad planning, blindness to the reality of a situation, and a lack of skill in project management.
It is a known fact that under a normal working environment, projects can be artificially time compressed up to 20% by working more hours. This is the equivalent of asking the entire team to work eight extra hours on Saturday. I define a normal working environment as one where people don't have their lives, liberty, or family at stake, and all they expect from their effort is a paycheck. This schedule can be kept up for months, if the team is well motivated.
| A Tale from the Pixel Mines | It was this schedule that compressed Ultima VIII after a last minute feature addition: Origin asked the team to ship the game in two extra languages: German and French. The team bloated to nearly three times it's original size, adding native German and French speakers to write the tens of thousands of lines of conversation and test the results. We worked overtime for 5 weeks—60 hours per week, and we took the sixth week and worked a normal workweek, which averaged 50 hours. This schedule went on from August to March, or eight months. Youth and energy went a long way, and in the end we did ship the game when the team thought we were going to ship the game, but everyone was exhausted beyond their limits. |
For short periods of time, perhaps a week or two weeks, truly extraordinary efforts are possible. Twelve hour days for a short burst can make a huge difference in your game. Well managed and planned, it can even boost team morale. It feels a little like summer camp. A critical piece of this strategy is a well-formed goal such as:
-
Fix 50 bugs per developer in one week.
-
Finish integrating the major subsystems of the game.
-
Achieve a playthrough of the entire game without cheating
The goal should be something the team can see on the horizon, well within sprinting distance. They also have to be able to see their progress on a daily basis. It can be quite demoralizing to sprint to a goal you can't see, because you have no idea how to gauge your level of effort.
| A Tale from the Pixel Mines | On Ultima VII, Richard Garriott was always doing crazy things to support the development team. One night he brought in steaks to grill on Origin's BBQ pit. Another night, very late, he brought in his monster cappuccino machine from home and made everyone on the team some latte. One Saturday he surprised the team and declared a day off, taking everyone sky diving. Richard was long past the time where he could jump into C++ and write some code, but his support of the team and simply being there during the wee hours made a huge difference. |
There's a dark side to overtime in the extreme that many managers and producers can't see until it's too late. It happened at Origin, and it happens all the time in other companies. When people work enough hours to push their actual pay scale below minimum wage, they begin to expect something extraordinary in return, perhaps in the form of end of project bonuses, raises, promotions, and so on.
The evil truth is that the company usually cannot pay anything that will equal their level of effort. The crushing overtime is a result of a project in trouble, and that usually equates to a company in trouble. If it wasn't so, company managers wouldn't push staggering overtime onto the shoulders of the team. At the end of the day, the project will ship, probably vastly over budget and most likely at a lower quality than was hoped. These two things do not translate into huge amounts of money flowing into company coffers, and subsequently into the pockets of the team.
A few months after these nightmare projects ship, the team begins to realize that all those hours amounted to nothing more than lost time away from home. Perhaps their firstborn took a few wobbling steps or spoke their first words, "Hey where in the hell is Daddy, anyway?" This frustration works into anger, and finally into people leaving the company for what they think is greener pastures. High turnover right after a project ships is pretty common in companies that require tons of overtime.
Someone once told me that you'll never find a tombstone with the following epitaph: "I wish I worked more weekends." As team member, you can translate that into a desire to predict your own schedule as best you can, and send up red flags when things begin to get off track. If you ever get to be a project lead, I hope you realize that there's a place for overtime, but it can't replace someone's life.
Pixel Fodder—Throw Warm Bodies at the Problem
Perhaps the second most common solution to projects seriously behind schedule is to throw more developers on the project. Well managed, this can have a positive effect, but it's never very cost effective, and there's a higher risk of mistakes. It turns out there's a sweet spot in the number of people that can work on any single project.
| A Tale from the Pixel Mines | Ultima Online was the poster child of a bloated team. In December of 1996, the entire Ultima IX team was moved to Ultima Online, in the hopes that throwing bodies at the problem would speed the project to completion. This ended up being something of a disaster, for a few reasons. First, the Ultima IX team really wanted to work on Ultima IX. Their motivation to work on another project was pretty low. Second, the Ultima Online team had a completely different culture and experience level, and there were clashes of philosophy and control. Third, Ultima Online didn't have a detailed project plan, somewhat due to the fact that no one had ever made a massive multiplayer game before. This made it difficult to deploy everyone in their area of expertise. I happened to find myself working with SQL servers, for example, and I didn't have a shred of experience! Through a staggering amount of work—an Origin hallmark—on the part of the original Ultima Online team and the Ultima IX newcomers the project went live less than nine months after the team was integrated. The cost was overwhelming, however, especially in terms of employee turnover in the old Ultima IX team. Virtually none of the programmers, managers, or designers of Ultima IX remained at Origin to see it completed. |
One effect of overstaffing is an increased need to communicate and coordinate amongst the team members. It's a generally accepted fact that a manager's effectiveness falls sharply if they have any more than seven reports, and it is maximized at five reports. If you have a project team of 12 programmers, 14 artists, and 10 designers, you'll have two programming leads reporting to a technical director, and a similar structure for artists and designers. You'll likely have a project director as well, creating a project management staff of ten people.
If your management staff is anything less than that, you'll probably run into issues like two artists working on the same model, or perhaps a programming task that falls completely through the cracks. To be honest, even with an experienced management team you'll never be completely free of these issues.
| A Tale from the Pixel Mines | Sometimes you get lucky, and you can add people to a project simply because a project is planned and organized in the right way. A good example of this is the Bicycle Cards project, basically a bunch of little games packaged up in one product. When some of the games began to run behind schedule, we hired two contractors to take on a few games apiece. The development went completely smoothly with seven programmers in parallel. Their work was compartmentalized, communication of their tasks were covered nearly 100% by the design document, and this helped ease any problems. |
They say that nine women can't make a baby in one month. That's true. There is also a documented case of a huge group of people who built an entire house from the ground up in three days, due to an intricately coordinated plan, extremely skilled people, and very specialized building techniques. Your project could exist on either side of these extremes.
Slipping the Schedule
This solution seems de rigor in the games industry, even with a coordinated application of crunch mode and bloating the team. There's a great poster of Ultima VII and Strike Commander that Origin published in 1992, in the style of movie posters that bragged "Coming this Christmas." It turns out that those posters got the season right, they just had the wrong year.
There's a long list of games that shipped before their time, perhaps the worst offender in my personal history was Ultima Online. There was even a lawsuit to that effect, where some subscribers filed a class action lawsuit against Electronic Arts for shipping a game that wasn't ready. Thankfully it was thrown out of court. A case like that could have had drastic effects on the industry!
The pressure to ship on schedule is enormous. You might think that companies want to ship on time because of the additional costs of the development team, and while the weekly burn rate of a gigantic team can be many hundreds of thousands of dollars, it's not the main motivation. While I worked with Microsoft, I learned that the manufacturing schedule of our game was set in stone. We had to have master disks ready by such and such a date, or we would lose our slot in the manufacturing facility. Considering that the other Microsoft project coming out that particular year was Windows XP, I realized that losing my place in line meant a huge delay in getting the game out.
While things like manufacturing can usually be worked out, there's another, even bigger, motivation for shipping on time. Months before the game is done, most companies begin spending huge money on marketing. Ads are bought in magazines or sometimes television, costing hundreds of thousands of dollars. You might not know this, but those special kiosks at the end of the shelves in retail stores, called endcaps, are bought and paid for like prime rental real estate, usually on a month by month basis. If your game isn't ready for the moment those ads are published or those kiosks are ready to show off your game, you lose the money. No refunds here!
This is one of the reasons you see the executives poking around your project six to eight months before you are scheduled to ship. It's because they are about to start writing big checks to media companies and game retail chains in the hopes that all this cash will drive up the sales of your game. The irony is, if the execs don't believe you can finish on time, they won't spend the big bucks on marketing, and your game will be buried somewhere on a bottom shelf in a dark corner of the store. Oh, and no ads either. You're best advertising will be by personal email to all your friends, and that just won't cut it. In other words, your game won't sell.
The difference between getting your marketing pressure at maximum and nothing at all may only be a matter of slipping a few weeks, or even a few days. What's worse, this judgment call is made months before you are at code complete—a time when your game might not be much more than a pretty face. Crazy, huh?
Probably the best advice I can give you is make sure you establish a track record of hitting each and every milestone on time, throughout the life of your project. Keep your bug count under control, too. These two things will convince the suits that you'll ship on time, with all the features you promised. Whatever you do, don't choose schedule slippage at the last minute. If you must slip, slip it once and make sure you give the suits enough time to react to all the promises they made on your behalf. This is probably at least six months prior to your release date, but it could be more.
Cutting Features and Postponing Bugs
Perhaps the most effective method of pulling a project out of the fire is reducing the scope of work. You can do it in two ways: nuke some features of the game and choose to leave some bugs in their natural habitat, perhaps to be fixed on the sequel. Unless you've been a bit arrogant in your project, the players and the media won't know about everything you wanted to install in the game. You might be able to shorten or remove a level from your game, reduce the number of characters or equipment, or live with a less accurate physics system.
Clearly, if you are going to cut something big you have to do it as early in the project as you can. Game features tend to work themselves in to every corner of the project, and removing them wholesale can be tricky at best, impossible at worst. Also, you can't have already represented to the outside world that your game has 10,000 hours of game play when you're only going to have time for a fraction of that. It makes your team look young and a little stupid.
| Best Practice | Always give yourself some elbow room when making promises to anyone, but especially the game industry media. They love catching project teams in arrogant promises. It's great to tell them things about your game, but try to give them specifics in those features you are 100% sure are going be finished. |
After code complete, and the programmers are doing virtually nothing but fixing bugs like crazy, an obvious solution for reducing the workload is to spirit away some of the less important bugs. As the ship date approaches, management's desire to "fix" bugs in this manner becomes somewhat ravenous, even to the point of leaving truly embarrassing bugs in the game, such as misspelled names in the credits or nasty crashes.
Anything can be bad in great quantities, and reducing your game's scope or quality is no exception. One thing is certainly true—your players won't miss what they never knew about in the first place. I can tell you that the book you now hold in your hands had three chapters that got cut. Perhaps you can guess what they might have contained. Just like any game project, my publisher and I decided to cut some chapters to get the book out on time, because that was more important than some of the other solutions, which included slipping the due date.
Only time will tell if we made the right decision, and the same will be true of your game project. It is incredibly difficult to step away from the guts of your project and look at it, objectively, from the outside. I've tried to do this many times, and it is one of the most difficult things to do, especially in those final days. Anyone who cares about their game won't want to leave a bug unfixed or cut a feature.
Ask yourself two serious questions when faced with this kind of decision: will my decision sell more copies? Will it keep someone from returning the game? If your answer is yes, do what it takes. Otherwise, move on and get your game shipped.
Personnel Related Problems
At the end of a project, everyone on the team is usually stretched to the limit. Good natured and even keeled people aren't immune to the stresses of overtime and the pressure of a mountain of tasks. Some game developers are far from good natured and even keeled! Remember always that whatever happens at the end of a project, it should be taken in the context of the stresses of the day, not necessarily as someone's habitual behavior. After all, if someone loses their cool at 3 a.m. after having worked 36 hours straight, I think a little slack is in order. If this same person loses his cool on a normal workday after a calm weekend, perhaps some professional adjustments are a good idea.
Exhaustion
The first and most obvious problem faced by teams is simple exhaustion. Long hours and missed weekends create pressure at home and a robotic sense of purpose at work. The team begins to make mistakes, and for every hour they work the project slips back three hours. The only solution for this is a few days away from the project. Hopefully you and your team won't let the problem get this bad. Sometimes all it takes is for someone to stand up and point to the last three days of non-progress and notice that the wheels are spinning but the car isn't going anywhere. Everyone should go home for 48 hours, even if it's Tuesday. You'd be surprised how much energy people will bring back to the office.
One other thing: They may be away from their desks for 48 hours but their minds will still have some background processes mulling over what they'll do when they get back to work. Oddly enough these background thoughts can be amazingly productive, since they tend to concentrate on planning and the big picture rather than every curly brace. When they get back, the additional thought works to create an amazing burst of productivity.
| A Tale from the Pixel Mines | Late in the Magnadoodle project for Mattel Media, I was working hard on a graphic bug. I had been programming nearly 18 hours per day for the last week, and I was completely spent. At 3 a.m., I finally left the office, unsuccessful after four hours working on the same problem, and went to sleep. I specifically didn't set my alarm, and I unplugged all the telephones. I slept. The next morning, I awoke at a disgusting 11 a.m. and walked into the office with a fresh cup of Starbuck's in hand. I sat down in front of the code I was struggling with the night before and instantly solved the problem. The bug that had eluded me for four hours the day before was solved in less than fifteen seconds. If that isn't a great advertisement for sleep gaining efficiency in a developer, I don't know what is. |
Morale
Team morale is directly proportional to their progress towards their goal, and isn't related to their workload. This may seem somewhat counter intuitive, but it's true. One theory that has been proposed regarding the people that built the great pyramids of Egypt is that teams of movers actually competed with each other to see how many blocks they could move up the ramps in a single day. Their workload and effort was backbreaking and their project schedule spanned decades. The constant competition, as the theory suggests, created high productivity and increased morale at the same time.
Morale can slide under a few circumstances, all of which are completely controllable. As the previous paragraph suggests, the team must be convinced they are on track to achieving their goal. This implies that the goal shouldn't be a constantly moving target. If a project continually changes underneath the developers, they'll lose faith that it will ever be completed. The opposite is also true—a well designed project that is built to spec as if it had blueprints is a joy to work on, and developers will work amazingly hard to get to a finish line they can see.
There's also a lot to be said for installing a few creature comforts for the development team. If they are working long hours, you'll be surprised what a little effort towards team appreciation will accomplish.
| Best Practice | Get out the company credit card and make sure people on the project are well cared for. Stock the refrigerator with drinks and snacks, buy decent dinners every night, and bring in donuts in the morning. Bring in good coffee and get rid of the cheap stuff. Every now and then, make sure the evening meal is a nice one, and send them home afterwards instead of burning the midnight oil for the tenth night in a row. |
Something I've seen in the past that affects morale is the relationship between the development team and the testing team. I've seen the entire range, from teams that wanted to beat each other with pipes to others that didn't even communicate verbally—they simply read each other's minds and made the game better. Someone needs to take this pulse every now and then, and apply a little rudder pressure when needed to keep things nice and friendly. Some warning signs to watch for include unfriendly japes in the bug commentary, discussion about the usefulness of an individual on either team or their apparent lack of skill, or the beginnings of disrespect for their leadership.
Perhaps the best insurance against this problem is forging personal relationships among the development leadership and testing leadership, and if possible with individuals on the team. Make sure they get a chance to meet each other in person if at all possible, which can be difficult since most game developers are a few time zones away from their test team. Personal email, telephone conversations, conference calls, and face to face meetings can help forge these professional friendships, and keep them going when discussions about bugs get heated.
This leads into something that may have the most serious affect on morale, both positive and negative. The developers need to feel like they are doing something worthwhile, and that they have the support of everyone. The moment they feel that their project isn't worth anything, due to something said in the media or perhaps an unfortunate comment by an executive, you can see the energy drain away to nothing. The opposite of this can be used to boost morale. Bring in a member of the press to see some kick-ass previews, or have a suit from the publisher shower the team with praise, and they'll redouble their effort. If you happen to work in a company with multiple projects, perhaps the best thing I've seen is one project team telling another that they have a great game. Praise from one's closest colleagues is far better than any other.
Other Stuff
Perhaps the darkest side of trouble on teams is when one person crosses the line and begins to behave in an unprofessional manner. I've seen everything from career blackmail to arrogant insubordination, and the project team has to keep this butthead on the team or risk losing their "genius." My suggestion here is to remember that the team is more important than any single individual. If someone leaves the team, even figuratively, during the project you should invite them to please leave in a more concrete manner.
Your Competition Beats You to the Punch
There's nothing that bursts your bubble quite as much as having someone walk into your office with a game in their hand, just released, that not only kicks butt but is exactly like your game in every way. You might think I'm crazy, but I'll tell you that you have nothing to worry about. The fact is that you can learn a lot from someone else's game simply by playing it, studying their graphics system, testing their user interface, and finding other chinks in their armor. After all, you can still compile your game, whereas they've burned theirs on optical media.
True, you won't be the first to market. Yes, you'd better be no later than second to market, and certainly you'd better make sure you don't repeat their mistakes. At least you have the benefit of having a choice, and you also have the benefit of dissecting another competitor's product before you put your game on the shelf.
| A Tale from the Pixel Mines | They say that loose lips sink ships, right? This is certainly true in the game industry. Strike Commander, Origin's first 3D game, was due out in Christmas of 1992. In the summer of 1992, Origin took Strike Commander to the big industry trade show at the time, the Consumer Electronics Show, and made a big deal of Strike Commander's advanced 3D technology. They went so far as to give away technical details of the 3D engine, to which the competition immediately researched and installed in their own games. Origin's competitive advantage was trumped by their own marketing department, and since the team had to slip the schedule past Christmas, the competition had more time to react. What a disaster! |
The game industry tends to follow trends until they bleed out. That's because there's a surprisingly strong aversion to unique content on the part of game executives. If a particular game is doing well, every company in the industry puts out a clone until there are fifty games out there that all look alike. Only the top two or three will sell worth a damn, so make sure you are in that top two or three.
There's No Way Out—Or Is There?
Sometimes you have to admit there's a grim reality—your game has coded itself into a corner. The testers say the game just isn't any fun. You might have gone down a dead-end technology track, such as coding your game for a dying platform.
What in the hell do you do now?
Mostly, you find a way to start over. If you're lucky, you might be able to recycle some code, art, map levels, or sounds. If you're really lucky, you might be able to replace a minor component and save the project. Either way, you have to find the courage to see the situation for what it is and act. Putting your head in the sand won't do any good.
| A Tale from the Pixel Mines | After Ultima IX was put on ice, and I was working hard on the Ultima Online project, I secretly continued work on Ultima IX at my house in the evenings and on weekends. My goal wasn't so much to resurrect Ultima IX or try to finish it single-handedly. I just wanted to learn more about 3D hardware accelerated polygon rasterization, which was pretty new at the time. I was playing around with Glide, a 3D API from 3DFx that worked on the VooDoo series of video cards. In a surprisingly little amount of work, I installed a Glide compliant rasterizer into Ultima IX, complete with a basic, ultra stupid, texture cache. What I saw was really amazing—Ultima IX running at over 40fps. The best framerate I'd seen so far was barely 10fps using our best software rasterizer. I took my work into Origin to show it off a bit, and the old Ultima IX team just went wild. A few months later, the project was back in development, with a new direction. Ultima IX would be the first Origin game that was solely written for hardware accelerated video cards. A bold statement, but not out of character with the Ultima series. Each Ultima game pushed the limits of bleeding edge technology every time a new one was published, and Ultima IX was no exception. |
One Last Word—Don't Panic
There are other things that can go terribly wrong on projects, such as when someone deletes the entire project from the network or when the entire development team walks out of the door to start their own company. Yes, I've seen both of these things happen, and no, the projects in question didn't instantly evaporate. Every problem can be fixed, but it does take something of a cool head. Panic and overreaction—some might say these are hallmarks of your humble author—rarely lead to good decisions.
Try to stay calm, and try to gather as much information about whatever tragedy is befalling you. Don't go on a witchhunt. You'll need every able-bodied programmer and artist to get you out of trouble. Whatever it is, your problem is only a finite string of 1s and 0s in the right order. Try to remember that you'll probably sleep better.
