Case Study 8 -- Testing RMS

"Oh, man!"

Dan Shelly could not believe his eyes. Another network error message had popped up on his screen. "That's the third one this morning! What is going on down there?"

He opened My Computer to check his mapped drives, trying to track which server was down. "Oh, great! That new one we put in for the AutoCAD folks. Boy, am I going to hear about this!" He thought for a moment, then stood up and grabbed his planner and notepad. "Might as well head down to the server area and see what's going on. Certainly not going to get anything else done here."

When he got to the large, glassed-in room that held the bulk of Ferguson and Bardell's server farm, Tim O'Brien, the Network Manager, was already seated in front of the newest server, a quad-processor behemoth with an outboard RAID 5 array and massive amounts of memory. Tim was working feverishly at a command prompt, while off to one side, Bill Pardi was offering unsolicited advice. As Dan entered the room, Tim sat back with a defeated look. "No go, Bill," he said. "Whatever Sam put on here has munged the registry big-time. I'll probably have to go to the backups."

"Hi guys," said Dan, affecting a cheery tone. "What's going on? Are we having fun yet?"

Bill was defensive. "We're having some problems with the new CAD server, and Tim and I are trying to see what can be done about them." Dan didn't let on that he knew what was going on. "Really?" he said. "I didn't know Tim called on you for network advice, Bill. Do you know what's causing the problem?"

Before Bill could say anything else, Tim looked up from the console. "Aw, Bill, quit your acting. Dan won't kill us. He'll only want to kill us." He turned to Dan. "The truth is that the developers needed to test some of the components they were building, and they wanted to run another load-simulation script on them. Because we've talked about getting another server like this one for the RMS project, they asked if they could install the components on here. I didn't want to do it, but they said that the components worked fine on their own machines and that no one would notice if they put them on the CAD box. So I said 'Yes.' The installation went okay, but when they started their tests, the box went down. We got it up again twice, but it looks like the third time is the charm. I'll try restoring the registry from last night, but if that doesn't work, I'll probably have to rebuild it using yesterday's backups."

Dan kept his voice level. "So let me get this straight." He turned to Bill. "Your developers needed to test some components, so you told them to put them on a production server?" He turned back to Tim. "And you let them?" Tim nodded glumly. Bill started to say something, but Dan held up a hand. "I don't want to hear it, Bill. We'll talk about this another time, another place. For now, you and I are going to leave so Tim can get this box back up as fast as possible. I'm tired of getting error messages while I'm working."

Tim and Bill looked puzzled and anxious. "Why are you getting error messages?" asked Tim. "You don't use this server."

"Didn't either one of you wonder why I was down here so soon after the server went down?" Dan asked. "Did you think it was just coincidence? I don't advertise the fact, but I map drives to every one of our servers every week. It's how I test performance and check whether our security policies are being followed. Today happened to be the day I mapped to this server."

Just then the three of them noticed Jim Stewart leaning against the door with his arms folded and a calm but stern look on his face.

"Jim! Not often we see you in this neck of the woods," said Dan. "Come and join us." He moved slightly to one side, blocking Jim's view of the console. "So, what are you doing down here?"

"Calculating," said Jim dryly. "Uh-oh," thought Dan, "here's trouble." Tim asked innocently, "Calculating what?"

"Calculating just how much this downed server is costing me," said Jim, his voice dropping in both pitch and temperature. "I can actually calculate it two or three different ways. Want to know how?"

"Not particularly," thought Dan, but he knew they were in far enough that the only way out was to keep going to the other side, so he said, "Sure, Jim, tell us how it looks to you."

As if he were teaching a college class, Jim said, "There's the wasted salary method. That's the easiest, because I know the hourly salary figure for every department, so I just multiply that for the affected departments by the hours the server is down. Then, there's the lost revenues method." He began to pace up and down. "I hate it when he paces," thought Dan, but he said nothing.

"Lost revenues are somewhat harder to calculate, because they vary month by month. But I can take the total profit for the year for a department, divide it by the number of hours to get an hourly rate, and use that.

"Finally, there is the opportunity cost method. What work did we lose by not getting this work done so that our resources were available to sell to someone else? This cost is pretty much speculative but can often yield rather spectacular results."

By this time, the knot in Dan's stomach had grown to about the size of a cantaloupe, and both Bill and Tim looked pretty sick. Not wanting to ask, but knowing he had to, Dan said, "And what numbers do you come up with for the CAD server, Jim?"

Jim stopped pacing and looked first at Dan, then at Bill and Tim. His voice rose. "Best as I can figure it, lost salaries work out to about $1,300 an hour. Lost revenues look like somewhere between $4,000 and $5,000 an hour. And opportunity costs? Let's just say, higher than that. Bottom line is, if you keep this server down the rest of the day, you're going to cost Ferguson and Bardell at least $25,000. So what are you doing about it?"

Dan affected a calmness he did not feel. "Jim, we've got one of the best in the business trying to fix it. I was going to talk with Bill later about what caused this, but I think now would be the best time, and I want you to be in on it. So why don't we all adjourn to the small conference room down the hall, and leave Tim to stop the dollars from going down the drain."