Testing Report

He looked at his notes. "I saved Testing for last, because I wanted to take some time to share with everyone else what we are doing here. Marta, why don't you outline what we are doing about testing RMS?"

Marta pulled some notes from her bag. "As most of you know, writing software is not something I have done a great deal of in the past, she said. "So, it was with some trepidation that I agreed to accept the Testing role on this team."

"We remember," Tim said dryly.

Marta continued, "Dan has given me some excellent coaching, and has also helped coordinate the relationship between the Testing role and the Development role. Additionally, Bill has been a big help, even though I'm not sure he has always agreed with my being on the team."

"Nothing personal, Marta," said Bill quickly. "You're a competent and intelligent worker, best as I can tell. I just think usability testing should be the responsibility of the development folks. But, since Dan sees it differently, I figured we might as well make the most of it."

"So what did the old curmudgeon do to help you out, Marta?" asked Tim with a grin.

"He and Dan came up with the idea of assigning a programmer to help me with the Testing role. This person is not one of the people working on building RMS, but is another member of the staff who acts as a technical assistant to me. I draw up the plans and am responsible for tracking the results and reporting to the team, while Mike—that's his name—helps with the actual testing. Or at least, the part of the testing that falls to us."

"I thought all testing was done by you," said Jane.

"Actually, no," said Marta. She turned to Dan. "I've prepared some transparencies that explain more of the process, if you don't mind."

"Not at all," said Dan. "It's all yours."

Marta described the difference between usage testing and coverage testing, and the subtypes of each. She explained that unit testing was the developers' responsibility, and that the developers were also doing functional tests. "Once they complete a feature set, though, they give it to us to validate that the function works," she said.

"How can they do that if not all the objects are complete?" asked Tim.

"They use stub procedures in the objects, which simply return without causing an error," Marta replied. "Sometimes they put in a message box, just so we know the stub actually got called. It's not as critical in RMS as it might be in a much larger project, because we'll be retesting everything when we do usage testing, and if we find anything then, the application is small enough that it won't be difficult to track down the offending object or component."

Tim looked at Bill. "Marta's got 'Build verification tests' listed, but I don't remember you or the others talking about doing daily builds. Are you?"

"Nope," said Bill, leaning back in his chair. "That's one of those ivory-tower things you read about, but very few development teams actually do. We tried it earlier this year on another project, and it became obvious very quickly that you have to adjust the frequency of builds to fit your project. For RMS, we're doing a build about every three days."

"And again," said Marta, "because the project is so small, we're not doing regression testing, either."

"Then what, exactly, is it that you and Mike are working on?" asked Jane. "I mean, it sounds like the developers are doing most of the work right now."

"They are," said Marta, smiling. "That's the approach we all decided on. If you remember, the Testing role doesn't really come to the forefront until the Stabilizing Phase. For now, we are preparing our test scripts as best we can, as well as running everything the developers give us through our test harness so we can get a head start on stress and performance testing. Mike and I are also working on our bug database."

Bill sat up straight. "Bug database? You hadn't mentioned that to me. What's that about?"

"Wait a minute, Bill, before we talk about bugs," said Dan. "I want to hear more about the test harness. Marta mentioned it in passing one day, but I want to hear the details. Then we'll learn about all the bugs in RMS."

"The test harness is pretty cool, I think," said Marta with a smile. "Mike wrote it over a weekend, and I was impressed, but he said it was not much different from others that were available. Basically, it is an automated component tester. We simply enter the name of the DLL, the method we want, any parameters it is expecting, and how many times we want it to be called in a row, and the test harness calls the component that many times and logs the results to a file. It also keeps track of things like memory usage, disk activity, and some other stuff, and writes those to the file after whatever number of repetitions we set. As long as we know the interface contract, we can test any object we want, automatically. Mike wants to expand it to test entire feature sets, but he hasn't had time yet. We've been testing all the components as the developers give them to us."

"Any problems yet?" asked Marilou.

"No, Bill's people seems to write pretty error-free code," said Marta.

"That's exactly right, and that's why you don't need any bug-tracking database!" said Bill heatedly. Glaring at Dan, he continued, "I think it's an insult to Sam, Beth, and the others working on RMS for Marta and Mike to even mention tracking bugs, much less to build a database to track them. Why, she herself said that she's getting error-free code!"

"Yes, but it's possible for code to be error-free and still contain bugs," said Dan.

Bill looked at the others as if to see whether they agreed that their leader had gone crazy. Then he turned back to Dan. "That's impossible! How can you have bugs in error-free code?"

"What's the definition of a bug?" Dan asked Marta.

"Any issue arising from the use of the product," Marta replied without hesitation.

"Is it necessarily a defect?"

"No, although it might be."

"What should be done about all such issues arising from the use of the product?"

"They should be reviewed and addressed prior to the release of the product."

"What are some examples of bugs that are not defects?"

Marta ticked the statements on her fingers as she spoke. "Enhancement requests; suggestions that are out of scope for the release; issues that arise over user preferences; and unavoidable design consequences."

"So you see, Bill," said Dan as he moved to the whiteboard, "you can write perfect, but not bug-free, code. It is the job of the Testing role, along with you and the rest of the RMS team, to deal with all issues in one way or another." Dan drew a circle on the board. "Here's the bug-tracking process. It's a lot like the risk-tracking process. It begins with the tester—Marta, Mike, one of us, one of the beta testers—reporting an issue. The issue gets entered into the database. As the development lead, Bill, you give it a priority rating and severity rating. You then assign the bug to a developer to resolve. Once the developer thinks the bug is resolved, he sends it back to the tester to see if it truly is resolved. If it is, the bug is retired. If it isn't, the tester reports it again and starts the process over." Dan turned to Marta. "Do you have the transparency that shows the priority and severity classifications?" She nodded and handed it to Dan, who put it on the overhead projector.

"As you can see, we classify all issues according to these ratings." He wrote the words quality bar on the board. "The quality bar is what the Testing role deems an acceptable quality for us to release the product. What is our quality bar for RMS, Marta?"

"No Severity 1 or 2 bugs and no Priority 1 or 2 bugs," said Marta without hesitation.

"So, as you can see," said Dan as he sat back down, "ultimately the Testing role determines whether or not we ship on time, or even at all. If the Testing folks say we haven't hit the mark, then we go back and do it again. It's a big responsibility, but it is the key to a quality product. That work begins during Developing, but finishes during Stabilizing. Any questions?"

"Just one," said Bill in a low tone. "What if we disagree with what Testing says? What if my people feel like they've hit the mark?"

"Then we bring it to the project team and we work it out here," said Dan evenly. "And remember, Bill, Testing's goal is the same as yours, and mine, and all of ours—to produce the very best application we can given the constraints we set in the beginning."