Chapter 9: Testing by the Numbers


Product metrics, such as the number of defects found per line of code, tell you how fit the game code is for release. Test metrics can tell you about the effectiveness and efficiency of your testing activities and results. A few pieces of basic test data can be combined in ways that reveal important information you can use to keep testing on track while getting the most out of your tests and testers.

Testing Progress

Collecting data is important to understanding where the test team is and where they are headed in terms of meeting the needs and expectations of the overall game project. Data and charts can be collected by the test lead or the individual testers. Take responsibility for knowing how well you're doing. For example, in order to estimate the duration of the test execution for any portion of the game project, estimate the total number of tests to be performed. This number is combined with data on how many tests can be completed per staff-day of effort, how much of a tester's calendar time is actually spent on testing activities, and how many tests you expect to be redone.

Figure 9.1 provides a set of data for a test team starting to run tests against a new code release. The project manager worked with the test lead to use an estimate of 12 tests per day as the basis for projecting how long it would take to complete the testing for this release. Thirteen days into the testing, the progress lagged what had been projected , as shown in Figure 9.2. It looks like progress started to slip on the fifth day, but the team was optimistic that they could catch up. By the tenth day they seemed to have managed to steer back toward the goal, but during the last three days the team lost ground again, despite the re-assignment of some people on to and off of the team.


Figure 9.1: Planned and actual test execution progress data.

Figure 9.2: Planned and actual test execution progress graph.

To understand what is happening here, collect data for each day that a tester was available to do testing and the number of tests he or she completed each day. This information can be put into a chart, as shown in Figure 9.3. The totals show that an individual tester completes an average of about four tests a day.


Figure 9.3: Test completion rate per tester per day.

Once you have the test effort data for each person and each day, you must compare the test effort people have contributed to the number of work days they were assigned to participate in system testing. Ideally, this ratio would come out to 1.00. The numbers you actually collect will give you a measurement of something you may have felt was true, but couldn't prove before: Most testers are unable to spend 100% of their time on testing. This being the case, don't plan on testers spending 100% of their time on a single task! Measurements will show you how much to expect from system testers based on various levels of participation. Some testers will be dedicated to testing as their only assignment. Others may perform a dual role, such as developer/tester or QA engineer/tester. Collect effort data for your team members that fall into each category as shown in Figure 9.4.


Figure 9.4: Tester participation rate calculations.

This data leads to a number of important points. One is that, given tester "overhead" tasks such as training, meetings, preparing for demos, and so on, a full-time tester may only be able to contribute about 75% of his or her time at best, and 50% ‚ 60% on average over the course of a long project. If you are counting on people with other responsibilities ‚ for example, artists , developers, or QA ‚ to help with testing, then expect only half as much participation as the full-time testers. Using the numbers in Figure 9.4, that would be about 30% of their total available time. You will need to make these measures for your own particular project.

Also, by combining the individual productivity numbers to get a team productivity number, you can see that this team performs only half as many tests as they could if they had 100% of their time to perform testing. This number can be combined with your effort estimate to give an accurate count of calendar work days remaining before testing will be finished. Using the number of 125 tests remaining, and a staff size of 11 testers, you would approximate 11 staff-days of testing remaining. However, now that you know what the team's productivity is, you divide 11 by 46% to get 24 calendar work days remaining, or nearly five "normal" work weeks. If you committed to the original, optimistic number of 11 days, there would be much gnashing of teeth when the tests weren't actually completed until three weeks after they were promised !

You need this kind of information to answer questions such as "How many people do you need to get testing done by Friday?" or "If I can get you two more testers, when can you be done?" Also burn into your mind that it's easier to stay on track by getting a little extra done day to day than trying to make up a big chunk in a panic situation (remember Rule 1 ). Going back to Figure 9.1 you can see that on 8-Jan the team was only six tests behind the goal. Completing one extra test on each of the previous six work days would have had the team on goal. If you can keep short-term commitments to stay on track, you will be able to keep the long- term ones.




Game Testing All in One
Game Testing All in One (Game Development Series)
ISBN: 1592003737
EAN: 2147483647
Year: 2005
Pages: 205

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net