Game Quality Appraisal | Game Testing All in One (Game Development Series)

The actual quality of the game is established by its design and subsequent implementation in code. However, appraisal activities are necessary to identify the difference between what was produced and what should have been produced. Once identified, these differences can be repaired before ‚ and sometimes after ‚ releasing the game.

Testing is considered an appraisal activity. It establishes whether the game code performs the functions for which it was intended. But testing is not the most economical way to find game defects; it's best to catch problems at the point they are introduced.

Having peers review game deliverables as they are being produced provides immediate feedback and the opportunity to repair problems before they are introduced and commingled with the rest of the game. It will be much harder and more expensive to find and repair these problems at later phases of the project.

Peer reviews come in different "flavors." In each case, there are times when you, the tester, will be required to participate. If you don't put in the necessary time and effort to contribute to the review, you and your team will be less likely to be asked to participate in the future. Make sure you take this responsibility seriously when your number gets called.

Walkthroughs

Walkthroughs are one form of peer review. A general outline of a walkthrough is as follows :

Leader (for example, the designer) secures a room and schedules the walkthrough
Leader begins the meeting with an overview of work including scope, purpose, and special considerations
Leader displays and presents document text and diagrams
Participants ask questions and raise issues
New issues raised during the walkthrough are recorded during the meeting

The room should comfortably fit the number of people attending and have a projector for presentations. A whiteboard or paper easel pad can be used by the leader or participants to elaborate on questions or answers. Limit attendance to 6 ‚ 8 people at most. This should not turn into a team meeting. Only include a representative from each project role that is potentially affected by the work you are walking through. For example, someone from the art team does not have to be in most code design walkthroughs, but there should be an experienced game artist there when graphics subsystem designs are being presented. Don't invite the test lead to every single walkthrough that affects the test team. If you do, then game knowledge and walkthrough experience won't get passed on to other testers. This also keeps the test lead from spending too much time on walkthroughs and not enough time on test leading. Work with the test lead to find other capable representatives on her team. If you are the test lead, send someone capable from your team in your place when you can.

Be sure to invite one or more developers to your test walkthroughs. It's a great way to find out if what you intend to test is really what the game is going to do once it's developed.

Conversely, get yourself invited to design and code walkthroughs. Brush up on the design techniques and programming language your team is using. Even if you don't have any comments to improve the author's work, you can use what you learn there to make your tests better.

It's also not a bad idea to use some walkthroughs as mentoring or growth opportunities for people on your team. The "guests" should limit their own questions and comments during the meeting to the material being presented and have a follow-up time with their "host" to go over any other questions about procedures, the design methodology being used, and so on. This probably should not be done for every walkthrough, but in situations where someone already has a background in the topic and/or is expected to grow into a lead role for some portion of the project.

Here's a list of representatives to consider inviting to walkthroughs of various project artifacts:

TDD ‚ tech lead, art director, producer, project manager
Story board ‚ producer, dev lead, artists
SQAP ‚ project manager, producer, development lead, test lead, QA lead, and engineer(s)
Code designs, graphics ‚ key developers, art representative, test representative
Code designs, other ‚ key developers, test representative
Code ‚ key developers, key testers
Test plan ‚ project manager, producer, development lead, key testers
Tests ‚ feature developer, key testers

Relevant topics to cover in walkthroughs include:

Possible implementations
Interactions
Appropriate scope
Traceability to earlier work products
Completeness

Issues raised during the walkthrough are also recorded during the meeting. Sometimes the presenter will realize a mistake simply by talking about his work. The walkthrough provides an outlet for that. One participant acts as a recorder, recording issues and presentation points that are essential to understand the material. Other participants may end up using the information for downstream activities, such as coding or testing. The leader is responsible for promptly closing each issue and distributing the meeting notes to the team within one week of the walkthrough. QA is expected to follow up by checking that the issues were indeed closed before any work was done based on the material that was walked through and that the notes were distributed to the participants.

Reviews

Reviews are a little more intimate than walkthroughs. Fewer people are involved ‚ typically 4 to 6 ‚ and the bulk of time is spent on the reviewers' comments.

Reviewers are expected to prepare their comments prior to the review meeting and submit them to the review leader so that they can be consolidated prior to the actual meeting. Comments sent electronically are easier to compile and understand. Be sure to let the review leader know when you are going to submit a pen-and-paper markup instead of an electronic file. The review leader may or may not be the author of the material being reviewed.

The review itself can be an in-person meeting between the author and reviewers or simply a review of the comments by the author alone who contacts individual reviewers if he has any questions about their issues. An in-between approach is for the author to look over the reviewer comments prior to the review meeting and limit the meeting time to discussions over the few issues that the author disagrees with or has questions about. This meeting can also take place virtually using network meeting software and phone headsets. That is especially useful for projects distributed across studios that are separated in space and time.

During the meeting, someone ‚ usually the review leader ‚ must take notes and publish the resolution of each item to the team. If the opinions of a reviewer differ from what the author believes should be done, decisions on technical matters are left to the author whereas procedural matters can be resolved by QA.

Checklist-Based Reviews

Another form of review takes place between only two people: the author and a reviewer. In this case, the reviewer follows a checklist to look for mistakes or omissions in the author's work. The checklist should be thorough and based on specific mistakes that are common for the type of work being reviewed. Requirements, code, and test reviews of this type would each use different checklists. At times it would even be appropriate to have checklists specific to a game project. These checklists should constantly evolve to include new types of mistakes that start to show up. Mistakes found during the checklist review that were not on the checklist should be recorded and considered for use in the next version. Technology, personnel, and methodology changes could all lead to new items being added to the checklist.

Inspections

Inspections are more structured than reviews. Fagan Inspections are one particular inspection methodology from which many others have been derived. They were defined by Michael Fagan in the 1970s based on his work at IBM, and are now part of the Fagan Defect-Free Process. You can find out more about his process at www.mfagan.com.

A Fagan Inspection follows these steps:

Planning
Overview
Preparation
Meeting
Rework
Follow-Up
Causal Analysis

The inspection meeting is limited to four people, with each session taking no more than two hours. Larger work should be broken up into multiple sessions. These guidelines are based on data that shows a decline in the effectiveness of the inspection if these limits are exceeded. If you don't know your inspection rates, such as pages per hour or lines of code per hour , measure them for the first 10 or so inspections you do. Then use those results to calculate how many sessions are needed for any future inspections.

In the Fagan Inspection method, each participant plays a specific role in the inspection of the material. The Moderator, who is not the Author, organizes the inspection and checks that the materials to be inspected satisfy predefined criteria. As with the checklist reviews, you will need to establish these criteria for different items that you will be inspecting. Once the criteria are met, the Moderator schedules the review meeting, plus an "overview" session that takes place prior to the review. This is to discuss the scope and intent of the inspection with the participants. Participants may also have questions that can be answered here or soon after the meeting. Typically there should be two working days between the overview and the inspection meeting. This is to give reviewers adequate preparation time.

Each of the inspectors is assigned a role to play in the inspection meeting. The Reader is expected to paraphrase the material being inspected. The idea is to communicate any implied information or behavior that the Reader interprets to see if it matches the Author's intended function. For example, here is a line of code to read:

 LoadLevel(level[17], highRes, 0);

You could just say "Call LoadLevel with level seventeen, high res and zero." A better reading for inspection purposes would be to say "Call LoadLevel without checking the return value. Pass the level information using a constant index of seventeen, the stored value of highRes and a hard-coded zero." This second reading raises the following potential issues:

The return value of LoadLevel is not checked. Should it return a value to indicate success, or a level number to verify the level you intended to load did in fact get loaded?
Using a constant index for the level number may not be a good practice. Should the level number come from a value passed to the routine that this code belongs to or should the number 17 be referenced by a more descriptive defined constant such as HAIKUDUNGEON in case something in the future causes the level numbering to be re-ordered?
The value of 0 provides no explanation about its function or the parameter it is being assigned to.

You can get similar results from reading test cases. Having another person try to literally understand your test steps word for word may not turn out as you intended.

The Tester does not have to be the person from the test team. This is a role where the person questions things like whether the material being inspected is internally consistent or consistent with any project documents it is based on. It is also good if the Tester can foresee how this material will fit in with the rest of the project and how it would potentially be tested .

A Recorder takes detailed notes about the issues raised in the inspection. The Recorder is a second role that can be taken on by any of the four people involved. The Reader is probably not the best choice for Recorder and you may find that it works best if the Moderator accepts the Recorder role. The Moderator also helps keep the meeting on track by limiting discussions to the material at hand.

Throughout the meeting the participants should not feel confined by their roles. They need to become engaged in discussions of potential issues or how to interpret the material. A successful inspection is one that invites the "Phantom Inspector." This is neither an actual person nor a supernatural manifestation. Rather, it is a term to explain the source of extra issues that are raised by the inspection team coming together and feeding off of each other's roles.

Once the meeting has concluded, the Moderator determines whether any rework is required before the material can be accepted. He continues to work with the Author to follow up on issues until they are closed. An additional inspection may be necessary, based on the volume or complexity of the changes.

The final step of this process involves causal analysis of the product (inspected item) faults and any inspection process (overview, preparation, meeting, and so on) problems. Issues can be discussed, such as how the overview could have been more helpful, or requiring stricter compiler flags to be set that could flag certain code defects prior to submitting the code for inspection.