Stage 4: Execute the Test

After the build process is complete, the test team performs a Build VerificationTest (BVT) before accepting the build and submitting it for further testing. After a successful BVT, the test team conducts a BAT, or smoke test, to verify that the build is stable enough for further testing. Frequently, these two tests are combined intoa single effort.

If the BAT test is successful, the test team accepts the build and begins design testing. The test team begins design testing by performing the various tests documentedin the DTCs. These tests are run on each major build, and the results are recorded and verified . Any deviations from the expected results are documented and are tracked through a bug-tracking tool.

Tracking Bugs

Any bug-tracking tool can be used to record bugs during the implementation phase. Each member of the test team needs specific permissions to access the bug-tracking tool. During testing, all members of the team should be able to record newly discovered bugs. To maintain good control over the status of bugs, only the test lead,test staff, or project managers should have the proper permissions to close bugs.

Bugs can be found in documentation, procedures, and other areas in addition to the system itself. All issues with the application or its supporting infrastructure should be captured in the bug-tracking system for resolution. The bug-tracking system becomes the team s to do list and can be used to quickly gauge the status of the project.

A tester logs the bugs in the tracking system by using a recognizable description and comprehensive detail. All bugs should contain the following information as a minimum:

• Error messages (recorded exactly as they appear on the screen or in log files)

• Version number of the build where the bug was found

• Exact, repeatable steps to reproduce the bug

• A summary of any unique configuration or environmental factors in place during the test

In addition, a tester can include a suggested resolution if the tester knows a likely resolution.

After developers fix a bug, they change its status from Open to Fixed. If they believe that a bug should not be fixed, they change its status to Duplicate, By Design, Not Repro, or Won t Fix. Bugs that are no longer listed as Open or Active must be given final resolution by the test team. The following list provides examples of how the test team resolves a bug by using the tracking tool:

• Bugs with the Resolution field marked as Fixed are assigned back to the originator of the bug to perform regression testing on the new build.

• Bugs with the Resolution field marked as Duplicate, By Design, Not Repro, or Won t Fix are closed by the test team if the team agrees with the resolution.

• If the test team strongly disagrees with the resolution, the test team changes the Assign To field to Active and discusses in the triage meeting whether the bug should be reactivated. If the bug fails regression, the test team will reactivateit by changing the status to Active. All active bugs will be reviewed again in the triage meeting. For more information about triage meetings, see Triage Process later in this chapter.

Assigning Severity to Bugs

Table 12.5 lists guidelines that can be used to assign severity to bugs.

Table 12.5: Bug Severity Guidelines

Severity	Most Common Types	Conditions Required
1	Bug blocks build or further testing of a feature. Bug affects further testing of a different feature that is being tested in parallel.	System does not work. User cannot even begin to use significant parts of the system.
2	Steps defined in the documentation are not viable . Results or behavior of a function or process contradicts expected results (as documented in functional specification). Results or behavior of a function or process contradicts logically expected results. Documented functionality is missing (in this case, test is blocked). Documentation is missing or inadequate.	If the following conditions are met, severity is 2: User has no simple workaround to mend situation. User cannot easily figure out workaround. The system cannot meet primary business requirements. If the preceding conditions are not met, severity is 3.
3	Function or process is broken. Results or behavior of a function or process contradicts expected results (as documented in functional specification). Results or behavior of a function or process contradicts logically expected results. Minor documentation errors and inaccuracies. Text is misspelled .	If the following conditions are met, severity is 3: User has a simple workaround to mend situation. User can easily figure out workaround. Bug does not cause a bad user experience. Primary business requirements are still functional. Bug does not block a significant number of other test cases. If the preceding conditions are not met, severity is 2.
4	Suggestions. Future enhancements.	Clearly not a product bug for this version.

Triage Process

During the test phase of a migration project, triage meetings are held to reviewthe bugs reported by the test team. Triage meetings are scheduled to occur regularly, usually three times a week during the first test pass and then daily as efforts approach the test completion deadline. Typically, the following people attend the triage meeting:

• Project manager

• Test manager

• Development manager

The test lead schedules and facilitates the triage meeting. Other team membersmay be brought to the meeting to discuss specific technical issues as needed.

The typical activities performed in a triage meeting and afterward are:

1. The test lead describes each newly recorded bug in the tracking system.

2. A common understanding of the problem is reached among members of the group .

3. The severity of each new bug is reviewed .Because severity is tied to the functionality delivered by the application, severity should not be redefined in triage meetings unless a group decision is also made to change the delivered functionality of the system.

4. Each bug is assigned a priority (high, medium, or low) based on factors suchas severity and developer availability.

5. Each bug and an action plan derived to facilitate resolution of the bug are assigned to the appropriate team member for resolution.

6. The test lead updates the tracking system based on the decisions made duringthe triage meeting.

7. Team members work on resolving the assigned bugs and update the resolution details in the bug-tracking system.

8. Testers regress the resolved bugs. If the test was successful, the documentsare updated according to the change control process and each bug is closed; otherwise , bugs are reactivated for further investigation.

9. The development team fixes the design bugs that the test team reported. Afterthe development team fixes these bugs, they are assigned back to the test team members who opened them originally. The test team then retests the implementation and closes each bug as it is fixed. After all bugs are fixed, a newer buildis created for retesting.

Generating a Test Report

When the test team completes the test cases, it prepares a test report to list the open bugs and to summarize system performance, system management findings, and suggested practices. The bugs are listed according to priority. The test report also highlights the impact on business if bugs are not fixed, along with a recommendation on whether to proceed with production.

For more information about test reports , see Reporting and Release Processes later in this chapter.