Certification Objective 9.03: Establishing Quality and Performance Metrics

Quality and performance metrics help to measure whether the application meets the planned expectations. Quality metrics is the examination of the quality of the code and the application. The goal of software quality is to reduce the time spent reworking code, either from requirements changes, design changes, or debugging errors. These metrics are the summary of the correct collection of requirements the correctness of the program logic and testing. Fixed date mindset is one method to enforce quality in an application. Bug convergence and zero bug bounce are two techniques for measuring the quality state of the solution and predicting the release date. Along with testing, managing the project effectively and determining organization performance goals also help with the quality of the product. The return on investment is the examination of how well the project went and how existing resources were utilized.

Fixed Ship Date Mindset

Having a fixed ship date mindset means having an attitude about schedules that helps the team meet due dates and establish the date the software will release to commercial users. It views a project’s projected ship date as both realistic and unchangeable. The concept of a fixed ship date does not mean the date cannot change, but it does mean any variation from the planned schedule needs to be justified. The idea is to prevent scope creep—the tendency to add additional features over the course of development. Using a fixed ship date establishes a high-level goal for the development team to achieve, and it forces justification for additional features or design changes, which can cause the date to slip.

Bug Convergence

Bug convergence is the point where the rate of bugs resolved exceeds the rate of bugs found. This helps to provide an indicator to customers and key stakeholders of whether the project is on track. Typically, many bugs are found early in the testing process, and the rate of bugs found exceeds the number of bugs the team is able to resolve each day. Over time, however, fewer bugs are found on average every day, forming a downward trend line. If bug convergence actually occurs significantly later than the targeted bug convergence milestone date, then the team knows that the release date is at risk. Figure 9-1 is an example of bug convergence; and it details three things on the left side: days, number of new bugs found, and number of bugs resolved. Looking at the graph, you can see that the two different-colored lines will meet at day 8, showing when bug convergence happened.

Figure 9-1: Bug convergence

On The Job

Note that if reported and fixed bug counts are not converging over a period of time, this is a warning sign that something serious is wrong with the project. Convergence fails to occur when as many new bugs are being reported as are being fixed. There are various explanations of what could be causing this, but the team must troubleshoot this as a serious issue.

Zero Bug Bounce

Zero bug bounce is when there are no active bugs, usually followed by a “bounce” in the number of active bugs. A “bounce” occurs when the product has finished testing a build cycle. This cycle could take place daily or on some other time interval. Zero bug bounce is a positive indicator for a project, indicating that development has caught up with the backlog of active bugs needing resolution. It is a sign that the quality of the builds are improving.

Evaluating Project Control

The evaluation of project control is the examination of the cost based on budget, project progress, and team performance. Many of these tasks can be performed by using traditional project management technologies and methods. The goal is to determine if the team knows what it is building, which goes back to planning the project. Effective planning reduces risks in such areas as defects, project costs, timing, and overall project quality. It sets up the project for success by providing more predictability, which in turn gives the team more control over outcomes. Effective project controls help to detect bugs early on, and costs can be controlled by examining what resources will be needed, when they will be needed, and how they will be utilized.

Evaluating Organizational Performance

Performance management includes activities to ensure that goals are met in an effective and efficient manner. It can focus on performance of the organization or department, or on the effectiveness of processes to build a solution. This is the examination of organizational members and processes. Just because a process has been used for years does not necessarily mean it should be used; it should be used only if it contributes directly to the preferred results of the organization. Performance management reminds us that being busy is not the same as producing results. It reminds us that training, strong commitment, and lots of hard work alone are not results. Further, evaluation performance should not end with the completion of the project. After the project has been completed, it is a good practice to review and discuss how the organization can work better together and what processes worked as they should.

Identifying Return on Investment

Identifying the return on investment is analyzing the current situation and determining whether making the software changes saves money or increases revenue. The savings can be in the form of reduced maintenance costs, fewer employees, or less hardware. When looking at these areas, it helps to ask the following questions:

• Does the solution automate current manual tasks, and how much time does it save?

• Does the solution consolidate or remove outdated hardware or software?

• Can you get rid of unnecessary maintenance contracts?

• Does the solution require fewer developers or support staff to maintain than before?

• Does the solution create new revenue?

These are some examples of what to look for, but many other areas of possible savings can be included in calculating the return on investment.