The elements of a software quality system

1.2 The elements of a software quality system

There are two goals of the software quality system (SQS). The first goal is to build quality into the software from the beginning. This means assuring that the problem or need to be addressed is clearly and accurately stated, and that the requirements for the solution are properly defined, expressed, and understood. Nearly all the elements of the SQS are oriented toward requirements validity and satisfaction.

In order for quality to be built into the software system from its inception, the software requirements must be clearly understood and documented. Until the actual requirements, and the needs of the user that they fulfill, are known and understood, there is little likelihood that the user will be satisfied with the software system that is delivered. Whether we know them all before we start, or learn some of them as we go, all requirements must be known and satisfied before we are through. Further discussion of requirements is provided in Chapters 6 and 10.

The second goal of the SQS is to keep that quality in the software throughout the software life cycle (SLC). In this chapter, the 10 elements of the SQS are introduced and their contributions to the two goals indicated.

The 10 elements of the SQS are as follows:

1. Standards;

2. Reviewing;

3. Testing;

4. Defect analysis;

5. Configuration management (CM);

6. Security;

7. Education;

8. Vendor management;

9. Safety;

10. Risk management.

While each element can be shown to contribute to both goals, there are heavier relationships between some elements and one or the other of the two goals. These particular relationships will become obvious as each element is discussed in the chapters that follow.

Every SLC model has divisions, or periods of effort, into which the work of developing and using the software is divided. These divisions or periods are given various names depending on the particular life-cycle paradigm being applied. For this discussion, the following periods of effort, together with their common names, are defined:

• Recognition of a need or problem (e.g., concept definition);

• Definition of the software solution to be applied (e.g., requirements definition);

• Development of the software that solves the problem or satisfies the need (e.g., design and coding);

• Proving that the solution is correct (e.g., testing);

• Implementing the solution (e.g., installation and acceptance);

• Using the solution (e.g., operation);

• Improving the solution (e.g., maintenance);

Regardless of their names, each division represents a period of effort directed at a particular part of the overall life cycle. They may be of various lengths and be applied in various sequences, but they will all exist in successful projects. Some newer life-cycle models or approaches for small to medium-sized projects (e.g., extreme programming, the rational unified process, and agile development) appear to circumvent some of these efforts. In reality, they merely reorder, reiterate, or resequence them.

There are also associations between certain elements and the various divisions or periods of the SLC. Again, most of the elements support most of the SLC, but certain elements are more closely associated with particular periods than with others.

Figure 1.1 displays the 10 elements as a cube supporting the goals of software quality and the periods of the SLC with which each element is most closely associated.

Figure 1.1: Quality tasks, life-cycle periods, and goals.

1.2.1 Standards

Software is becoming a science. The old days of free-form creativity in the development of software are gradually giving way to more controlled and scientific approaches. As some writers have said, "Software is moving from an arcane art to a visible science."

As implied by Figure 1.2, the standards manual can have inputs from many sources. Standards are intended to provide consistent, rigorous, uniform, and enforceable methods for software development and operation activities. The development of standards, whether by professional societies such as the Institute of Electrical and Electronics Engineers (IEEE), international groups such as International Organization for Standardization/International Electrotechnical Commission Joint Technical Committee One (ISO/IEC JTC1), industry groups, or software development organizations for themselves, is recognizing and furthering that movement.

Figure 1.2: Standards sources.

Standards cover all aspects of the SLC, including the very definition of the SLC itself. More, probably, than any of the other elements, standards can govern every phase of the life cycle. Standards can describe considerations to be covered during the concept exploration phase. They can also specify the format of the final report describing the retirement of a software system that is no longer in use.

Standards come into being for many reasons. They might document experience gained in the day-to-day running of a computer center, and the most efficient methods to be used. Laws and government regulations often impose standard procedures on business and industry. Industries can band together to standardize interfaces between their products such as in the communications areas. Contracts often specify standard methods of performance. And, in many cases, standards arise out of good common sense.

Whether a standard comes from within a company, is imposed by government, or is adopted from an industry source, it must have several characteristics. These include the following:

• Necessity. No standard will be observed for long if there is no real reason for its existence.

• Feasibility. Common sense tells us that if it is not possible to comply with the tenets of a standard, then it will be ignored.

• Measurability. It must be possible to demonstrate that the standard is being followed.

Each of these characteristics supports the total enforceability of the standard. An unenforceable standard is of no use to anyone.

Software standards should be imposed so that the producer of a software product or component can pay attention to the technical aspects of the task, rather than to the routine aspects that may be the same for every task. Standards, such as those for document formats, permit the producer to concentrate on technical issues and content rather than format or layout details.

Standards, while worthwhile, are less than fully effective if they are not supported by policies that clearly indicate their imposition. It should be the intent of responsible management to see that they are followed and enforced. Specific practices for standard implementation are often useful. In this way, adherence to the standard may be more uniform.

Lastly, not everything must be standardized. Guidelines that call out the preferred methods or approaches to many things are fully adequate. A set of standards that covers every minute aspect of an organization's activity can lose respect simply from its own magnitude. Competent and comprehensive guidelines give each person some degree of freedom in those areas where specific methods or approaches are not absolutely necessary. This leaves the standards to govern those areas where a single, particular way of doing business is required.

1.2.2 Reviewing

Reviews permit ongoing visibility into the software development and installation activities.

Product reviews, also called technical reviews, are formal or informal examinations of products and components throughout the development phases of the life cycle. They are conducted throughout the software development life cycle (SDLC). Informal reviews generally occur during SDLC phases, while formal reviews usually mark the ends of the phases. Figure 1.3 illustrates this point.

Figure 1.3: SDLC reviews.

Informal reviews include walk-throughs and inspections. Walkthroughs are informal, but scheduled, reviews, usually conducted in and by peer groups. The author of the subject component-a design specification, test procedure, coded unit, or the like-walks through his or her component, explaining it to a small group of peers. The role of the peers is to look for defects in or problems with the component. These are then corrected before the component becomes the basis for further development.

Inspections are a more structured type of walk-through. Though the basic goal of an inspection-removal of defects-is the same as that of the walk-through, the format of the meeting and the roles of the participants are more strictly defined, and more formal records of the proceedings are prepared.

Process reviews may be held at any time. The purpose of a process review is to examine the success of the software process in effect. Data for the review is collected in the technical reviews and is usually based on defects identified by the technical reviews. Opportunities for improvements to the current process are sought. Management reviews are specialized process reviews, done on behalf of senior management, to examine project status and effective use of resources based on the current process.

Also included within the quality control activity of reviewing are audits. Audits are examinations of components for compliance with a content and format specification or for consistency with or comparison to a predecessor. An in-process audit of the unit development folder (UDF)-also called the software development file in some organizations-is usually informal. It compares the content and status of the UDF against standards governing the preparation and maintenance of the UDF. Its goal is to ascertain the UDFs are being used as required.

The physical audit (PA), often included as a part of the CM process, is an example of a formal audit. It compares the final form of the code against the final documentation for that code. The goal of the physical audit is to assure that the two products, documentation and code, are in agreement before being released to the user or customer. Another formal audit is the functional audit. The functional audit (FA), again often a CM responsibility, compares the test results with the currently approved requirements to assure that all requirements have been satisfied.

1.2.3 Testing

Tests provide increasing confidence and, ultimately, a demonstration that the software requirements are being satisfied. Test activities include planning, design, execution, and reporting. Figure 1.4 presents a simple conceptual view of the testing process. The basic test process is the same, whether it is applied to system testing or to the earliest module testing.

Figure 1.4: Simplified test process.

Test planning begins during the requirements phase and parallels the requirements development. As each requirement is generated, the corresponding method of test for that requirement should be a consideration. A requirement is faulty if it is not testable. By starting test planning with the requirements, nontestability is often avoided. In the same manner that requirements evolve and change throughout the software development, so, too, do the test plans evolve and change. This emphasizes the need for early, and continuing, CM of the requirements and test plans.

Test design begins as the software design begins. Here, as before, a parallel effort with the software development is appropriate. As the design of the software takes form, the test cases, scenarios, and data are developed that will exercise the designed software. Each test case also will include specific expected results so that a pass-fail criterion is established. As each requirement must be measurable and testable, so must each test be measurable. A test whose completion is not definitive tells little about the subject of the test. Expected results give the basis against which the success or failure of the test is measured.

Actual testing begins with the debugging and early unit and module tests conducted by the programmer. These tests are usually informally documented (perhaps by notations in the UDF) and are not closely monitored by the software quality practitioner since they are frequently experimental and meant to help the programmer in his or her day-to-day software generation. Formal test execution generally begins with integration tests in which modules are combined into subsystems for functional testing. In larger systems, it is frequently advisable to begin formal testing at the module level after the programmer has completed his or her testing and is satisfied that the module is ready for formal testing.

An important aspect of complete testing is user acceptance testing. In these, usually last, tests, the concentration is on actual, delivered functionality and usability in the user's environment.

Test execution requires the use of detailed test procedures. These are step-by-step directions that tell the test conductor exactly what to do as the test is run. Every action, input, expected output, and response should be documented so that the test conductor is not put into the position of making test design decisions while the test is being run. Preparation of the test procedures is begun during the design phase and completed during the coding and debugging activities. By the time the coding phase is complete, all preparations for the formal testing activities should also be in place. Test cases, scenarios, data, and procedures, together with expected results and completion criteria, should be ready to be applied from module testing (if included on the particular project) through qualification and acceptance tests.

Test reports document the actual results of the testing effort as it progresses. For each test that is run, a report of the expected results, actual results, and the conclusions of the test conductor concerning success of the test should be prepared. Included in the report are the anomalies that were found and recommended action with respect to them. Errors, defects, faults, questionable or unexpected results, and any other nonpredicted outcomes are recorded and assigned for action. Once the anomaly has been addressed, the test, or an appropriate portion thereof, will be rerun to show that the defect has been corrected. As the tests progress, so do the levels of detail of the test reports, until the final acceptance test report is prepared that documents the fitness of the software system for use in its intended environment.

1.2.4 Defect analysis

Defect analysis is the combination of defect detection and correction, and defect trend analysis. Defect detection and correction, together with change control, presents a record of all discrepancies found in each software component. It also records the disposition of each discrepancy, perhaps in the form of a software problem report or software change request.

As shown in Figure 1.5, each needed modification to a software component, whether found through a walk-through, review, test, audit, operation, or other means is reported, corrected, and formally closed. A problem or requested change may be submitted by anyone with an interest in the software. The situation will be verified by the developers, and the CM activity will agree to the change. Verification of the situation is to assure that the problem or need for the change actually exists. CM may wish to withhold permission for the change or delay it until a later time; perhaps because of concerns such as interference with other software, schedule and budget considerations, the customer's desires, and so on. Once the change is completed and tested, it will be reported by CM to all concerned parties, installed into the operational software by the developers or operations staff, and tested for functionality and compatibility in the full environment.

Figure 1.5: Typical change procedure.

This procedure is required for the ongoing project to make sure that all defects found are properly fixed and closed. It also serves future projects by providing a means for feeding defect information back into the development life cycle and modifying the software development process so that future occurrences of certain defects are reduced. Figure 1.6 places the change procedure into the larger picture of development process analysis and improvement.

Figure 1.6: Development process improvement.

A running record of defects, their solutions, and status is provided by the defect trend analysis effort. (The actual changes are made according to the configuration control process.) As previously mentioned, the record of defects and their solutions can serve to do the following:

• Prevent defects from remaining unsolved for inappropriate lengths of time;

• Prevent unwarranted changes;

• Point out inherently weak areas in the software;

• Provide analysis data for development process evaluation and correction;

• Provide warnings of potential defects through analysis of defect trends.

Formal recording and closure procedures applied to defects are insufficient if corresponding reports are not generated so that project management has visibility into the progress and status of the project. Regular reports of defect detection and correction activity keep management apprised of current defect areas and can warn of potential future trouble spots. Further, analysis of ongoing defect and change reports and activities provide valuable insight into the software development process and enhance the software quality practitioner's ability to suggest error-avoidance and software development process modification.

1.2.5 Configuration management

CM is a three-fold discipline. Its intent is to maintain control of the software, both during development and after it is put into use and changes begin.

As shown in Figure 1.7, CM is, in fact, three related activities: identification, control, and accounting. If the physical and functional audits are included as CM responsibilities, there are four activities. Each of the activities has a distinct role to play. As system size grows, so does the scope and importance of each of the activities. In very small, or one-time use, systems, CM may be minimal. As systems grow and become more complex, or as changes to the system become more important, each activity takes on a more definite role in the overall management of the software and its integrity. Further, some CM may be informal for the organization itself, to keep track of how the development is proceeding and to maintain control of changes, while others will be more formal and be reported to the customer or user.

Figure 1.7: CM activities.

Configuration identification is, as its name implies, the naming, and documentation, of each component (document, unit, module, subsystem, and system) so that at any given time, the particular component of interest can be uniquely identified. This is important when documenting, testing, changing, or delivering software to the customer; in other words, throughout the entire SLC. Unless it is known which specific version or component of the software is being affected, (i.e., coded, changed, tested) the software is out of control. Tests may be run on the wrong version of the code, changes may be made to an obsolete version of a document, or a system composed of the wrong versions of the various components may be delivered to the user or customer.

Configuration control is that activity that prevents unauthorized changes to any software product. Early in the SLC, documentation is the primary product. Configuration control takes on an increasingly formal role as the documents move from draft to final form. Once published, any changes to the documents are formally processed so that capricious, unnecessary, or unapproved changes are not made. As the life cycle moves into the coding, testing, and operation and maintenance phases, changes to either documents or code are closely controlled. Each change is verified for necessity and correctness before approval for insertion, so that control of the software can be maintained.

Configuration accounting keeps track of the status of each component. The latest version or update of each software component is recorded. Thus, when changes or other activities are necessary with respect to the component, the correct version of the component can be located and used. Each new edition of a document, each new assembly or compilation of the code, each new build of the software system is given a new specific identifier (through configuration identification) and recorded. All changes to that version or edition of a component are also referenced to it so that, if necessary, the history of activity with respect to any component can be recreated. This might be necessary in the loss of the current version or to return to a previous version for analysis or other purposes. Ultimately, the deliverable version, or build, of the software is created. Configuration accounting helps manage the builds and make the build process repeatable.

One last point should be made. That is, for very long term or long-lived systems, the development environment itself may need to be configuration managed. As operating systems, platforms, languages, and processes evolve over time, it may not be possible to recreate a given system without the original development environment and its components.

1.2.6 Security

Security activities are applied both to data and to the physical data center itself. These activities are intended to protect the usefulness of the software and its environment.

The highest quality software system is of no use if the data center in which it is to be used is damaged or destroyed. Such events as broken water pipes, fire, malicious damage by a disgruntled employee, and storm damage are among the most common causes of data center inoperability. Even more ominous is the rising incidence of terrorist attacks on certain industries and in various countries, including our own, around the world.

Another frequent damager of the quality of output of an otherwise high-quality software system is data that has been unknowingly modified. If the data on which the system is operating has been made inaccurate, whether intentionally or by accident, the results of the software will not be correct. To the user or customer, this appears to be inadequate software.

Additionally, though not really a software quality issue per se, is the question of theft of data. The security of stored or transmitted data is of paramount concern in most organizations. From the theft of millions of dollars by interception of electronic funds transfers to an employee who just changes personnel or payroll records, data security is a major concern.

Finally, the recent onslaught of hackers and software attackers and the burgeoning occurrences of viruses also need to be considered. These threats to software quality must be recognized and countered.

The role of the software quality practitioner is, again, not to be the policeperson of the data or to provide the data or data center security. The software quality practitioner is responsible for alerting management to the absence, or apparent inadequacy, of security provisions in the software. In addition, the software quality practitioner must raise the issue of data center security and disaster recovery to management's attention.

1.2.7 Education

Education assures that the people involved with software development, and those people using the software once it is developed, are able to do their jobs correctly.

It is important to the quality of the software that the producers be educated in the use of the various development tools at his or her disposal. A programmer charged with writing object-oriented software in C++ cannot perform well if the only language he or she knows is Visual Basic. It is necessary that the programmer be taught to use C++ before beginning the programming assignment. Likewise, the use of operating systems, data modeling techniques, debugging tools, special workstations, and test tools must be taught before they can be applied beneficially.

The proper use of the software once it has been developed and put into operation is another area requiring education. It this case, the actual software user must be taught proper operating procedures, data entry, report generation, and whatever else is involved in the effective use of the software system's capabilities.

The data center personnel must be taught the proper operating procedures before the system is put into full operation. Loading and initializing a large system may not be a trivial task. Procedures for recovering from abnormal situations may be the responsibility of data center personnel. Each of the many facets of operating a software system must be clear so that the quality software system that has been developed may continue to provide quality results.

The software quality practitioner is not usually the trainer or educator. These functions are normally filled by some other group or means. The role of the software quality practitioner is, as always, to keep management attention focused on the needs surrounding the development and use of a quality software system. In this case, the software quality practitioner is expected to monitor the requirements for, and the provision of, the education of the personnel involved in the SLC.

Lastly, the support personnel surrounding software development must know their jobs. The educators, CM and software quality practitioners, security and database administrators, and so on must be competent to maintain an environment in which quality software can be built, used, and maintained.

1.2.8 Vendor management

When software is purchased, the buyer must be aware of, and take action to gain confidence in, its quality. Not all purchased software can be treated in the same way, as will be demonstrated here. Each type of purchased software will have its own software quality system approach, and each must be handled in a manner appropriate to the degree of control the purchaser has over the development process used by producer. The following are three basic types of purchased software:

1. Off-the-shelf;

2. Tailored shell;

3. Contracted.

Off-the-shelf software is the package we buy at the store. Microsoft Office, Adobe Photoshop, virus checkers, and the like are examples. These packages come as they are with no warrantee that they will do what you need to have done. They are also almost totally outside the buyer's influence with respect to quality.

The second category may be called the tailored shell. In this case, a basic, existing framework is purchased and the vendor then adds specific capabilities as required by the contract. This is somewhat like buying a stripped version of a new car and then having the dealer add a stereo, sunroof, and other extras. The only real quality influence is over the custom-tailored portions.

The third category is contracted software. This is software that is contractually specified and provided by a third-party developer. In this case, the contract can also specify the software quality activities that the vendor must perform and which the buyer will audit. The software quality practitioner has the responsibility in each case to determine the optimum level of influence to be applied, and how that influence can be most effectively applied. The purchaser's quality practitioners must work closely with the vendor's quality practitioners to assure that all required steps are being taken.

Attention to vendor quality practices becomes extremely important when the developer is off-shore or remote, such as in India, Nepal, or another country.

1.2.9 Safety

As computers and software grow in importance and impact more and more of our lives, the safety of the devices becomes a major concern. The literature records overdoses of medicines, lethal doses of radiation, space flights gone astray, and other catastrophic and near-catastrophic events. Every software project must consciously consider the safety implications of the software and the system of which it is a part. The project management plan should include a paragraph describing the safety issues to be considered. If appropriate, a software safety plan should be prepared.

1.2.10 Risk management

There are several types of risk associated with any software project. Risks range from the simple, such as the availability of trained personnel to undertake the project, to more threatening, such as improper implementation of complicated algorithms, to the deadly, such as failure to detect an alarm in a nuclear plant. Risk management includes identification of the risk; determining the probability, cost, or threat of the risk; and taking action to eliminate, reduce, or accept the risk. Risk and its treatment is a necessary topic in the project plan and may deserve its own risk management plan.