Criteria for Model Evaluation

For reliability models, in 1984 a group of experts (Iannino et al., 1984) devised a set of criteria for model assessment and comparison. The criteria are listed as follows , by order of importance as determined by the group :

  • Predictive validity: The capability of the model to predict failure behavior or the number of defects for a specified time period based on the current data in the model.
  • Capability: The ability of the model to estimate with satisfactory accuracy quantities needed by software managers, engineers , and users in planning and managing software development projects or controlling change in operational software systems.
  • Quality of assumptions: The likelihood that the model assumptions can be met, and the assumptions' plausibility from the viewpoint of logical consistency and software engineering experience.
  • Applicability: The model's degree of applicability across different software products ( size , structure, functions, etc.).
  • Simplicity: A model should be simple in three aspects: (1) simple and inexpensive to collect data, (2) simple in concept and does not require extensive mathematical background for software development practitioners to comprehend, and (3) readily implemented by computer programs.

From the practitioner's point of view and with recent observations of software reliability models, we contend that the most important criteria are predictive validity, simplicity, and quality of assumptions, in that order of importance. Capability and applicability are less significant. As the state of the art is still maturing and striving to improve its most important objective (predictive accuracy), the extra criteria of demanding more functions (capability) for multiple environments (applicability) seems burdensome. Perhaps the accuracy of software reliability models can best be summarized as follows: Some models sometimes give good results, some are almost universally awful , and none can be trusted to be accurate at all times (Brocklehurst and Littlewood, 1992). A model with good predictive validity but poor capability and narrow applicability is certainly superior to one with good capability and wide applicability but with very poor ability to predict.

In contrast to the order of importance determined by the 1984 group, we think that simplicity is much more important, second only to predictive validity. Experts in software reliability models are usually academicians who are well versed in mathematics and statistics. Many modeling concepts and terminologies are outside the discipline of computer science, let alone easy to comprehend and implement by software developers in the industry. As mentioned earlier, some reliability models have not been tested and used in real-life development projects simply because they are not understandable. Simplicity, therefore, is a key element in bridging the gap between the state of the art and the state of practice in software reliability modeling.

The quality of the assumptions is also very important. Early models tend to have restrictive and unrealistic assumptions. More recent models tend to have more realistic assumptions. Better assumptions make the model more convincing and more acceptable by software practitioners; they also lead to better predictive validity.

What Is Software Quality?

Software Development Process Models

Fundamentals of Measurement Theory

Software Quality Metrics Overview

Applying the Seven Basic Quality Tools in Software Development

Defect Removal Effectiveness

The Rayleigh Model

Exponential Distribution and Reliability Growth Models

Quality Management Models

In-Process Metrics for Software Testing

Complexity Metrics and Models

Metrics and Lessons Learned for Object-Oriented Projects

Availability Metrics

Measuring and Analyzing Customer Satisfaction

Conducting In-Process Quality Assessments

Conducting Software Project Assessments

Dos and Donts of Software Process Improvement

Using Function Point Metrics to Measure Software Process Improvements

Concluding Remarks

A Project Assessment Questionnaire

show all menu

Metrics and Models in Software Quality Engineering
Metrics and Models in Software Quality Engineering (2nd Edition)
ISBN: 0201729156
EAN: 2147483647
Year: 2001
Pages: 176
Similar book on Amazon © 2008-2017.
If you may any questions please contact us: