Defect Taxonomies

'Failure' was simply not a word that would ever cross the lips of Miss Evelyn Duberry, mainly because Evelyn, a haughty socialite with fire-red hair and a coltish gate, could pronounce neither the letters 'f' nor 'r' as a result of an unfortunate kissing gesture made many years earlier toward her beloved childhood parrot, Snippy.

— David Kenyon

Introduction

What is a taxonomy? A taxonomy is a classification of things into ordered groups or categories that indicate natural, hierarchical relationships. The word taxonomy is derived from two Greek roots: "taxis" meaning arrangement and "onoma" meaning name. Taxonomies not only facilitate the orderly storage of information, they facilitate its retrieval and the discovery of new ideas. Taxonomies help you:

  • Guide your testing by generating ideas for test design
  • Audit your test plans to determine the coverage your test cases are providing
  • Understand your defects, their types and severities
  • Understand the process you currently use to produce those defects (Always remember, your current process is finely tuned to create the defects you're creating)
  • Improve your development process
  • Improve your testing process
  • Train new testers regarding important areas that deserve testing
  • Explain to management the complexities of software testing
  Key Point

A taxonomy is a classification of things into ordered groups or categories that indicate natural, hierarchical relationships.

In his book Testing Object-Oriented Systems, Robert Binder describes a "fault model" as a list of typical defects that occur in systems. Another phrase to describe such a list is a defect taxonomy. Binder then describes two approaches to testing. The first uses a "non-specific fault model." In other words, no defect taxonomy is used. Using this approach, the requirements and specifications guide the creation of all of our test cases. The second approach uses a "specific fault model." In this approach, a taxonomy of defects guides the creation of test cases. In other words, we create test cases to discover faults like the ones we have experienced before. We will consider two levels of taxonomies—project level and software defect level. Of most importance in test design are the software defect taxonomies. But it would be foolish to begin test design before evaluating the risks associated with both the product and its development process.

Note that none of the taxonomies presented below are complete. Each could be expanded. Each is subjective based on the experience of those who created the taxonomies.


Project Level Taxonomies

SEI Risk Identification Taxonomy

The Software Engineering Institute has published a "Taxonomy-Based Risk Identification" that can be used to identify, classify, and evaluate different risk factors found in the development of software systems.

Table 15-1: The SEI Taxonomy-Based Risk Identification taxonomy.

Class

Element

Attribute

Product Engineering

Requirements

Stability

Completeness

Clarity

Validity

Feasibility

Precedent

Scale

Design

Functionality

Difficulty

Interfaces

Performance

Testability

Code and Unit Test

Feasibility

Testing

Coding/Implementation

Integration and Test

Environment

Product

System

Engineering Specialties

Maintainability

Reliability

Safety

Security

Human Factors

Specifications

Development Environment

Development Process

Formality

Suitability

Process Control

Familiarity

Product Control

Development System

Capacity

Suitability

Usability

Familiarity

Reliability

System Support

Deliverability

Management Process

Planning

Project Organization

Management Experience

Program Interfaces

Management Methods

Monitoring

Personnel Management

Quality Assurance

Configuration Management

Work Environment

Quality Attitude

Cooperation

Communication

Morale

Program Constraints

Resources

Schedule

Staff

Budget

Facilities

Contract

Types of Contract

Restrictions

Dependencies

Program Interfaces

Customer

Associate Contractors

Subcontractors

Prime Contractor

Corporate Management

Vendors

Politics

If, as a tester, you had concerns with some of these elements and attributes, you would want to stress certain types of testing. For example:

If you are concerned about:

You might want to emphasize:

The stability of the requirements

Formal traceability

Incomplete requirements

Exploratory testing

Imprecisely written requirements

Decision tables and/or state-transition diagrams

Difficulty in realizing the design

Control flow testing

System performance

Performance testing

Lack of unit testing

Additional testing resources

Usability problems

Usability testing

ISO 9126 Quality Characteristics Taxonomy

The ISO 9126 Standard "Software Product Evaluation—Quality Characteristics and Guidelines" focuses on measuring the quality of software systems. This international standard defines software product quality in terms of six major characteristics and twenty-one subcharacteristics and defines a process to evaluate each of these. This taxonomy of quality attributes is:

Table 15-2: The ISO 9126 Quality Characteristics taxonomy.

Quality Characteristic

Subcharacteristic

Functionality

(Are the required functions available in the software?)

Suitability

Accuracy

Interoperability

Security

Reliability

(How reliable is the software?)

Maturity

Fault tolerance

Recoverability

Usability

(Is the software easy to use?)

Understandability

Learnability

Operability

Attractiveness

Efficiency

(How efficient is the software?)

Time behavior

Resource behavior

Maintainability

(How easy is it to modify the software?)

Analyzability

Changeability

Stability

Testability

Portability

(How easy is it to transfer the software to another operating environment?)

Adaptability

Installability

Coexistence

Replaceability

Each of these characteristics and subcharacteristics suggest areas of risk and thus areas for which tests might be created. An evaluation of the importance of these characteristics should be undertaken first so that the appropriate level of testing is performed. A similar "if you are concerned about / you might want to emphasize" process could be used based on the ISO 9126 taxonomy.

These project level taxonomies can be used to guide our testing at a strategic level. For help in software test design we use software defect taxonomies.


Software Defect Taxonomies

In software test design we are primarily concerned with taxonomies of defects, ordered lists of common defects we expect to encounter in our testing.

Beizer s Taxonomy

One of the first defect taxonomies was defined by Boris Beizer in Software Testing Techniques. It defines a four-level classification of software defects. The top two levels are shown here.

Table 15-3: A portion of Beizer's Bug Taxonomy.

1xxx

Requirements

11xx

Requirements incorrect

12xx

Requirements logic

13xx

Requirements, completeness

14xx

Verifiability

15xx

Presentation, documentation

16xx

Requirements changes

2xxx

Features And Functionality

21xx

Feature/function correctness

22xx

Feature completeness

23xx

Functional case completeness

24xx

Domain bugs

25xx

User messages and diagnostics

26xx

Exception conditions mishandled

3xxx

Structural Bugs

31xx

Control flow and sequencing

32xx

Processing

4xxx

Data

41xx

Data definition and structure

42xx

Data access and handling

5xxx

Implementation And Coding

51xx

Coding and typographical

52xx

Style and standards violations

53xx

Documentation

6xxx

Integration

61xx

Internal interfaces

62XX

External interfaces, timing, throughput

7XXX

System And Software Architecture

71XX

O/S call and use

72XX

Software architecture

73XX

Recovery and accountability

74XX

Performance

75XX

Incorrect diagnostics, exceptions

76XX

Partitions, overlays

77XX

Sysgen, environment

8XXX

Test Definition And Execution

81XX

Test design bugs

82XX

Test execution bugs

83XX

Test documentation

84XX

Test case completeness

Even considering only the top two levels, it is quite extensive. All four levels of the taxonomy constitute a fine-grained framework with which to categorize defects.

At the outset, a defect taxonomy acts as a checklist, reminding the tester so that no defect types are forgotten. Later, the taxonomy can be used as a framework to record defect data. Subsequent analysis of this data can help an organization understand the types of defects it creates, how many (in terms of raw numbers and percentages), and how and why these defects occur. Then, when faced with too many things to test and not enough time, you will have data that enables you to make risk-based, rather than random, test design decisions. In addition to taxonomies that suggest the types of defects that may occur, always evaluate the impact on the customer and ultimately on your organization if they do occur. Defects that have low impact may not be worth tracking down and repairing.

Kaner, Falk, and Nguyen s Taxonomy

The book Testing Computer Software contains a detailed taxonomy consisting of over 400 types of defects. Only a few excerpts from this taxonomy are listed here.

Table 15-4: A portion of the defect taxonomy from Testing Computer Software.

User Interface Errors

Functionality

Communication

Command structure

Missing commands

Performance

Output

Error Handling

Error prevention

Error detection

Error recovery

Boundary-Related Errors

Numeric boundaries

Boundaries in space, time

Boundaries in loops

Calculation Errors

Outdated constants

Calculation errors

Wrong operation order

Overflow and underflow

Initial And Later States

Failure to set a data item to 0

Failure to initialize a loop control variable

Failure to clear a string

Failure to reinitialize

Control Flow Errors

Program runs amok

Program stops

Loops

IF, THEN, ELSE or maybe not

Errors In Handling Or Interpreting Data

Data type errors

Parameter list variables out of order or missing

Outdated copies of data

Wrong value from a table

Wrong mask in bit field

Race Conditions

Assuming one event always finishes before another

Assuming that input will not occur in a specific interval

Task starts before its prerequisites are met

Load Conditions

Required resource not available

Doesn't return unused memory

Hardware

Device unavailable

Unexpected end of file

Source And Version Control

Old bugs mysteriously reappear

Source doesn't match binary

Documentation

None

Testing Errors

Failure to notice a problem

Failure to execute a planned test

Failure to use the most promising test cases

Failure to file a defect report

Binder s Object Oriented Taxonomy

Robert Binder notes that many defects in the object-oriented (OO) paradigm are problems using encapsulation, inheritance, polymorphism, message sequencing, and state-transitions. This is to be expected for two reasons. First, these are cornerstone concepts in OO. They form the basis of the paradigm and thus will be used extensively. Second, these basic concepts are very different from the procedural paradigm. Designers and programmers new to OO would be expected to find them foreign ideas. A small portion of Binder's OO taxonomy is given here to give you a sense of its contents:

Table 15-5: A portion of Binder's Method Scope Fault Taxonomy.

Method Scope

 

Fault

Requirements

 

Requirement omission

Design

Abstraction

Low Cohesion

Refinement

Feature override missing

Feature delete missing

Encapsulation

Naked access

Overuse of friend

Responsibilities

Incorrect algorithm

Invariant violation

Exceptions

Exception not caught

Table 15-6: A portion of Binder's Class Scope Fault Taxonomy.

Class Scope

 

Fault

Design

Abstraction

Association missing or incorrect

Inheritance loops

Refinement

Wrong feature inherited

Incorrect multiple inheritance

Encapsulation

Public interface not via class methods

Implicit class-to-class communication

Modularity

Object not used

Excessively large number of methods

Implementation

Incorrect constructor

Note how this taxonomy could be used to guide both inspections and test case design. Binder also references specific defect taxonomies for C++, Java, and Smalltalk.

Whittaker s How to Break Software Taxonomy

James Whittaker's book How to Break Software is a tester's delight. Proponents of exploratory testing exhort us to "explore." Whittaker tells us specifically "where to explore." Not only does he identify areas in which faults tend to occur, he defines specific testing attacks to locate these faults. Only a small portion of his taxonomy is presented:

Table 15-7: A portion of Whittaker's Fault Taxonomy.

Fault Type

Attack

Inputs and outputs

Force all error messages to occur

Force the establishing of default values

Overflow input buffers

Data and computation

Force the data structure to store too few or too many values

Force computation results to be too large or too small

File system interface

Fill the file system to its capacity

Damage the media

Software interfaces

Cause all error handling code to execute

Cause all exceptions to fire

Vijayaraghavan s eCommerce Taxonomy

Beizer's, Kaner's, and Whittaker's taxonomies catalog defects that can occur in any system. Binder's focuses on common defects in object-oriented systems. Giri Vijayaraghavan has chosen a much narrower focus—the eCommerce shopping cart. Using this familiar metaphor, an eCommerce Web site keeps track of the state of a user while shopping. Vijayaraghavan has investigated the many ways shopping carts can fail. He writes, "We developed the list of shopping cart failures to study the use of the outline as a test idea generator." This is one of the prime uses of any defect taxonomy. His taxonomy lists over sixty high-level defect categories, some of which are listed here:

  • Performance
  • Reliability
  • Software upgrades
  • User interface usability
  • Maintainability
  • Conformance
  • Stability
  • Operability
  • Fault tolerance
  • Accuracy
  • Internationalization
  • Recoverability
  • Capacity planning
  • Third-party software failure
  • Memory leaks
  • Browser problems
  • System security
  • Client privacy

After generating the list he concludes, "We think the list is a sufficiently broad and well-researched collection that it can be used as a starting point for testing other applications." His assertion is certainly correct.

A Final Observation

Note that each of these taxonomies is a list of possible defects without any guidance regarding the probability that these will occur in your systems and without any suggestion of the loss your organization would incur if these defects did occur. Taxonomies are useful starting points for our testing but they are certainly not a complete answer to the question of where to start testing.


Your Taxonomy

Now that we have examined a number of different defect taxonomies, the question arises—which is the correct one for you? The taxonomy that is most useful is your taxonomy, the one you create from your experience within your organization. Often the place to start is with an existing taxonomy. Then modify it to more accurately reflect your particular situation in terms of defects, their frequency of occurrence, and the loss you would incur if these defects were not detected and repaired.

  Key Point

The taxonomy that is most useful is your taxonomy, the one you create.

Just as in other disciplines like biology, psychology, and medicine, there is no one, single, right way to categorize, there is no one right software defect taxonomy. Categories may be fuzzy and overlap. Defects may not correspond to just one category. Our list may not be complete, correct, or consistent. That matters very little. What matters is that we are collecting, analyzing, and categorizing our past experience and feeding it forward to improve our ability to detect defects. Taxonomies are merely models and, as George Box, the famous statistician, reminds us, "All models are wrong; some models are useful."

To create your own taxonomy, first start with a list of key concepts. Don't worry if your list becomes long. That may be just fine. Make sure the items in your taxonomy are short, descriptive phrases. Keep your users (that's you and other testers in your organization) in mind. Use terms that are common for them. Later, look for natural hierarchical relationships between items in the taxonomy. Combine these into a major category with subcategories underneath. Try not to duplicate or overlap categories and subcategories. Continue to add new categories as they are discovered. Revise the categories and subcategories when new items don't seem to fit well. Share your taxonomy with others and solicit their feedback. You are on your way to a taxonomy that will contribute to your testing success.


Summary

  • Taxonomies help you:

    • Guide your testing by generating ideas for test case design
    • Audit your test plans to determine the coverage your test cases are providing
    • Understand your defects, their types and severities
    • Understand the process you currently use to produce those defects (Always remember, your current process is finely tuned to create the defects you're creating)
    • Improve your development process
    • Improve your testing process
    • Train new testers regarding important areas that deserve testing
    • Explain to management the complexities of software testing
  • Testing can be done without the use of taxonomies (nonspecific fault model) or with a taxonomy (specific fault model) to guide the design of test cases.
  • Taxonomies can be created at a number of levels: generic software system, development paradigm, type of application, and user interface metaphor.


References

Beizer, Boris (1990). Software Testing Techniques (Second Edition). Van Nostrand Reinhold.

Binder, Robert V. (2000). Testing Object-Oriented Systems: Models, Patterns, and Tools. Addison-Wesley.

Carr, Marvin J., et al. (1993) "Taxonomy-Based Risk Identification." Technical Report CMU/SEI-93-TR-6, ESC-TR-93-183, June 1993. http://www.sei.cmu.edu/pub/documents/93.reports/pdf/tr06.93.pdf

ISO (1991). ISO/IEC Standard 9126-1. Software Engineering - Product Quality - Part 1: Quality Model, ISO Copyright Office, Geneva, June 2001.

Kaner, Cem,Jack Falk and Hung Quoc Nguyen (1999). Testing Computer Software (Second Edition). John Wiley & Sons.

Whittaker, James A. (2003). How to Break Software: A Practical Guide to Testing. Addison Wesley.

Vijayaraghavan, Giri and Cem Kaner. "Bugs in your shopping cart: A Taxonomy." http://www.testingeducation.org/articles/BISC_Final.pdf




A Practitioner's Guide to Software Test Design
A Practitioners Guide to Software Test Design
ISBN: 158053791X
EAN: 2147483647
Year: 2003
Pages: 161
Authors: Lee Copeland

Flylib.com © 2008-2020.
If you may any questions please contact us: flylib@qtcs.net