Chapter 7: How to Build a Test Inventory | Software Testing Fundamentals: Methods and Metrics

A well-crafted test inventory, like a grocery list, is as succinct as possible. Ideally, each item should fit on one line. All the description and detailed specifics of the test exist but are stored elsewhere, usually later in the test agreement or test plan. Each main topic or test category will have more detailed layers of tests beneath it.

My management always wanted all the information that they needed, and only the information they needed, on as few sheets of paper as possible. The first test inventory I ever submitted was a photocopy of the test description section of the table of contents of my test plan. It was about five pages long. Management liked it so well that I have never changed the practice. In project management this is called a rollup.

Starting with the Requirements

When I start an inventory, I begin with the development project requirement documents. These projects may be called by any number of names; the point is, I am looking for the level of project requirement that has a budget attached to it. I try very hard to make sure all budget items are accounted for on the inventory. I use these requirements to build a preliminary inventory. Next, I catalog all of the related development projects that I can identify under these requirements. Then I enumerate all the major functions and cross-reference them to their budgeted projects. It looks like this:

Requirement (with budget)

Project under this requirement
- Function under this project
  - Testable items under this function

There can be many testable items under a function, many functions under a project, and many projects under a requirement. When it is all rolled up, it just looks like a requirement. It's a lot like nested file folders on your hard drive. In fact, I have met many testers who store test cases in this type of structure.

Inventories from different projects can look very different. Following are two samples from the real-world examples discussed earlier. The first is a preliminary test inventory from the structured RAD example in Chapter 4, "The Most Important Tests (MITs) Method." At this point, it is only a list of the requirements; there is nothing to "roll down" yet. This is possible if it's a plan-driven project with formal budgeting procedures. If it is a RAD/Agile effort, you may need to be more creative, as we will see in the second example: a preliminary test inventory from a Web development effort also described in Chapter 4.

Sample Inventory from the Structured RAD Project

In this example, I describe a set of formal and politically correct procedures for building your test inventory and discovering the scope of your test effort. This method works well in large organizations where political and budget boundaries may be sensitive issues. It can scale from small to huge, and it is suitable for heavyweight as well as middleweight projects. This example is from a full system integration test effort in a large system, so the scope is extensive, as you will see in the coming chapters.

This system test effort differs from individual integration test and IT-conducted system tests in that it seeks to verify that critical business functions are operating properly across the entire system. The system test effort includes function, performance, and load testing. Rather than focusing on the new data flows through the system, it will focus on day-to-day business functions both before and after the system is subjected to the new data flows. In theory, very little function will change, but loads on various systems will be increased and new processes will be inserted in the system to handle the new data formats. As such, the final inventory will include extensive dependency and cross-reference information.

The test inventory is the tool used in this system test effort to identify the scope of the test effort and prioritize it based on each inventory item's risk potential. The inventory is intended to be an enumeration of the software system testable items that have been identified in the entire project. The initial test inventory was prepared from the available project documentation and is included in the master test plan. Initial priority ratings were applied to each item in the inventory based on the available project documentation. The inventory also contains the reference to the systems touched by a given item. This initial test inventory serves as a starting place for the Subject Matter Expert (SME) interview process.

The Billing System Preliminary Inventory

Table 7.1 is an excerpt from the preliminary test inventory prepared for the billing system real-world example discussed in Chapter 4. This excerpt included only the Project Development Requirement documents (PDRs) applying to one area of the billing system. It was prepared and sorted by the PDR. This example already includes a preliminary prioritization assigned by the test team based on the PDR documentation and budget.

Table 7.1: Preliminary Test Inventory from the Billing System Example, Sorted by PDR
PDR	PROJECT DESCRIPTION	P
MGT0026	AcqrdCo to ParentCo Property Management	1^[a]
MGT0027	Treasury System	1
MGT0030	Convert AcqrdCo to ParentCo Expenditure Billing Sys	1
MGT0033	Fixed Assets & Project Accounting	2
MGT0034	Interface AcqrdCo to ParentCo Oracle General Ledger	4
MGT0098	TAX	1.5
MGT0145	Budgets-AcqrdCo Acquisition	1
MGT0150	Convert AcqrdCo to ParentCo Risk Management	2
MGT0201	Convert AcqrdCo to ParentCo Oper. Support/Environ	2
MGT0202	Taxi and T&E Lodging	5
MGT0203	AcqrdCo to ParentCo Car Repair Billing	3
MGT0218	Convert AcqrdCo to ParentCo ORACLE Purch and Mat	2
MGT0219	Convert AcqrdCo to ParentCo Accounts Payable Sys	1
^[a]P = Priority (1 = highest, 5 = lowest)

This inventory was created in a spreadsheet using Microsoft Excel. One of the advantages of this representation is that the entire inventory can be sorted by any column. This sample is sorted by PDR, but it can just as easily be sorted by contact, size, or priority. Another advantage is that the spreadsheet makes it easy to apply formulas and keep track of totals.

The description and dependency sections need to be kept brief, less than 255 characters. The main reason is that you can't copy and paste an entire sheet that contains oversized fields. When you copy an entire sheet, all fields are truncated to 255 characters max; you have to copy larger fields individually, which is problematic and slow. Many more columns will be added to this inventory as the enumeration process continues, and too much text in the terse description becomes a formatting problem and a distraction. Use hyperlinks to related documents to carry details forward.

On the Project Web Site

When I create one of these inventories today, I like to do it in a collaborative Web site. Each PDR can then be a hyperlink to the actual documentation in a secure, single-source environment. Using the technology in dynamically generated Web pages allows me to create a completely traceable inventory that is linked to all its parent and child documents.

I use the Microsoft SharePoint Team Services Web site to support team collaboration and documentation for my projects. It is included free with Microsoft Office 2002; so many people already have it. It supports collaboration features on both PC and Mac systems and is discussed in detail in Chapter 8, "Tools to Automate the Test Inventory."

Table 7.2 shows the preliminary inventory from Table 7.1 after the addition of the notes from the interviews. This inventory is sorted first by contact and then by priority.

Table 7.2: Preliminary Billing Inventory with the Interview Input, Dependencies, Environmental Description, and Test Requirements
PDR	Project Description	Test Area	Priority	Test Order	Test Number	IT Contacts	Dependencies and Notes	EDI	HRIS	IIDS	Mercury	TMC	TWSNet	TYMS
MGT0030	Convert AcqrdCo to ParentCo Expenditure Billing Sys	G&A	2	1		SME1, Dev1, BIZPart1	Test together 0033 + 0030 + 0218. Dependent on MGT0026 shared file: Rents Receivable file with Payroll (MGT0207) PeopleSoft + GL MGT0034		x
MGT0026	AcqrdCo to ParentCo Property Management	G&A	3	2		SME1, Dev1, BIZPart1	Works with MGT0030 shared file: Rents Receivable file with Payroll (MGT0207) PeopleSoft + General Ledger MGT0034		x
MGT0203	AcqrdCo to ParentCo Car Repair Billing	G&A	3	2		SME1, Dev1, BIZPart1	Needs MGT - Car Maint. (the mechanical part) owner SME2 w/Payroll (MGT0207) PeopleSoft + GL MGT0034		x		x
MGT0098	TAX	G&A	2	1		SME1, Dev1, BIZPart2	With Payroll (MGT0207) PeopleSoft + GL MGT0034	x	x
MGT0218	Convert AcqrdCo to ParentCo ORACLE Purch and Mat	G&A	2	1		SME1, Dev1, BIZPart2	together 0033 + 0030 + 0218 w/Payroll (MGT0207) Peoplesoft + GL MGT0034	x
MGT0027	Treasury System	G&A	4	2		SME1, Dev1, BIZPart2	With Payroll (MGT0207) PeopleSoft + GL MGT0034	x	x
MGT0034	Interface AcqrdCo to ParentCo Oracle General Ledger	G&A	4	2		SME1, Dev1, BIZPart2	With Payroll (MGT0207) PeopleSoft + GL MGT0034	x	x
MGT0033	Fixed Assets & Project Accounting	G&A	5	2		SME1, Dev1, BIZPart2	Test together 0033 + 0030 + 0218 w/Payroll (MGT0207) PeopleSoft + GL MGT0034	x	x
MGT0145	Budgets - Conrail Acquisition	G&A	5	2		SME1, Dev1, BIZPart2	With Payroll (MGT0207) PeopleSoft + GL MGT0034	x	x				x
MGT0219	Convert AcqrdCo to ParentCo Accounts Payable Sys	G&A	5	99	X	SME1, Dev1, BIZPart2	Effectively Closed
MGT0150	Convert AcqrdCo to ParentCo Risk Management	G&A	2	1		SME-Dev2, Dev3	Data conversion	x
MGT0201	Convert AcqrdCo to ParentCo Oper. Support/Environ	G&A	2	1		SME-Dev2, Dev3	Data conversion	x		x		x	x	x
MGT0202	Taxi and T&E Lodging	G&A	3	3		SME-Dev2, Dev3	Manual work around available for split day.						x

The List Builder function allows me to import a spreadsheet with one sheet in it. It automatically converts the range of fields I select into a dynamic list in my project database. Figure 7.1 shows the preliminary inventory with interview notes in list form. This one was imported from the spreadsheet shown in Table 7.2. I can then add fields, create custom views, and let the whole team update the list online by clicking on the Edit button, and I can export the entire list back out to a spreadsheet any time I want. Again, the field size cannot exceed 255 characters or else the import fails.

click to expand
Figure 7.1: The inventory converted into a dynamic list on the project's Web site. (Powered by Microsoft SharePoint Team Services.)

As indicated earlier, I have been experimenting with high-function Web sites as document automation tools for some time-with mixed results. Like any other form of automation, they require a change in the way people do their work, and that initiative requires incentive. Recently, I have proved enough savings from the use of a team site to make a very compelling case for management to demand these changes. See Chapter 8, "Tools to Automate the Test Inventory," on using a Share-Point Team Services Web Site in your test effort for more information on this emerging way of doing business.

Preliminary Inventory from a Web Project

The test inventory shown in Table 7.3 is an example of a preliminary or first-draft inventory from an early e-commerce application. This is how the inventory would look before any test analysis or design sessions. It is the second release of the product and has some categories that a new first-release product would not have, such as bug fixes.

Table 7.3: Preliminary Inventory from a Web Project
Sample Application (Release 2.0)
BUG FIX INFORMATION
Fix for error #123 (see req. B477).
Fix for error #124 (see req. B501).
NEW FUNCTION (SEE REQ. D071 AND D072)
New menu option #3: View mini clip.
Purchase option: Not available in some states.
Minimum order must be $30.00.
Method of payment limited to 2 credit cards.
STRUCTURAL/ENVIRONMENT INFORMATION
Enhancement-automatic detection for 50 modems. (Rel. 1 had auto-detect for 3 classes only).
Software installation is automatic at logon.
EXISTING APPLICATION BASE FUNCTION
Standard base function tests still apply: All test suites for Version 1.0 will be run.
Our best system simulator. (automated suite BSIM01 67% coverage of Release 1 Test Inventory for the Best Simulator functionality).
Message data flow checker (automated suite DFCHECK 47% coverage of Release 1 test inventory for the data flow checker functionality).
Screen comparison- Pixel viewer (automated suite PIXVIEW 77% coverage of Release 1 test inventory for the pixel viewer functionality).
ENVIRONMENT CATALOG
Operating Systems:
Client: Microsoft Windows 3.1 and higher, Win 95, Win 97, NT 3.51 with patches from #4 pack applied.
Host: To be determined.
Network: Under investigation by Net. Engineering plan due (?)
Hardware:
Computers: All machines on the operating system compatibility list.
Modems: All machines on the operating system compatibility list.

Each of the main categories in the inventory was taken from a different project document or captured in a design meeting. If there is no suitable documentation, create a table containing every major feature and known or suspected facts about the software. These features may come from the marketing and design descriptions of the product, the user interface menus, or from the definitions of program procedures, functions, or object classes. As in the previous example, this preliminary inventory serves as the starting place for the SME interviews. The following paragraphs describe the main categories in the Web project inventory.

Bug Fix Information

This information would probably not exist in the first release of a product. In a new release of an existing product, there are almost always some bug fixes included in the release. The source documents here are the bug logs, the bug fix reports, and the product release notes.

New Function (See Req. D071 and D072)

This category comes from the design documents. The notes "See Req. D071 and D072" refer to specific requirements in the design document. If the project is a first-time implementation, all the functionality will probably be new. If it is a new release of an existing product, only a fraction of the functionality will be new.

Structural Information

These functions are closely related to the system rather than to the user interface. These are typically features that are not easily tested from the user interface and require separate attention. In most system test efforts, each of these categories has its own separate test inventory.

Application Base Function

This category is a listing of any existing test suites that will be used to reverify the existing functions in the system.

Environment Catalog

An environment catalog or inventory that lists the required hardware and software environments serves as the basis for this part of the test inventory. Testers usually identify a subset of all supported environments from the catalog for the test effort.