The Evolving Test Inventory

The inventory precedes the test plan, or, in my case, the contract to test. When the inventory is fully fleshed and agreed upon, I can add the other items that the test plan needs in order to be acceptable. In the case of the heavyweight real-world example that we discuss throughout the book, the test plan was actually a master test plan that enumerated the test plans of the projects that it comprised and described how those projects would become integrated, verified, and validated. Its inventory contained all the functions and features of the projects it contained, appropriately rolled up to the high level. It was over 500 pages and contained references to over 1,000 related project documents. The project documentation for all the related projects filled an entire bookcase.

In a Web-based or other lightweight project, there is probably only one project. Its functions and features are enumerated in the inventory, as we discussed in Chapters 6 and 7. The inventory is the bulk of the test plan. It might be 10 pages long. I recently completed a project that used a PowerPoint slide deck and an Excel spreadsheet as the test plan; it was primarily made up of the test inventory. Most of the plan was built using FrontPage to create the links and deploy the slides. Most of the composition and information gathering was done online using our SharePoint Team Services Web site and its dynamic lists. The project was integrated, tested, and deployed with no major problems, on time and within the budget. It had project team members in four countries and three time zones, speaking three different languages. One of the biggest savings in the project was because we didn't even run up a long-distance phone bill.

Note 

The inventory document is a compilation that I will add to my test plan, or put another way, the inventory is the foundation of my test plan.

The Process in Brief

I like to start my test inventories in PowerPoint. PowerPoint is a good tool to help sketch the project quickly in an outline. This tool makes it easy to change the order and importance of any item or slide simply by dragging it wherever it should go and dropping it there. Even if there is extensive preexisting documentation, PowerPoint makes it easy to account for all the major pieces and produce an outline quickly.

In some projects, I am handed a stack of documents when I walk in the door; in others, I have to interview the developers, sales staff, and customer service to find out what the project entails. In either of these situations, I will probably construct the preliminary outline in PowerPoint and then export it to Word for embellishment and details. The heading levels that I apply to outline elements in PowerPoint are carried across to Word, and they allow me to import my rolled-up inventory items with no extra work. I use the outline view in both PowerPoint and Word to organize inventory items and to perform the rollup. And I can generate the short list that I will use in the columns of the inventory that I put into table form either in Word or Excel.

I can use Word's internal RTF fields to link project documents into my new inventory. But for the most part, I use hyperlinks for this purpose today.

Once the interviews are complete and I have my prioritization underway, I usually move the larger projects to a spreadsheet. You can simply copy and paste the columns from the Word table to the spreadsheet. For the expanded inventories that include cross-references to environments and functions, a spreadsheet is the tool of choice because it is so convenient to keep the various tables on individual pages in the spreadsheet workbook.

With the interactive data-sharing technology available today in the PC environment, it does not really matter which of these tools you choose. They each offer certain advantages, and you can pass your work back and forth between them with great facility. Don't be afraid of learning more about a Microsoft Office tool.

Finally, and most progressive of all, I can import my spreadsheets directly into my SharePoint Team Services Project Web site and allow the entire team to participate in the project. In the next sections, I will show you what the process looks like in each of these tools.

Inventory in PowerPoint

The preliminary test inventory used in the test estimation process normally starts with items from the project documentation-the requirement, function, and design specifications. I begin a list that includes all of these items that I know about, and I add new ones as they appear. Often, the inventory is built during design and review meetings. In any event, the process needs a starting place. I use PowerPoint more and more frequently as the tool to document this early "coming together" phase of the test inventory.

If the project documentation is not suitable, then a list of the primary functions and a list of all the menu options in the software is a good starting point. These functions can be taken from the user interface or source code listings, if necessary. Finally, the environment requirements need to be established and documented as discussed in Chapter 7, "How to Build a Test Inventory." In this type of effort, I would normally use PowerPoint to capture the initial test inventory and then import it into a Word document.

Figure 8.1 shows an example of this process. It features the sample inventory from the book. The outline view of PowerPoint is a good recording tool. I use it extensively in meetings to keep my notes.

Figure 8.1: Inventory in PowerPoint Outline view.

PowerPoint is a powerful composition tool that is nimble enough to allow the user to keep up with a spirited design meeting. You can make structural changes interactively and then share your resulting outline with others. The outline is a properly formatted set of document headings, and you can easily move it to other document formats like RTF, Word's DOC, and HTML.

You can export the PowerPoint outline directly to Word by selecting File, Send To, menu option, and then selecting Microsoft Word. You then have several options that let you control the format of the exported information in your Word document.

PowerPoint's terse bullet-oriented makeup cannot carry the detail layers forward. This may not be perceived as a problem in a RAD/Agile shop, where there are no detail layers, but it can be crippling in a plan-driven environment. When you actually have data under these topics that you have to keep track of, you have outgrown PowerPoint, and it's time to move your outline to Word. Which brings me to my next topic: keeping my inventory in a document.

Inventory in a Document

For my verbose inventory document, I want to collect or link all the project information that I can into a single document, and I need the ability to get whatever view of that information that I want. After all, I am not creating the inventory because someone told me to. I am creating the inventory because it is my most powerful tool. As such, I need to be able to find anything quickly, and I need to view the inventory in as much or as little detail as necessary at that moment. For example:

• Initially, I need the high-level view; I don't want to get bogged down in details. But when I am trying to construct scenarios, I need to be able to find out what's under the hood.

• I need to move huge topics around without losing bits or leaving pieces behind. I want a single source for my high-level views that will let me dig down through all the information that I have if necessary.

• I need to include all types of documents, pictures, equations, tables, and even spreadsheets.

• For communications with others, such as meetings, I need to produce a short high-level view, and for my test effort, I need a view that brings selected topics to the foreground without losing all the other topics.

The Hood and What's Under It

As I mentioned, my management likes their information in short, succinct bursts. The man who was my vice president throughout all my years at Prodigy maintained that if you couldn't fit it into three to five bullets on a page, then you didn't understand it well enough to explain it to him. In all my years, I have kept this as a guiding principle. It has served me well.

Consequently, when I prepare for a report or status meeting, I want the shortest list possible-without losing the feeling that there is depth behind it. So I evolved a system to use the Outline view and the table of contents field in my verbose inventory document to do this.

Inventory in Outline View

I can show only the highest levels or I can show all the main topics and their subtopics just by selecting the heading levels that I want to display. A heading level 1 is the highest, usually the title of the chapter, and I typically don't go below 5.

The Office Tools: Collaboration and High-Function Web Sites

I use Microsoft Word to create and maintain my verbose inventory document. Word's superior table manipulation features, its ability to incorporate graphics and link multiple documents, and its superior built-in graphing tools make it ideal for these tasks. one other feature that has become very important is that these documents can be saved as Hypertext Markup Language (HTML) and put online in an internal private Web site, or Intranet, to create a single-source test repository that is usable by people who don't have Microsoft Office.

Any of the major Windows-based word processors will probably perform most of the tasks described here. With Word, you can also send documents and receive them from the other Office tools like PowerPoint, and you can copy and paste tables to and from Excel. There is little or no conversion required on your part.

Today, I use Office 2002 because of its rich support for Web content and for the collaboration features it offers. At the time of this writing, most people are using Office 2000 and Microsoft Word for Windows version 6.0.

I use Microsoft SharePoint Team Services to power my test Web sites. The product is an add-on to the Internet Information Server that works in conjunction with the FrontPage server extensions to provide a host of Web components that interact with Microsoft SQL Server to create an automated interactive Web site. The SharePoint Team Services site also supports direct editing of office documents by the Office 2002 applications. The site offers a rich set of project management tools, as well as document repositories. Figure 8.2 shows the home page of testersparadise.com, the Web site that supports this book. All the content on this page is generated dynamically. You add new announcements and events by simply filling out a form and submitting it. Team members who have contributor status can participate in all the interactive activities on the site.

Figure 8.2: TestersParadise.com Home Page, an example of a high-function test Web site.

Using Office 2002 and its support for SharePoint Team Services, I have been able to move my inventories to high-function SharePoint Web sites and Include even more powerful searching capabilities across entire libraries of documents. The high-function Web sites also make It possible to Include databases In my inventory, complete with stored queries and the ability to create ad hoc queries on the fly. I am currently using Microsoft SharePoint Server and Microsoft SharePoint Team Services Web sites to accomplish this. See http://www.testersparadise.com for examples.

To start the test inventory document, I can either import my PowerPoint outline as I just described or I can create my outline directly in Word. Figure 8.3 shows the inventory document in Outline View mode. This is the same inventory that we just saw in PowerPoint in Figure 8.1, after it has been imported into Word. If you click on any of the + signs, the text below the heading is revealed. Outline view in Word acts a lot like the rollup view in most project management tools. (If you are wondering why I don't just use a project management application for this work, see the sidebar, Testing Is Not a Project-Or Is It?) So, it is popular in status meetings. The problem is that you can't print it, which is probably why most people use PowerPoint to display this outline. However, if I need to print the rolled-up view (outline) of my inventory document, I use a table of contents. An example of this is coming up next.

Figure 8.3: Inventory from a Web project as an outline in Microsoft Word.

Inventory in a Table of Contents

When you assign heading levels to your inventory items, it is possible to get a succinct listing of them in a table of contents, or TOC. To generate a TOC, put your cursor where you want the table of contents to appear in the document. In Microsoft, click on the Insert menu, choose Field, then scroll down to TOC and select it. Choose any formatting options you want, and click OK.

Figure 8.4 shows a table of contents complete with page numbers. I often take paper copies of the inventory in this form to status meetings. If there are issues that we need to discuss, I can add them as headings under the appropriate topic and print the table of contents with these items showing. My management has always enjoyed the brevity of this kind of presentation in meetings.

Figure 8.4: The table of contents showing the inventory.

Testing Is Not a Project-Or Is It?

We should discuss project management tools for a moment so that you might benefit from my experiences trying to use them in software test efforts, and also so you will understand why I am not discussing any here.

Given that I am describing "rolling up" projects into a set of high-level items of work, many folks ask, why not just use project management skills and tools to "manage" the test effort? The short answer is that testing is not a project; it is a process. When a project is done, you check it off and go on. You almost never look back, except to celebrate. Budget has nice start and end dates to work with, and you can claim that you are finished spending money on that particular item. Testing, on the other hand, is never "done." The test may pass this time and fail next time. Consequently, we are never "done" testing; we simply stop at some point and ship the product. Budget is generally not satisfied because of this and several other factors discussed throughout this book. One reason why project managers don't want to touch a test effort is that it's too easy to end up looking bad. Ask any project manager if you don't believe me.

Consequently, a pure application of project management methods doesn't work any better than traditional software quality assurance methods. (Project management is a discipline that grew out of civil engineering.) This book is full of the project management methods that I have found workable in a test effort; however, the software tools are not so flexible. I spent two years trying to use two of the top project management software applications in my test efforts because they were required. Using them added a huge amount of documentation overhead to the test efforts, but worse than that, we couldn't share the information from the management tool with the documentation. So I had my outline (tasks) in one place and my detail layers in other places. It took a full-time administrator just to make sure that tasks were linked to their supporting documentation or point of origin-and, most especially, to somebody's budget.

I'm not saying it can't be done. I'm just saying that traditional project management tools have not been "sharp" tools when applied to test efforts. On a brighter note, the SharePoint Team Services Web sites offer strong support for project tools, like task lists. They also are good at sharing data, so you can import your task list from an Excel spreadsheet, and you can export it back out again so it can be used elsewhere. One final note, a product to watch for possible uses in a test effort is Microsoft Project, which is now Web-based. Who knows, it may end up bringing project management techniques to the test effort.

For the next step, creating a table to contain the inventory, it would be nice to just copy the items from the TOC, but unfortunately, in a Word document, the table of contents cannot simply be copied to another document and remain static during that process. It cannot because a TOC is a dynamically generated field where the headings in a document are collected.

One way to produce a permanent list from the table of contents field is to copy the table of contents and paste it into a new document. Use the Save As menu option to select the document type: Text with Line Breaks. This will remove the special field coding, leaving only the plain text of the categories. Close the text file. Repeat this procedure for each document that contains inventory items you want to gather. This process allows you to extract the inventory items so that you can do other things with them, like put them into a table or a spreadsheet. I will discuss building a table with them next.

Inventory in a Table

When you put your test inventory into a table, you can sort it by any column and perform calculations, like adding up the number of tests or the number of hours required to run a group of tests.

When you want to create the test inventory in a table, use the Insert File option to insert the plain text tables of contents into the test inventory document. Organize the categories in any way that makes sense, but keep each test item or category heading on its own line.

When you have assembled a list, select all of the inventory items and use the Convert Text to Table option under the Table menu to turn the entire listing into a one-column table. Add columns to this table as you need them. These columns will hold the individual counts for each row, or test, in the inventory.

Tip 

If these topics are new to you, read the sections on document headings and automatic tables in your word processor user guide or online help for specific instructions.

The New Function item in Table 8.1 is an example of functions taken from design documents. View Mini Clip is a menu option from the user interface. The two-credit-card limitation might have come from the interface or from some design or requirements document.

SAMPLE APPLICATION (RELEASE 2.0)

Bug Fix Information

Fix for Error #123 (see req. B477)

Fix for Error #124 (see req. B501)

New Function (see req. D071 & D072)

New menu option #3: View Mini Clip

Purchase option: Not available in some states

Minimum order must be $30.00

Method of payment limited to 2 credit cards

Structural / Environment Information

Enhancement-automatic detection for 50 modems. (Rel. 1 had auto-detect for 3 classes only)

Software installation is automatic at logon

Existing Application Base Function

Standard base function tests still apply: All test suites for Version 1.0 will be run.

Our Best System Simulator

(automated suite BSIM01 67% coverage of Release 1 Test Inventory for the Best Simulator functionality)

Message Data Flow Checker

(automated suite DFCHECK 47% coverage of Release 1 Test Inventory for the Data Flow Checker functionality)

Screen Comparison-Pixel Viewer

(automated suite PIXVIEW 77% coverage of Release 1 Test Inventory for the Pixel Viewer functionality)

Environment Catalog

Operating Systems:

Client: Microsoft Windows 3.1 and higher, Win 95, Win 97, NT 3.51 with patches from #4 pack applied.

Host: To be determined.

Network: Under investigation by Net, Engineering plan due?

Hardware:

Computers: All machines on the operating system compatibility list.

Modems: All machines on the operating system compatibility list.

Notice the hyperlinks (underlined topics) in the functions that link to the requirement. The tests identified from these documents are usually high-level and general in nature, so they correspond to a category or type of test. Subsequent analysis will add tests to each of these categories. Often the categories evolve in the test suites. These categories usually appear as a heading level one, or the highest-level heading in the test inventory. Bug Fix Information in Table 8.1 is another example of using existing documents or references to existing documentation, such as the bug reports to define the testable item.

If you plan to construct your inventory using existing documentation, get electronic copies of the relevant specifications and requirements documents from your project. If electronic copies of the document are not available, consider faxing or scanning the documents and running character recognition software to produce an electronic copy. You have several choices on how to include your documents. You can use the include files method to link individual files into your document, or you can use hyperlinks within your document. Either way, if linked documents are updated, your document automatically reflects the changes. However, the Word Outline view and table of contents may not be useful to you, since Word does not follow hyperlinks when building these views. Therefore, your outline won't be easy to update automatically.

One of the major obstacles in document automation is the lack of a common standard for documentation. The developers and testers may use text files or a PC-based word processor, while the documentation group may use a publisher's tool such as Quark or an Adobe pdf file. Make sure that all the tools can produce a common high-level file format that supports the inclusion of graphics and tables. Acceptable examples in Microsoft Word document format are Rich Text Format (RTF), Standard General Markup Language (SGML), and HyperText Markup Language (HTML). I prefer HTML because it is easier to automate the creation and distribution of HTML files than it is with any of the other formats. HTML documents are small and can be easily distributed across networks and viewed on any platform that has an HTML browser.

Typically, project documents come in any number of formats. Start by converting any documents that are not in the correct format. Next, make sure that each document has a table of contents. Generate one if necessary. If the document was prepared in a text format or without heading level definitions, it will be necessary to apply the correct heading level style to the document's headings before a table of contents can be generated automatically. This operation is worthwhile because these headings will be used throughout the test effort.

Inventory in a Spreadsheet

When you begin to add environments and functions columns to your table, it can quickly grow beyond Word's capabilities. When this happens, it is time to copy your table to a spreadsheet. One of the great things about the spreadsheet is that you can add new pages (called sheets in Excel) as you need them. Figure 8.5 shows the inventory after it was copied into a new workbook. The inventory spreadsheet has sheets for the environmental catalog, functions, and test scenarios. All of these sheets are linked. The spreadsheet makes it possible to sort and filter the data. These features allow you to answer questions like, "What are the most important tests I can run to test EDI?" in seconds.

Figure 8.5: The environment catalog sheet from the inventory spreadsheet.

I always reserve budget in my projects for a spreadsheet specialist, a mechanic. By setting up the automatic calculations and shared data values, a good mechanic can create a spreadsheet workbook that instantly updates data on related sheets whenever you change anything in the spreadsheet. The workbook can also present the latest results in pivot charts, graphs, and whatever visualization is most effective for you at the moment.

When it comes time to test, I sort the test scenarios and filter the sheet to produce a list of the tests that have to be run to test a specific PDR, feature, or environmental aspect, such as a particular database. I can copy the resulting worksheet, print it, or publish it on the Web. My testers have a succinct list of tests that need to be performed.

Fundamental Principle: Never Rekey Data

The quality assurance process in software development is already in a position where it cannot keep up with development technologies.

Any process that requires the same data to be manually rekeyed into the system is badly flawed. Every time the same data is manually rekeyed, money is wasted. Every data entry point is a source of errors being introduced.

The cost of tracking and integrating review commentary increases with the number of reviewers.

Every time paper documentation is circulated, the potential for version control problems is expanded. The probability of outdated paper-borne information causing failure rises, and the cost of ensuring that everyone is on the same version increases.

Whatever tools and processes are selected, make sure that the documents are sharable and come from a single source.

Inventory in a High-Function Web Site

As I write this, I have to start by saying that putting your test collateral on a Web site is not always efficient. If the culture in your company does not embrace this technology yet, or if the control of Web resources is controlled by feudal Web lords, then I recommend you tread lightly on this topic. Rest assured; it won't be this way for long.

I tried to use traditional HTML document-based Web sites to aid my test projects for most of the 1990s. They were only "one-way" tools at best. At worst, they took resources away from the actual testing and gave the vultures something else to criticize. With few exceptions, a straight Web site isn't good for much more than publishing bug stats and test documents.

A high-function Web site is a different animal, however. The SharePoint Team Services Web site is a collaborative and interactive place. First of all, the administrative overhead is almost nothing once the administrator learns the product. It takes about five minutes to generate a fully functioning interactive Web site with five built-in levels of group security: browse, contribute, author, advanced author, and administrator. In addition, there is a full complement of pages containing lists (see Figure 8.6), including a document folder, announcements, events with a calendar, and a link list-all ready for your team to use. And, of course, you can create more if you need them.

Figure 8.6: The SharePoint Team Services Lists page.

The Team Site is powered by SQL Server and several types of canned functionality with which you never need to worry. You customize the way your features work and administer your site using online forms. You don't need to know HTML, although it doesn't hurt, and you don't need to use a Web page creation tool like FrontPage, but that doesn't hurt either-especially if you want to customize your pages as I have for testersparadise.com. The List Builder tool is happy to absorb spreadsheets and allow you to create custom editable views of them.

Other features included out of the box are any number of forums; announcements; events, complete with calendar; and several built-in views, such as Today's Events, My Events, and All Events. The list enumeration page is shown in Figure 8.6. There is a contacts list that team members can add to, a link list where team members can add hyperlinks with explanative text. The system provides several other canned lists, and you can create any number of custom lists.

List data is stored in an SQL Server database, so when someone edits the data on one of the views, it is updated in the database and the new value is automatically sent the next time anyone requests a list that contains that data. So there is no worry about propagating stale data from multiple data sources. SharePoint Team Services takes care of all the data updates and concurrency issues.

There is also a survey tool that allows you to create surveys on the fly. Surveys are a good consensus-building tool. For example, I like to use them when I establish priorities. (An example of this type of survey is in Chapter 9, "Risk Analysis.") All of these features can be created and modified by whatever groups the administrator chooses.

Figure 8.7 shows the PDR view of the inventory after the data was imported from the Excel spreadsheet. Notice the menu on the left side of the page; it shows three views available. These hyperlinks to other views are very much like the tabs on the bottom of the sheets in the spreadsheet. These views are easy to create, and you can create as many of them as you need from the data on the spreadsheet. I am often asked to create special views for different groups, like management.

Figure 8.7: PDR view of the inventory in a Team Services list.

Figure 8.8 shows the Environment view of the inventory list. The user can sort any list view by any column simply by clicking on the column heading. The user can also filter the views so that only items meeting the filter criteria are displayed. For example, you can filter all PDRs by their group, display only the PDRs that use a particular environment, and so on. Users can also subscribe to a list and receive automatic notification via email when something changes on the list.

Figure 8.8: Environment Catalog view of the inventory in a Team Services list.

As nice as the lists are, it is much more difficult to keep running totals and perform other calculations in this canned environment; so the spreadsheet still offers some advantages. The good news is, permissioned users can export a list back to Excel 2002 and work on the data there.