Building Path Test Estimates Quickly
|
|
Let's take a look at what's behind door number 1 ...
The screen shown in Figure 12.6 is from the second release of the shopping application called Tester's Paradise. The Tester's Paradise application began as a pseudo-application to allow safe use of statistics gathered in real three-bubble applications. It was such a good idea that it has since taken on a life of its own. The most current release is running on the World Wide Web.
Figure 12.6: The main menu from the second release of the Tester's Paradise application.
If the user logs on to the network and navigates to Tester's Paradise Release 2, the menu shown in the figure is presented. It has five options; each one has a number of function paths behind it.
Application function paths are the logical paths through a function in an application or a system, as opposed to the paths through a block of source code, which is called a function. For example, following menu options through an application to perform some function, like "purchase a data flow checker," is exercising a function in the Tester's Paradise application. The actual number of paths through the source code of a program function is usually considerably larger than the number of system or program function paths. An application function path is most commonly accessed by the user via a menu option.
The fastest way to get a quick estimate of how many functional tests need to be run is to count the navigation options on the system menus. These options are effectively the doors in the system maze. In a Web application, these navigation options are the hyperlinks. If the tester can't have a map of all the paths in the system up front, he or she can at least have a count of how many doors there are to be opened.
Step 1: Building a Menu Map of the System
Many types of maps can be built to help a tester understand and keep track of the system he or she is testing. The first map built for a system is usually the logic flow map for the highest level of functionality. In most cases, this is the map of the menu system or user interface. All the other maps for the system are maps of the details behind the menu options on this first high-level map.
Step 2: Counting All the Menu Options in the System
Counting all the menu options is the fastest way to determine the gross number of independent paths through the system. This is a big job, but it's not anywhere near as big a job as drawing all the exceptions.
| Note | The number of independent paths from a menu is equal to the number of options on the menu. |
| Assumption # 1. | The user will never get stuck on a menu. |
Figure 12.7 shows how the menu options can be modeled as a logic flow diagram. A menu is a case statement. In order to leave a menu, the user must select something. This means that the number of decisions on any menu is equal to the number of options minus one (the one that is selected). This method of counting ignores the trivial case where the user never leaves the menu-never gets out of the maze. If that case is considered, then the number of decisions is increased by 1. Each menu option then becomes a decision, and the number of paths on a menu is equal to the number of options plus 1.
Figure 12.7: The Tester's Paradise main menu modeled as a logic flow diagram.
D = Number of options - 1
The IP = D + 1
-
= (Number of options -1) + 1 = Number of options
| Assumption #2. | The Cancel button is the same as the Close action from the window system menu. This means that it will not be counted as a separate option. It is assumed that the system menu will always work correctly. |
| Assumption #3. | The Esc key will always back the user out of the current operation, and it will always work reliably. |
 |
I have recently encountered 3 different applications with "no-exit" pages-apparently there is a flaw in one of the new jsp designer tools that leaves out the navigation button(s) required to proceed to the next step, so users get stranded on the page. They can use the browser back button to go back, but they can't proceed forward. I guess if you live long enough you will see all your assumptions fail at some point.
|
These assumptions may be incorrect, but they are the reason that the Esc option and the system menu are neglected in all these menus. If any of these assumptions is proven to be incorrect, the condition that was neglected because of the assumption will have to be included and the number of paths on each menu will increase by one option. This will increase the total independent paths (TIPs) for each menu by 1 as well.
The illustration shows the first set of application screens presented by the designers. The arrows trace the new application function, View Portable System Monitor, from the main menu. There are eight menu options on the View Portable System Monitor function path. Each menu option has logic paths associated with it. The lower-level menus for Our Best System Simulator, Message Data Flow Checker, and the Screen Comparison-Pixel Viewer are not shown in this illustration. The IP total of the menus under each of these main options is shown next to each option.
Given the simplifying assumptions, there are 23 distinct menu options being considered in the Tester's Paradise Portable System Monitor menus. Their distribution is shown in Figure 12.8.
Figure 12.8: The Tester's Paradise Portable System Monitor menu.
The menu option paths are not the only paths that must be tested, but menu options are important because they serve as key items in the test inventory. Other types of tests that are related to a menu option are typically grouped under the menu option title in the test inventory. Testing how many times the program is going to spin through a loop will be controlled by the data used in the function test. The test data is determined separately. The data tests for the entry fields on the form are discussed separately in Chapter 13, "Data Analysis Techniques."
Step 3: Counting the Exceptions for Each Menu Option
Each menu option has at least one exception condition associated with it. The most common case is the error window that opens if any menu selection is unavailable and when the menu option should not be available. One of the most frequently neglected test sets is the verification that menu options that should not be available during some function tests really are not available or are "grayed out."
One exception for each menu option is usually a reasonable estimate; however, this number should be tailored to suit the particular situation. Data entry fields that are required will have one or more error messages for each type of incorrect data response or data exception. Data-dependent error messages will be accounted for in the test inventory as part of the data analysis.
Calculate the IPs for the number of exceptions that will be tested. This number can be refined further if detailed analysis is conducted. This example will estimate based on one exception for each menu option.
Sample Test Inventory 2
The inventory created using the word processor is used as the base. New rows are added for the functions, and new columns are added for the path tests.
|
Fill any empty cells with "0" so that the Table Formula function can be used to automatically calculate the totals. The cells in the rightmost column, "TIPs," use the formula =SUM(LEFT). The totals in the bottom row, "Totals," use the formula =SUM(ABOVE).
For additional detailed information, select the Index function in Word's online Help (F1) and search for "formula." There is a topic that explains in detail how to perform calculations in a table.
|
| Tester's Paradise (Release 2.0) | Total Path Tests | |||
|---|---|---|---|---|
| Menu Option Paths | Exception Paths | Program Paths | TIPs | |
| Project Information: | 0 | 0 | 0 | 0 |
| Fix For Error #123 | 0 | 0 | 0 | 0 |
| Fix for Error #124 | 0 | 0 | 0 | 0 |
| Tester's Paradise Main Menu | 5 | 5 | 0 | 10 |
| Our Best System Simulator | 0 | 0 | 0 | 0 |
| Message Data Flow Checker | 0 | 0 | 0 | 0 |
| Screen Comparison - Pixel Viewer | 0 | 0 | 0 | 0 |
| Portable System Monitor (New Function) | 5 | 5 | 0 | 10 |
| Specifics and Options | 3 | 3 | 0 | 6 |
| Add-on Platform Adapters | 3 | 3 | 0 | 6 |
| View Portable System Monitor | 2 | 2 | 0 | 4 |
| Display Portable System Monitor | 0 | 0 | 0 | 0 |
| Order Form | 3 | 3 | 0 | 6 |
| Arrange Payment | 2 | 2 | 0 | 4 |
| Order Confirmation | 1 | 1 | 0 | 2 |
| Support Packages | 3 | 3 | 0 | 6 |
| Return to Main Menu | 1 | 1 | 0 | 2 |
| Cancel | 1 | 1 | 0 | 2 |
| Installation is automatic at logon | 0 | 0 | 0 | 0 |
| Totals | 29 | 29 | 0 | 58 |
Step 4: Building the Logic Flow Maps for the System
Once all the menu options have been counted and cataloged, the next step is to find out what is going on behind all those menu option doors. It is time to build the logic flow maps of the system. Ideally, the processes behind the menu options will be described correctly in the project documentation. If the documentation is inadequate, or nonexistent, the information can be collected by interviewing the developers responsible for each part of the system. If the modules come from an outside party, it will be necessary to conduct interviews with developers within that organization to construct the logic flows. Interviews of some type may be the only way to verify the correctness of the documentation prior to actual tests of the system.
Some Procedural Notes on Interviews
I use a combination of techniques to conduct these interviews. The techniques that are used to conduct inspections and the techniques that are used to conduct audits supplemented by basic sales techniques used to identify compelling events work well.
People are individuals. Each person will require a unique approach. But these techniques are simple and fundamental. They work for almost every personality type or style. They can be used alone or in combination to formulate and execute an approach that will get good-quality results with virtually anyone.
| Note | Software testing is a contact sport. |
A successful tester has a lot of good contacts. I actively cultivate the developers on every project. I usually do this simply by being interested in what they are doing. I ask a lot of questions about "how things work." I not only want to know, I need to know. It is not usually hard to get the developer talking about her or his project. But like inspections, these interviews can be done well or poorly. Developers' time is in short supply and it must be utilized profitably.
1. Keep It Short and Sweet
The interview should be kept as short as possible. Keep the questions simple. "What happens first?" "What happens next?" "If this happens, then what?" It is possible to develop a detailed logic flow diagram after only a few minutes of interview time. Keep a log of how much time was spent in each session, who was there, general notes, and the logic flow diagram, or changes to a logic flow diagram, that came out during the session. Development managers will be quick to bar a tester from distracting a developer if they do not see a benefit from these discussions. By keeping these records and measurements, I have often been able to show that these techniques allow me to create project documents at a fraction of the normal cost.
2. Focus on the Facts
There will be many bits of murky, vague, or fuzzy logic during the design and implementation stages of a project. If a person cannot explain something succinctly, he or she either doesn't understand it or the person has a communications problem. In either case, a confrontational technique is rarely the most productive approach for getting an answer.
Instead, focus attention on something inanimate, like the logic flow diagram, a theoretical representation of the project. A drawing on the whiteboard or a picture on paper (that the tester can write on and take away) is less threatening than an across-the-table face-to-face discussion. If the answers aren't immediately forthcoming, write the questions down and come back to them later. At least the questions have been asked, and the next time there will probably be answers.
3. Draw the Logic Flow Maps
Draw the logic flow maps as soon as possible after the interview while the details are still fresh. Write down all your questions, and write down the answers you are given. Keep this question-and-answer log. I usually ask the same questions many times and of many people. I pay special attention to any questions that get different answers from different people or several different answers from the same person over time.
Never underestimate your powers of observation. If something doesn't feel right, there is probably a reason. If it seems like some topic is being avoided, it probably is. The normal reason for avoiding a topic is that it is not well understood. If the developers can't clarify a point for the testers, somebody does not understand the topic. I have found that if I have this problem, I am probably not asking the right question.
4. Conduct a Review of the Logic Flow Maps with the Developers
Take the logic flow maps of the system back to the developers for review. Go over the maps with the developers; don't just drop them off. This is the part of the process where I learn the most about what is really going on.
The review, like the interview, should not take long. I ask the developers to go over the diagrams with me. I ask if I have understood them correctly and if the maps are correct. The process of reviewing these maps almost always leads to disclosures of other related processes that were not mentioned previously for one reason or another.
Usually it takes one initial session and two to three review sessions to have an accurate map of the system, as it exists in reality or in concept at the current stage of development. As the logic flow changes during development, I encourage developers to make updates to the logic flow and forward them to me. The gain is that the logic flow maps are much quicker to create, update, and maintain than wordy documents. And they are mathematically verifiable.
In a perfect world, the designers would initialize the logic flow maps and the maps would be passed along with the system as it progresses through the development cycle. The diagrams would be updated by development as the system was developed and would always be available to other groups, such as the testers. The good news is that logic flow maps are such a flexible tool it doesn't matter who draws them or when.
Back to Work: Some Things to Remember
The additive property of linearly independent paths was introduced in the previous chapter. This property will be used to arrive at the total number of paths identified in the system after the analysis digs down into the internal processes of the application to identify them separately. The first interviews should focus on understanding how many of the menu processes will need to be analyzed in depth.
| Note | The TIP is equal to the sum of the IPs in the system. |
The highest level of logic flow maps generally comes directly from the GUI menus. Figure 12.9 is the logic flow map for the menu option View Portable System Monitor, shown earlier in Figure 12.8. It will be used as the starting point for the interviews and the analysis of the logic behind the menu options.
Figure 12.9: Logic flow map for the View Portable System Monitor menu option.
In this case, the user can either Order Portable System Monitor and Return to Main Menu, or the user can Return to Main Menu without ordering. There are two paths, IP = 2, through this logic flow diagram.
| Note | Assigning any element a total independent path count of 1, TIP = 1, states implicitly that the element is assumed to be stable. Its internal paths are being ignored. |
By assuming an element in a system is stable, we are conveniently ignoring the number of actual paths being executed inside it. If there is no apparent reason to examine the internal paths of an element, it is assigned a path count of 1. The assumption is that the element will only have one effective path, the correct path. If the assumption is found to be in error, the element can be analyzed and its internal path count added to the inventory. Tests can then be written to exercise the most important of those paths.
The First Logic Flow Maps
The logic flow maps in Figure 12.10 were the result of the first interview with developers and designers. This map shows the logic flow diagrams for Display Portable System Monitor and Order Portable System Monitor. Notice that the View Portable System Monitor logic flow diagram shows the number of IPs that have been counted inside the Display Portable System Monitor process node and the Order Portable System Monitor process node. This type of notation can be used on the menus to show how many paths are actually associated with each program menu option.
Figure 12.10: Logic flow maps for the entire PSM menu.
Once the user selects View Portable System Monitor, a sequence of events is triggered. The result is that a picture of the Portable System Monitor is delivered to the user, decompressed if necessary, and displayed. The function path that determines which type of picture to send and performs whatever processing is needed to display the picture is automatic as soon as the user selects View Portable System Monitor.
The Display Portable System Monitor logic flow diagram is a process-level diagram. Normally this type of analysis is sufficient to arrive at a good estimate of the number of tests that exist in a system. If a more detailed or rigorous approach is warranted, the number of internal IPs through the system and the network can be counted. The number of paths through the source code of the modules can be determined, if necessary, by automated tools that perform source code analysis like LDRA or Cantata. It is not usually cost-effective to conduct manual path analysis at the source code level.
The Arrange Payment process was given an IP = 5 as an estimate by the group as the time allotted for the interview ran out. Such estimates are normal and healthy. It is far better to record the best guess and publish the fact that there are untrustworthy paths behind the Arrange Payment door than to leave the process with an IP = 1, which implies that it is trustworthy. But they usually require additional analysis as the analysis process moves forward.
Step 5: Recording the Paths
Record the counts of the individual IPs on the main menu map and in the inventory and total the paths of interest on the lowest-level menu. Carry the internal path counts from the lower level menus up to the next-level menu and sum them. Repeat this step until all the internal paths in the system have been counted.
Table 12.2 shows the record from the Portable System Monitor.
| DISPLAY PSM PROGRAM PATHS | IP |
|---|---|
| Display PSM | 4 |
| Order Form | 5 |
| Arrange Payment | 6 |
| Return to Main Menu | 2 |
| Total IP | 17 |
| Order confirmation has been neglected for the moment. | |
The Menu Map
Notice that there is no distinction between different types of paths. The path triggered by selecting a menu option is counted in the same way and given the same importance initially as a path through source code or a path through the network. This approach to modeling a system is effective on both finite state and real-time event-driven systems. For event-driven systems, a set of assumptions are made about the system's state.
The other three options, Our Best System Simulator, Message Data Flow Checker, and the Screen Comparison-Pixel Viewer, were analyzed in Release 1. The only difference in these options is that the order process in Release 1 instructed users to dial an 800 number and place their orders through a sales representative. For the most part, the path counts shown in this inventory for those options are taken based on the actual tests used in the Release 1 test effort.
| Tester's Paradise (Release 2.0) | Total Path Tests | |||
|---|---|---|---|---|
| Menu Option Paths | Exception Paths | Program Paths | TIPs | |
| Project Information: | 0 | 0 | 0 | 0 |
| Fix For Error #123 | 0 | 0 | 3 | 3 |
| Fix for Error #124 | 0 | 0 | 0 | 0 |
| Tester's Paradise Main Menu | 5 | 5 | 0 | 0 |
| Our Best System Simulator | 0 | 0 | 65 | 65 |
| Message Data Flow Checker | 0 | 0 | 61 | 61 |
| Screen Comparison - Pixel Viewer | 0 | 0 | 76 | 76 |
| Portable System Monitor (New Function) | 5 | 5 | 0 | 10 |
| Specifics and Options | 3 | 3 | 0 | 6 |
| Add-on Platform Adapters | 3 | 3 | 0 | 6 |
| View Portable System Monitor | 2 | 2 | 0 | 4 |
| Display Portable System Monitor | 0 | 0 | 4 | 4 |
| Order Form | 3 | 3 | 6 | 12 |
| Arrange Payment | 2 | 2 | 5 | 9 |
| Order Confirmation | 1 | 1 | 0 | 2 |
| Support Packages | 3 | 3 | 0 | 6 |
| Return to Main Menu | 1 | 1 | 0 | 2 |
| Cancel | 1 | 1 | 0 | 2 |
| Installation is automatic at logon | 0 | 0 | 0 | 0 |
| Totals | 29 | 29 | 220 | 268 |
|
|
EAN: 2147483647
Pages: 132