Making Test Scripts Dynamic

Test scripts represent the activities of a typical user. However, during the performance tests, multiple virtual users execute the same test script. Obviously, we cannot build a different script for each user when we potentially simulate hundreds or thousands of users. Instead, we parameterize one script to represent multiple users. For example, if your users log on using their account numbers , make the account number a parameter in the test script. Most test tools allow you to correlate this parameter with a list of potential values. The tool then inserts these values into the script automatically at run time. If the tool inserts a different value for each simulation of the script's login, it simulates multiple user logins with just one script. Parameters require sufficient variation to successfully reproduce user activity. Always buying or selling the same stock does not accurately represent typical user activity. Provide a sufficient pool of test data to simulate production usage.

Identifying script parameters is sometimes difficult. To more easily identify dynamic data in a script, we often record two scripts and make comparisons between them. This comparison allows us to identify the script portions that are common across multiple user requests, as well as the portions differing between requests . For example, we might record the following Browse script in Pet Store:

1. Go to home page.

2. Select category Fish.

3. A list of Fish is returned by the server.

4. Select the first fish, Angelfish, on the returned page.

After completing this script, create a new script, repeating the same steps with only minor changes in the entered data. For example:

1. Go to home page.

2. Select category Reptiles .

3. A list of Reptiles is returned by the server.

4. Select the second reptile, Rattlesnake, on the returned page.

Now you have two scripts of the same steps but using different data inputs. The differences between these two scripts give you the dynamic data to modify for each simulated user iteration. Listing 7.3 shows the two Pet Store scripts, Browse1 and Browse2, recorded from the two scenarios listed above.

Listing 7.3 Example scripts with dynamic data. 2002 Mercury Interactive Corporation.

 Browse1() { //subset of Pet Store browse script (first browse recording) //first URL request to Pet Store home page   web_url("language",      "URL=http://ruthless/estore/control/language?language=English",      "TargetFrame=",      "Resource=0",      "RecContentType=text/html",      "Referer=",      "Snapshot=t1.inf",      "Mode=URL",      LAST); //browse for fish   web_url("nav-fish.gif",      "URL=http://ruthless/estore/control/category?category_id=FISH",      "TargetFrame=",      "Resource=0",      "RecContentType=text/html",      "Referer=http://ruthless:80/estore/control/language?language=English",      "Snapshot=t2.inf",      "Mode=URL",      LAST); //browse for Angelfish   web_url("Angelfish",      "URL=http://ruthless/estore/control/product?product_id=FI-SW-01",      "TargetFrame=",      "Resource=0",      "RecContentType=text/html",      "Referer=http://ruthless:80/estore/control/category?category_id=FISH",      "Snapshot=t3.inf",      "Mode=URL",      LAST);    return 0; Browse2() { //subset of Pet Store browse script (second browse recording) //first URL request to Pet Store home page   web_url("language",      "URL=http://ruthless/estore/control/language?language=English",      "TargetFrame=",      "Resource=0",      "RecContentType=text/html",      "Referer=",      "Snapshot=t1.inf",      "Mode=URL",      LAST); //browse for reptiles   web_url("nav-reptiles.gif",      "URL=http://ruthless/estore/control/category?category_id=REPTILES",      "TargetFrame=",      "Resource=0",      "RecContentType=text/html",      "Referer=http://ruthless:80/estore/control/language?language=English",      "Snapshot=t2.inf",      "Mode=URL",      LAST); //browse for rattlesnakes   web_url("Rattlesnake",      "URL=http://ruthless/estore/control/product?product_id=RP-SN-01",      "TargetFrame=",      "Resource=0",      "RecContentType=text/html",      "Referer=http://ruthless:80/estore/control/category?category_id=REPTILES",      "Snapshot=t3.inf",      "Mode=URL",      LAST);   return 0; 

As you can see, these scripts differ in their web_url parameters. Specifically, the first script uses category_id=FISH and product_id=FI-SW-01 (Angelfish). The second script uses category_id=REPTILES and product_id=RP-SN-01 (Rattlesnake). Parameterizing category_id and product_id gives us a general script suitable for simulating multiple users.

This simple example demonstrates the fundamental concepts behind making your test scripts more dynamic. Let's further examine some of the different types of dynamic data your scripts might need to support.

Support Dynamic Decisions

Your customers do not always perform the same actions in every interaction with your application. You don't want every virtual client to execute exactly the same selections. In the Pet Store example, you don't want all users to select Fish from the home page. This does not represent real user activity from a database caching and locking perspective. You want the simulated users to follow different paths and access different data. For example, on the Pet Store home page, you want users to dynamically choose between Fish, Dogs, Reptiles, Cats, and Birds.

Select from File

Many test tools make it easy to identify the dynamic data and replace it with parameters. Instead of hard-coding the category name in the browse script, we replace it with a variable. For example, Listing 7.4 shows a subset of the LoadRunner script with the category_id replaced with a parameter named Category . Pet Store contains five fixed categories. If we provide these category values in a file, LoadRunner replaces the Category parameter with a value from the file at runtime.

Listing 7.4 Example script with parameters. 2002 Mercury Interactive Corporation.

 web_url("nav-fish.gif",      "URL=http://ruthless/estore/control/category?category_id={Category}",      "TargetFrame=",      "Resource=0",      "RecContentType=text/html",      "Referer=http://ruthless:80/estore/control/language?language=English",      "Snapshot=t2.inf",      "Mode=URL",      LAST); 

Figure 7.4 shows an example of associating the Category parameter with a file containing the five valid categories within the LoadRunner GUI. When this browse script runs, the tool selects the category sequentially from the five pet categories for each different user simulated during the run. LoadRunner optionally selects the category at random from those provided, if specified. Figure 7.4 shows file category.dat , containing the categories Fish, Dogs, Reptiles, and Cats. In this case, as shown at the bottom of the screen capture, we choose Sequential as our parameter value selection strategy.

Figure 7.4. LoadRunner parameter properties GUI. 2002 Mercury Interactive Corporation.

Select Link by Random Number

We also may insert dynamic behavior into our scripts by dynamically selecting links within a page. For example, Segue SilkPerformer assigns a number corresponding to each link in the script. ^[6] In the Pet Store demo Browse script, Fish corresponds to Link 5, Dogs corresponds to Link 6, and so forth. This allows the tool to select links using random numbers. In the subset of the Pet Store script shown below, the hard-coded link to Fish is replaced with a random number between 5 and 9 corresponding to one of the five valid links on the page.

^[6] Thanks to Stephan Asboeck for his help with all the SilkPerformer example scripts.

 dclrand //random variable    rNcategory : RndUniN(5..9); dcltrans    transaction TMain    begin //original browse link //WebPageLink("nav-fish", "Product Category"); // Link 5 //replace hard-coded link with random number between 5 and 9    WebPageLink(NULL, "Product Category", rNcategory); 

In this case, both the selection of the link randomly by number or by using a parameter from a file yield the same desired result: The first selection from the home page exercises all five options with just one script.

Dynamically Created Web Pages

In the Pet Store demo, the home page contains a predefined set of five categories, and our script dynamically selects one. However, not every page request returns a known set of options. Test scripts must also address pages containing data dynamically created by the web application. For example, Figure 7.5 shows the results from a Browse for Fish. The web application creates a web page with four links for Angelfish, Goldfish, Koi, and Tiger Shark. Looking at the hierarchy from Figure 7.2, every Browse operation within the web application returns different data and often different quantities of data. For example, the Browse operation for Fish returns four fish, while a Browse request for Reptiles returns only two reptiles, Iguana and Rattlesnake.

Figure 7.5. Result of Browse for Fish. Sun Microsystems. Reprinted by permission of Sun Microsystems.

In the previous section, we parameterized the Browse script to dynamically select a different category. Therefore, when the script runs, the number and list of pets returned differs depending on the category. To properly use one script to simulate a user who has selected Reptiles and also a different user who has selected Fish, the script needs to know the valid choices that the application generates. A simple parameter replacement approach no longer works. This is the trickiest part of writing good test scripts for highly dynamic sites.

Adding this type of intelligence to a script typically requires custom programming of the script. A common technique uses a function provided by your load driver to read the links on a returned web page. (Expect this level of functionality only in the better load drivers.) The script then selects one of these links at random as the next step in the script. This technique generally requires multiple steps, including the following:

1. Reading the returned page

2. Determining the number of links on the page

3. Randomly choosing one of the links

Because this requires custom programming, implementations vary between load tools. Let's look at two of the different ways to accomplish dynamic selection.

Using Link Numbers

Our first solution uses SilkPerformer to demonstrate dynamic selection. As shown earlier, SilkPerformer numbers each link on a page. (The numbering starts from the beginning of a recorded script.) Silk provides a special function, WebPageQueryLink , to parse a web page. In the example below, the script parses the web page for all the links containing product_id and records this count in a variable, nLinks . When this section of the script runs after selecting the category Fish, nLinks is 4; however, when this script runs after selecting the category Reptiles, nLinks is only 2.

 nLinks:=0; while WebPageQueryLink("product_id", nLinks+1) > 0 do nLinks := nLinks + 1 end; 

Looking at the link number SilkPerformer placed in the recorded script comments (see the following code segment), note that the first link on the Product page is Link 10. If our number of available links, nLinks , is 4 (indicating a Fish selection), Links 10, 11, 12, and 13 contain product links (our four Fish choices). Therefore, we want to randomly select a link between 10 and 13. However, if nLinks is 2 (indicating a Reptile selection), we only select from Link 10 or 11 (our two reptile choices). As shown in the script below, we accomplish this by replacing the hard-coded link with a random unique number between 10 and (10 + nLinks 1).

 //original link //WebPageLink("Angelfish", "Product Category (#1)"); // Link 10 //replace with random number WebPageLink(NULL, "Product Category(#1)", RndUniN(10..10+nLinks-1)); 

It's also straightforward to save the link counts from previous pages rather than hard-coding link numbers. Use the SilkPerformer WebPageQueryLink function to save the actual link names in a similar fashion to the LoadRunner example discussed next.

Using Link Names

Another solution involves reading the links from the returned web page into an array and then replacing the hard-coded link with a randomly selected link from the array. Let's look at an example of this technique using LoadRunner. First, add a call to the web_reg_save_param inside the script. This is LoadRunner's command to parse the returned web page, and save an array of values matching a left boundary (LB) and right boundary (RB) specified. (These boundaries act as parsing delimiters.) For example, the following command triggers a parse of the web page returned after selecting a category:

 web_reg_save_param("sub_pet","LB=\">\n    ","RB=\n\t  </a>",      "Search=body","Ord=all",LAST); 

After reading and parsing the links into an array, the script uses LoadRunner functions combined with standard C programming to randomly select one of the links. This section of script below demonstrates this.

 {         //determine number of links           Count = atoi(lr_eval_string("{sub_pet_count}"));           //randomly pick a url link           sprintf(Param,"{sub_pet_%d}",atoi(lr_eval_string("{Num}")) % Count + 1);          //assign the link to the Text of the LoadRunner statement          sprintf(Link,"Text=%s",lr_eval_string(Param)); } 

In this case, the first command finds the number of elements in the array (the number of links on the page). The second command randomly picks one of these links, and the third command puts this link into a string. The final script step actually invokes the randomly selected link, using the web_link command, as shown below:

 web_link("Select_Pet_Submanual",         Link,         "Snapshot=t3.inf",         LAST); 

As you can see, this is a little more complicated than simple parameter substitution, but it provides a very powerful and flexible technique to exercise many different selections. In addition, this technique keeps your scripts independent from the content of your test database. In fact, this technique works on our home page as well. Instead of choosing Fish, Dogs, and so forth from a file, we select the category at random. This gives us more flexibility to select new items as the web site changes without modifying our scripts.

Of course, this technique also allows us to test the search function of the web site (the number of links returned are not known beforehand). This also works well for web sites inserting dynamic information into their links (such as object IDs and the like). As we cannot predict the format of the link during scripting, we use dynamic link resolution to pick a link at random during runtime.

Just as with any part of performance testing, consider the trade-offs when using dynamic selection functionality from any vendor. While dynamic selection gives the scripts more flexibility, parsing each page returned may generate more overhead than a hard-coded URL strategy. However, keep in mind that dynamic data selection often provides the only viable alternative for performance testing complex web sites.

Dynamic Data Entry

We parameterized the Pet Store script for dynamic selections of links. Web sites often contain additional dynamic elements, most notably data values entered by the user. When developing a script, consider how to simulate this dynamic data entry. For example, common dynamic data entries often include the following:

1. User identifiera name or account ID

2. User-entered dataan address information, shipping information, etc.

3. Search parametersthe item to search for, matches to display, and so on.

4. Order quantitieshow many of a particular item to buy

Many test tools simplify dynamic data entry simulation. Typically, the test tool accomplishes this by automatically generating random data. Earlier, we showed dynamic selection using random data from a file as well as using a random number. For dynamic data entry, the following types of values make good candidates:

1. Current date/time

2. Random numbers

3. Unique numbers

4. Virtual user IDs

5. Data from a file

For example, the Pet Store application contains a parameter for "Quantity" . Rather than hard-coding this value, many test tools support supplying a range of random numbers (perhaps between 1 and 5 in this case) to use during the test run. The user ID also makes a good candidate for dynamic entry inside Pet Store. Several options exist for managing this variable, including generating a unique user ID from a bounded range, selecting the number from a file, or matching the user ID to the simulated virtual user ID. Whichever technique you use to generate user IDs, make sure your test database contains valid data matching these user IDs. (See Appendix C for examples of dynamic data entry in the Pet Store scripts.)

Provide Sufficient Data

As we mentioned earlier, if your web site uses a large database (containing accounts or catalog items, for example), exercise significant portions of this data within your performance test. For example, if an e-Commerce site contains 100,000 products, sample hundreds or thousands of these items in the test scripts. Otherwise, the database cache might satisfy an unusually high number of requests and skew the test with a much lower average response time. Of course, some web sites might receive a significant volume of requests for just a few items (such as a hit CD or a best-selling book) even though they carry a large database of other selections. Consider simulating these conditions in your performance tests as well.

If you need to exercise large volumes of data, keep the following considerations in mind:

• Exercise a reasonable percentage of the possible dynamic selections.

• Choose from a wide selection of dynamic data entry values.

Let's discuss each of these considerations in more detail.

Exercise a Reasonable Percentage of the Dynamic Selections

Sometimes the web site returns the database items as dynamic selections (for example, a search returning a link to each matching item). The random selection technique we discussed earlier provides an excellent solution for obtaining good coverage of these items. This technique works well because it both exercises all the links on the page and does not tie the script directly to the test data.

The LoadRunner and SilkPerformer scripts using this technique do not use the actual product_id s such as FI-xx-xx and link names such as Angelfish . This practice allows you to develop scripts with a small set of data, and yet later use these same scripts against large databases or even against the production site, all without having to change the script.

Choose from a Wide Selection of Dynamic Data Entry Values

Randomly selecting links on pages covers browse paths, but users exercise other code paths by entering data. For example, the Search function requires the test script to choose a search value from a wide selection of search attributes (such as book titles or ticker symbols). Because of caching concerns, every virtual user cannot search for the same thing. For example, we wouldn't want to test the Pet Store search feature by only looking for Angelfish.

In most cases, however, we want the value entered to match items in the database. Often, if the script provides an invalid value, the web application generates errors. (Actually, we may provide a few bad searches to test our error path performance, but these usually account for only a few of our test iterations.) Selecting this data from a file generated directly from the database usually works best. Some tools actually allow you to import valid parameter values directly from a database.