Running ACT

Running ACT

This section provides you with a starting point from which to gain familiarity with the ACT stress testing tool. A brief description of the user interface is provided with a discussion on how to create test scripts. Examples are provided for recording a test script from scratch and for leveraging ACT to create a test harness that will directly stress the SQL tier. Finally, this section concludes by providing guidelines on how to run a stress test using ACT. Other topics discussed include the optimal number of scripts to create, the appropriate number of clients to use, and the optimal amount of load to produce (measured in SBCs).

Overview of the ACT User Interface

The ACT user interface is straightforward. After launching ACT, you will see the screen shown in Figure 3-3.

figure 5-3 the act launch screen

Figure 3-3. The ACT launch screen

The window is divided into two sections a left and right pane. The left pane is used for navigation and the right pane is used to display details. In the left pane, the tree control displays a root node named ACTSamples, which represents an ACT project. Only one ACT project can be open at a time. By default, this project is stored in a folder with the same name as the project under C:\Documents and Settings\<user>\My Documents\ACT Projects. The project file with the extension .act is located in this folder and contains pointers to all items referenced by the project such as tests and reports.

NOTE
If you use a share to store My Documents, the ACTUser account must have read and write permissions to this share otherwise ACT will not be able to run tests. It will state that ACTUser does not have permissions to the folder in the status pane.

At the next level, there are three nodes: Tests, Results, and Users; these are described below:

Tests

This node enumerates all test scripts created by a user. Selecting a test script under this node displays the code in the right pane, where modifications can be made. Every test script has properties, such as duration, that control how a test script will run. To see the properties of a particular test, you simply right-click a test and choose Properties. A properties page will appear, as shown in Figure 3-4:

figure 5-4 viewing the properties of a test

Figure 3-4. Viewing the properties of a test

You can create new test scripts by right-clicking the Tests node and choosing New Test from the context menu. You can also start and stop tests through the context menu of a particular test. A single ACT client can run only one test at any given time.

Results

This node displays the results of each test run of every scenario on the right hand pane. Clicking the Results node will display the screen shown in Figure 3-5.

figure 5-5 reviewing test results

Figure 3-5. Reviewing test results

The right side of the screen is divided into three regions. The top left region lists all tests that previously ran. Under each test is a collection of results for every test run, listed in descending order by timestamp. To view the test results, you simply select a particular test run, select the type of report from the dropdown list under Report, and select the category from the list. All of the report details will be listed in the lower region of the right pane on the screen.

ACT provides different categories of reports such as Overview, Graphs, and Requests. You can view information such as requests per second (RPS), Time To Last Byte (TTLB), details on page view statistics and even performance counters through this reporting feature. All of this information is stored on the file system as XML documents. Please consult the ACT help file for further details.

Users

This node lists user groups that can be accessed by tests. Clicking on a user group displays the individual users within the group in the right-hand pane. Usernames and corresponding passwords can be added by simply selecting a blank line in the right-hand pane and typing a value. These values can also be deleted by highlighting lines and pressing the Delete key. Another useful feature that ACT supports is importing username/password combinations from a comma delimited text file. You can use this feature by selecting a user group and then choosing Import Users from the Actions menu.

Every browser connection initiated by ACT needs a username and password combination to log into a Web application. If the Web application being accessed requires Basic or NTLM authentication, this is where the users (or usernames) can be created.

NOTE
From the perspective of manageability among different test runs, you should create a user group and then create the required number of users within the user group. The number of users created within a user group has to be greater than or equal to the number of SBCs. Alternatively, if the Web application is set for Anonymous connections, you can let ACT generate users automatically through the properties page (Users tab).

Creating a Test Script

To demonstrate ACT as a stress tool, we have created some test scripts that exercise the search and browse features of the IBuySpy sample site. IBuySpy is an online store application that is intended to showcase the .NET platform and technologies. We refer to the IBuySpy sample .NET application throughout this book. A copy of this application is available for viewing at http://www.ibuyspystore.com/. You can also download and install the application from http://www.ibuyspy.com/downloads.htm.

For better manageability of the test scripts, we advise you to create a new project for each testing initiative. By creating a separate project in its own folder, you can archive the entire folder at the end of the testing cycle and know that all test scripts and data related to the testing effort have been saved. To create a project, choose New Project from the File menu. In the New Project dialog box, specify a name for the project. For simplicity we have named the project IBuySpy, and the location has also been changed to C:\Project\. This action causes a folder called IBuySpy to be created under C:\Project. The folder is empty until you save the project, at which time a file by the name of IBuySpy.act is created along with some other files which comprise the project. ACT will create separate files for reports from each test run including test properties, project properties, user groups and client groups, all of which are formatted in XML.

Recording a Test Script

A simple way to create a test script is by recording a walkthrough of a user scenario. To start a recording session, right-click the Tests node and select New Test from the context menu to bring up the New Test Wizard. Click Next and then select Record A New Test from the wizard. Proceed by clicking Next, and then click the Start Recording button. The default browser will launch at this point and you can walk through the following search user scenario.

  1. Browse to http://<YourServerName>/

  2. Enter boat in the Search text box located at the top right of the Web page.

  3. Click Go to execute the search.

After completing the scenario, click the Stop Recording button. All of the requests captured during the session are visible in the list box but cannot be modified at this point. Click Next and specify a name for the script. Click Next one more time and then click Finish. The newly recorded test should appear in the tree control on the left panel of the screen.

Below is an excerpt of a script that exercises the search feature of IBuySpy:

Search.vbs

Option Explicit Dim fEnableDelays fEnableDelays = False Sub SendRequest1() Dim oConnection, oRequest, oResponse, oHeaders, strStatusCode If fEnableDelays = True then Test.Sleep (0) Set oConnection = Test.CreateConnection("YourServerName", 80, false) If (oConnection is Nothing) Then Test.Trace "Error: Unable to create connection to YourServerName" Else Set oRequest = Test.CreateRequest oRequest.Path = "/StoreVBVS/default.aspx" oRequest.Verb = "GET" oRequest.HTTPVersion = "HTTP/1.0" set oHeaders = oRequest.Headers oHeaders.RemoveAll oHeaders.Add "Accept", "image/gif, image/x-xbitmap, image/jpeg,_ image/pjpeg, application/vnd.ms-excel,_ application/vnd.ms-powerpoint, application/msword, */*" oHeaders.Add "Accept-Language", "en-us" oHeaders.Add "User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0;_ Windows NT 5.1)" 'oHeaders.Add "Host", "YourServerName" oHeaders.Add "Host", "(automatic)" oHeaders.Add "Cookie", "(automatic)" Set oResponse = oConnection.Send(oRequest) If (oResponse is Nothing) Then Test.Trace "Error: Failed to receive response for URL to " + _ "/StoreVBVS/default.aspx" Else strStatusCode = oResponse.ResultCode End If oConnection.Close End If End Sub Sub SendRequest2() Dim oConnection, oRequest, oResponse, oHeaders, strStatusCode If fEnableDelays = True then Test.Sleep (130) Set oConnection = Test.CreateConnection("YourServerName", 80, false) If (oConnection is Nothing) Then Test.Trace "Error: Unable to create connection to YourServerName" Else Set oRequest = Test.CreateRequest oRequest.Path = "/StoreVBVS/IBuySpy.css" oRequest.Verb = "GET" oRequest.HTTPVersion = "HTTP/1.0" set oHeaders = oRequest.Headers oHeaders.RemoveAll oHeaders.Add "Accept", "*/*" oHeaders.Add "Referer", "http://YourServerName/StoreVBVS/default.aspx" oHeaders.Add "Accept-Language", "en-us" oHeaders.Add "User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0;_ Windows NT 5.1)" 'oHeaders.Add "Host", "YourServerName" oHeaders.Add "Host", "(automatic)" oHeaders.Add "Cookie", "(automatic)" Set oResponse = oConnection.Send(oRequest) If (oResponse is Nothing) Then Test.Trace "Error: Failed to receive response for URL to " + _ "/StoreVBVS/IBuySpy.css" Else strStatusCode = oResponse.ResultCode End If oConnection.Close End If End Sub Sub SendRequest3() Dim oConnection, oRequest, oResponse, oHeaders, strStatusCode If fEnableDelays = True then Test.Sleep (20) Set oConnection = Test.CreateConnection("YourServerName", 80, false) If (oConnection is Nothing) Then Test.Trace "Error: Unable to create connection to YourServerName" Else Set oRequest = Test.CreateRequest oRequest.Path = "/StoreVBVS/images/sitebkgrdnogray.gif" oRequest.Verb = "GET" oRequest.HTTPVersion = "HTTP/1.0" set oHeaders = oRequest.Headers oHeaders.RemoveAll oHeaders.Add "Accept", "*/*" oHeaders.Add "Referer", "http://YourServerName/StoreVBVS/default.aspx" oHeaders.Add "Accept-Language", "en-us" oHeaders.Add "User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0;_ Windows NT 5.1)" 'oHeaders.Add "Host", "YourServerName" oHeaders.Add "Host", "(automatic)" oHeaders.Add "Cookie", "(automatic)" Set oResponse = oConnection.Send(oRequest) If (oResponse is Nothing) Then Test.Trace "Error: Failed to receive response for URL to " + _ "/StoreVBVS/images/sitebkgrdnogray.gif" Else strStatusCode = oResponse.ResultCode End If oConnection.Close End If End Sub  Sub SendRequest17() Dim oConnection, oRequest, oResponse, oHeaders, strStatusCode If fEnableDelays = True then Test.Sleep (6910) Set oConnection = Test.CreateConnection("YourServerName", 80, false) If (oConnection is Nothing) Then Test.Trace "Error: Unable to create connection to YourServerName" Else Set oRequest = Test.CreateRequest oRequest.Path = "/StoreVBVS/SearchResults.aspx" oRequest.Verb = "POST" oRequest.HTTPVersion = "HTTP/1.0" oRequest.EncodeBody = False set oHeaders = oRequest.Headers oHeaders.RemoveAll oHeaders.Add "Accept", "image/gif, image/x-xbitmap, _ image/jpeg, image/pjpeg, application/vnd.ms-excel,_ application/vnd.ms-powerpoint, application/msword, */*" oHeaders.Add "Referer", "http://YourServerName/StoreVBVS/default.aspx" oHeaders.Add "Accept-Language", "en-us" oHeaders.Add "Content-Type", "application/x-www-form-urlencoded" oHeaders.Add "User-Agent", _ "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)" 'oHeaders.Add "Host", "YourServerName" oHeaders.Add "Host", "(automatic)" oHeaders.Add "Pragma", "no-cache" oHeaders.Add "Cookie", "(automatic)" oHeaders.Add "Content-Length", "(automatic)" oRequest.Body = "txtSearch=boat&image1.x=13&image1.y=11" Set oResponse = oConnection.Send(oRequest) If (oResponse is Nothing) Then Test.Trace "Error: Failed to receive response for URL to " + _ "/StoreVBVS/SearchResults.aspx" Else strStatusCode = oResponse.ResultCode End If oConnection.Close End If End Sub Sub Main() call SendRequest1() call SendRequest2() call SendRequest3()  call SendRequest17() End Sub Main

Analyzing a Recorded Test Script

At a high level, you can see that each request is encapsulated in a subroutine labeled SendRequest#() where # is an ordered sequence. All calls to these subroutines are then encapsulated within a Main() subroutine. The starting point of execution is at the top of the test script where variables are declared and initialized. At this point, execution jumps to the last line where Main() is called. Within the Main() subroutine, each request is called and then the script ends. Subsequent iterations repeat all the steps described above.

The code makes use of ACT objects that facilitate connecting to a Web server and sending requests. These objects fall into what is known as the Test Object Model. At the root of this object model is the Test Object. This particular test script uses the Test, Headers, and Request objects. ACT also supports an Application Model that can be utilized to run tests programmatically. More details on the two object models can be found in the help file installed with ACT.

The search test script that we recorded demonstrates how to issue GETs (subroutine SendRequest1) and POSTs (SendRequest17). Additionally, you can see how to specify request headers. For example, to get ACT to handle cookies automatically during script playback, you can add the following line of code as illustrated in the previous recorded test script:

oHeaders.Add "Cookie", "(automatic)"

To send content in the request body, as is the case with POST requests, ACT will automatically determine the content length using the following line of code as also illustrated in the previous recorded test script:

oHeaders.Add "Content-Length", "(automatic)"

By examining the search test script and test scripts included in the ACTSamples project, which is part of Visual Studio .Net, you can gain familiarity with the object model, which will enable you to create and modify tests effectively.

Creating a Test Script Manually

Step 2 of the New Test Wizard allows for an empty test to be created. A walkthrough with the wizard produces the following single line script:

Test.SendRequest("http://localhost")

The argument in the above line can be changed to exercise the task of hitting the default page for the IBuySpy application. The line below illustrates this:

Test.SendRequest("http://YourServerName/StoreVBVS")

The left side of the default page of the IBuySpy application contains links for each of the product categories. Moving the mouse over the product category links displays the URLs in the status bar. By adding more lines to the test script, similar to the one to access the default page, a test script can be created to simulate browsing through the product categories. Additional logic can be added to randomize the selection of links between test script iterations. An example of such a test script follows.

Browse.vbs

Option Explicit '///////////////////////////////////////////////////////// ' Description: Browse different products in random order. ' Summary: ' Below is a sample ACT test script that simulates ' browsing through different product categories in ' random order. '///////////////////////////////////////////////////////// Dim i, sParams, sServerName, oUser '///////////////////////////////////////////////////////// ' TODO: -Enable Integrated security only, via the Security ' -tab in IIS ' -Replace these sample values with your site values ' -Set the username (domain\username" and password sServerName = "YourServerName" 'Get a reference to the current ACT generated user Set oUser = Test.GetCurrentUser oUser.Name = sServerName & "\USERNAME" oUser.Password = "UserPassword1" '///////////////////////////////////////////////////////// 'Generate a new seed for random numbers Randomize() 'Generate a random number between 0 and 6 i = Cint(7 * Rnd()) 'Output the random number to the trace file Test.Trace "i = " & cstr(i) Select Case i Case 0 sParams = "CategoryID=14&selection=0" Case 1 sParams = "CategoryID=15&selection=1" Case 2 sParams = "CategoryID=20&selection=2" Case 3 sParams = "CategoryID=18&selection=3" Case 4 sParams = "CategoryID=17&selection=4" Case 5 sParams = "CategoryID=19&selection=5" Case 6 sParams = "CategoryID=16&selection=6" End Select 'Request the default page Test.SendRequest("http://" & sServerName & "/storevbvs/Default.aspx") 'Request a product category Test.SendRequest("http://" & sServerName & _ "/StoreVBVS/productslist.aspx?" & sParams) 'This Request has been added to mark the end of a user scenario Test.SendRequest("http://" & sServerName & _ "/storevbvs/Default.aspx?test=count")

This test script was set to generate users automatically, and the Web application was configured to require integrated security. To allow a successful login by the test script, a reference to the user object generated by ACT is obtained and then a username and password are specified through the properties of the User Object. You would have to create this username as a local account on the Web server prior to running the ACT test script.

NOTE
The last request in the test script demonstrates how to add a marker to the IIS log (W3SVC format). We choose test=count in our example since it is not interpreted by the Web application logic. You can use this to count completed user scenarios (transactions) after the test run. Another instance where this technique is useful would be in identifying throughput for Web applications in which you have different parameters representing different transactions which are passed to the same page.

Creating a test script to Stress the SQL Tier

ACT can also be used to make SQL calls through ActiveX Data Objects (ADOs). This is useful since ACT supports the creation of multiple threads and can thus simulate multiple concurrent connections to the database. Another reason for using this method would be if you are at a point in the development life cycle where the Web and data tiers are not ready to be integrated.

NOTE
You should ensure that the SQL Server User Options (that is, the Transaction Isolation level) for the SQL connection created by the test script and those for a connection created by the application are the same.

By starting with an empty test script, you can connect to a SQL server and execute a stored procedure or any dynamic SQL statements. For example, the search functionality of the IBuySpy application calls the following stored procedure:

ProductSearch @Search = N'search string'

The test script that follows demonstrates connecting to a SQL server directly using the Scripting Library and ADO and exercising the database component of the search functionality exposed by IBuySpy. This script also shows how to retrieve input data from an external file.

Search_SQL.vbs

Option Explicit '///////////////////////////////////////////////////////// ' Description: Directly Stress SQL Tier via ADO calls ' Summary: ' Below is a sample ACT test script that be used to ' make SQL calls via ADO. This script passes ' parameters from a text file to execute a search ' stored procedure. '///////////////////////////////////////////////////////// Dim oFso, oCsvFile, oSQLConn, sConnStr, sSearchStr, sServerName Dim sUserName, sPassword, sPath '///////////////////////////////////////////////////////// ' TODO: - Set the Server Name ' - Create a SQL server account with dbo privileges ' to the Store database ' - Verify path to the input file search.txt sServerName = "juaceeval5" sUserName = "USERNAME" sPassword = "UserPassword1" sPath = "D:\Chapter_3\ACT_Project\search.txt" '///////////////////////////////////////////////////////// 'Create an ADO Connection object Set oSQLConn = CreateObject("ADODB.Connection") sConnStr = "driver={SQL Server};Server=" & sServerName & "_ ;Database=Store;u;pwd=" & sPassword oSQLConn.Open sConnStr '///////////////////////////////////////////////////////// ' Open a text file containing search sConnStrings Set oFso = CreateObject("Scripting.FileSystemObject") Set oCsvFile = oFso.OpenTextFile(sPath ,1) '///////////////////////////////////////////////////////// 'Repeat until end of file is reached Do While oCsvFile.AtEndOfStream <> True 'Read a sSearchString from the text file sSearchStr = oCsvFile.ReadLine 'Output the search sConnString to the ACT log file Test.Trace sSearchStr '///////////////////////////////////////////////////////// ' Execute the ProductSearch stored procedure oSQLConn.execute "exec ProductSearch @Search = N'" &_ trim(sSearchStr) & "'" '///////////////////////////////////////////////////////// Loop 'Close the text file oCsvFile.Close Set oCsvFile = Nothing Set oFso = Nothing 'Close the database connection oSQLConn.Close Set oSQLConn = Nothing

Setting Test Properties Prior to Script Playback

Before running a test, there are certain test properties that must be set to override the default settings. These properties can be set via the test property page. The following properties can be set:

  • Test load level

    (General tab). This is achieved by setting a value for simultaneous browser connections. One browser connection can be thought of as virtual users accessing the application. For example, if you want to simulate 10 concurrent connections to the Web application, set this to 10. ACT will spawn 10 SBCs, each of which will execute the test script.

  • Test Duration

    (General tab). There are two choices here; either a time can be set or a specified number of iterations can be set. It should be noted that the number of iterations is the total number of iterations across all connections. If the simultaneous browser connections setting is 10 and the number of iterations setting is 100, then each connection will execute the script such that the sum of all executions across the 10 connections does not exceed 100 iterations. For a stress test, we typically set the duration by specifying a time value which allows sufficient warm-up time for the load to reach a steady-state. This time value can be determined by observing the CPU utilization or requests per second for various sample runs of the test script against the server. ACT allows for this transient behavior to be eliminated in its reports by letting the user specify a warm-up time.

  • Detailed Reporting

    (General tab). ACT can summarize the statistics for each request by page name, or it can simply aggregate the statistics across all requests. To get more detailed reporting, the check box Generate detailed test results located on the Advanced Settings dialog box should be checked. Clicking the Advanced button on the General tab of the test Properties dialog box brings up the Advanced Settings window.

    NOTE
    To reduce the reports generation time and out of memory errors generated from long test runs, we recommend that this setting be turned off for tests that generate a large number of unique requests.

  • Users

    (Users tab). ACT can generate users automatically, or you can specify a user group which has predefined users. For Web applications that require specific username/password combinations, users can be created in a user group and then the group can be selected through this tab from the properties page. The number of users defined in the group needs to be equal to or greater than the number of simultaneous browser connections. Note that all iterations of the script will use the same user unless the test script programmatically calls the GetNextUser method of the Test object. More details on the Test object s methods are discussed in the ACT help file.

    ACT can speed up the creation of users by generating users for a test. You can invoke this feature by first creating a user group. Next, select the user group and choose Generate Users from the Action menu. This will bring up the Generate Users dialog box, where the number of users, username prefix, and password are specified. The newly created user group must also be associated with a given test for this user group to be used.

  • Counters

    (Counters tab). The same counters that are captured through Performance Monitor are exposed on the Counters tab. You can set the sampling duration by specifying a value for the Counter interval. To add counters, click the Add button and then from the Browse Performance Counters dialog box select the target machine and the desired counters. Counters added will appear in the Performance Counters list box on the Counters tab of the test property page.

Script Modification Avoiding Record and Playback

Web applications typically have dynamic content that is generated based on user input. These applications often have a data tier implemented using a database management system such as SQL Server. For this reason, a simple record and playback of a test script will not always accurately simulate real-world usage. To illustrate, a scenario may involve exercising the search functionality of an application. Recording such a scenario will only capture one search string that was entered during the recording session. Playing back such a script for multiple iterations will not create realistic traffic, because the search query and search results may be cached. In such a case, the test results will be skewed, and the test will not be representative of actual user interaction with unique search strings.

Recording is a quick and easy starting point in test script creation and can be used to capture most, if not all, of the Web requests, such as GET and POST methods. Once a recorded test script has been created, you may have to modify the code to allow for dynamically changing user data. For example, a Web application may have a scenario which involves user account registration as in an online store. Recording such a test script will embed data like the username that was used at record time. Playing back such a test script does not work since the user account will have already been created. This test script has to be modified to use a different value for the user account between successive iterations so the same account previously created will not be attempted more than once. One approach is to modify the test script to read user account data from a text file and supply this data as part of the Web request. You will have to create sufficient data so that the test script can run for a desired duration or complete a desired number of iterations.

NOTE
To avoid having to create enormous amounts of data that can be used for successive test runs, you should backup your application s SQL databases before your test run and restore the database to its previous state before each subsequent test run. This method will allow you to use the same data file for repeated iterations of the test.

There may be scenarios that require data generated in one step to be used in subsequent steps, such as a GUID referencing an address. In such a case, the test script may have to be modified to parse through the response from a request obtaining the dynamic data which can then be assigned to a variable and subsequently used in the following steps. Fortunately, ACT exposes a Response object for such a situation to capture the Web response of a client request. For an example of using the Response object, see the code sample in the section "Parsing ViewState within ACT".

Debugging ACT Test Scripts

ACT helps with modification and debugging of test scripts by supplying a log file which can capture messages generated during playback. The ACT log file is named ACTTrace.log by default and is located under Program Files\Microsoft ACT. Through the Project Property page (Debugging tab), you can change the location of this file, specify a maximum size for this file (after which it gets recycled) and enable or disable logging entirely. The Test object also allows for tracing to be controlled programmatically. Tracing levels can be set via the TraceLevel property and user defined messages can be output via the Trace method. You can also use the trace log file to uncover errors at runtime during test execution. The example code that follows demonstrates setting the trace level to log all messages and to write a custom message to the trace log file:

Test.TraceLevel = -1 ' The tracelevel property can be set to one of the following values ' -1: log all information ' 0: disable logging ' 1: log internal program information only ' 2: log external information only, from the Test.Trace method. ' This is the default setting. Test.Trace("Trace level is set to " & CStr(Test.TraceLevel))

NOTE
Note that unless the TraceLevel is set, no data will be written to the ACTTrace.log unless you specifically call Test.Trace.

Test Script Verification

Test scripts should always be verified for accuracy before engaging in an actual test run. Some scenarios may only read data from a database while others may insert, update or delete data. You should verify such data changes before a test script is considered complete. For example, scenarios that involve data insertion can be checked by counting rows of data within your database before and after a test run and comparing the numbers with a walkthrough of the scenario. The number of data insertions for a single user walkthrough and the number for a single test script execution should be the same for an accurate test script.

Another good technique to verify test script functionality is to look at the IIS log file. By comparing the IIS log of a walkthrough and the IIS log of a test run, you can ensure that all the right requests are being performed in the correct order. Occasionally, applications may perform some action via client-side script which does not get captured. By parsing through the IIS logs you can find such omissions.

NOTE
Note that this chapter specifically discusses Microsoft Internet Information Server; however, ACT can be used to apply stress against any Web server for performance testing.

Test script verification can include both capturing SQL calls through SQL Profiler for a walkthrough and comparing these calls to ensure that all the necessary calls are being generated in the correct sequence. SQL Profiler configuration is discussed in more detail in Chapter 8.

Scripts and Load Clients

ACT Developer Edition was not designed to run on multiple machines. It is only licensed to run on one client at a time. Depending on the application being tested, one client machine may be inadequate to generate sufficient load to effectively stress your Web application without the client machine itself becoming the bottleneck.

NOTE
Client bottlenecks can be identified when the resources such as CPU, Memory, and Network I/O are fully utilized. Another method to check a client bottleneck, if you suspect one, is to run a test to determine the throughput and then repeat the same test by dividing up the SBCs across additional clients. If the throughput is much greater when you add additional clients, then it is safe to assume that your client was a bottleneck.

Most Web applications that we test require multiple client machines to be used in generating sufficient load. With ACT, the only way to do this is to purchase and install additional copies of Visual Studio .NET for each required stress client. You then must recreate the tests, setting the properties and users individually on each machine. To aid in this task, you can simply copy the entire project folder to each client machine, modify input data files where necessary, open up the project and run a particular test.

NOTE
The maximum number of browser connections in ACT is limited to 2000. This should be more than sufficient for a single client machine. If for some reason you require more than 2000 browser connections for a single load client, you can modify the XML test properties directly, but you should not open the properties page in the user interface after making the change. Alternatively, you can double-up the connections per thread by adding another CreateConnection within your test script.

Another possible reason for using multiple stress clients is that ACT executes one test run per stress client at a given time. Most applications have more than one scenario and you can run multiple scenarios concurrently (a mixed test) by using at least one client machine per scenario.

This raises the question of how many test scripts need to be created. Should you use one large test script that exercises all of the scenarios, or should you use multiple test scripts, one for each scenario? We prefer to create multiple test scripts, because this facilitates separate test runs for each scenario and helps us identify bottlenecks in individual scenarios. Alternatively, you can execute multiple user scenarios with one large test script using the following logic:

Sub main() WhichScenarioToRun = Int(Rnd * 100) Randomize Select Case WhichScenarioToRun Case 0 To 75 Call myTest1 Case 76 To 87 Call myTest2 Case 87 To 100 Call myTest3 End Select End Sub

For the IBuySpy sample application, we have created separate test scripts for Browsing, Searching, Registration, and Checkout which can be found on this book s companion CD.

Executing a Performance/Stress Test

Determining how much stress is sufficient depends on the goals outlined prior to testing. You have to define the criteria which indicate when appropriate stress levels are reached. To aid in this decision, some examples of possible test criteria are included:

  • Stop increasing load when an excessive number of errors occur in the event or Web server logs.

  • Increase load to a point where throughput diminishes.

  • Set a threshold on CPU utilization (for example greater than 75 percent).

  • Set a threshold on memory utilization.

  • Set a threshold on page response time.

  • Obtain a predefined number of requests per second or concurrent connections that the Web application must be able to handle, based on business requirements.

Once the test criteria have been defined, several test runs are required at different load levels (simultaneous browser connections). By starting off low and letting the test execute for a sufficient time to allow an adequate number of user scenarios to be completed, you can determine the throughput for a particular scenario within an application. While increasing load, you should also monitor the stress on the client machine to ensure that the stress client is not saturated. Additional clients can be used to ensure that individual client machines do not become a bottleneck or a source of errors during a stress test. During these preliminary tests, which we refer to as smoke tests, you should monitor the application servers to ensure that all the performance criteria are being met. You can also generate a tabulation of application throughput. By repeating these tests, you will eventually reach a point where one or more of the threshold criteria are met. This is an indication that the appropriate level of load has been reached. At this point, you are ready to conduct a test run of longer duration with the appropriate load using only the necessary monitoring tools to minimize the server s resource utilization.

Depending on the scope of testing, you may have one or more test scripts. We recommend that separate tests be run for each scenario in addition to running all scenarios concurrently with a specified weighting. Running scenarios separately will aid in uncovering bottlenecks in the system or code that affect only that specific area of application logic. These isolated bottlenecks will help you in identifying later bottlenecks that are due to code interaction in the mixed scenario test run. The mixed scenario test run (where all scenarios are run concurrently) helps stress the application in a more realistic fashion, representative of actual application usage. This will help you reveal application bottlenecks, such as blocking and timeouts, that are due to the interaction of all scenarios.

Overall test duration is another important variable to define in the stress testing effort. This variable really depends on how fast an application can process requests and complete user transactions (scenarios). Other factors that can influence the length of a test are the length of time required to reach a steady state and the amount of available test data. Stress tests that are used to calculate throughput under load are typically run for at a minimum of 5 to 30 minutes.

NOTE
Total Transactions = Total successful Page View/s or SQL Row Counts that indicate a completed scenario

Transactions per minute = Total Transactions / Test time interval

Transactions per second = Transactions per minute / 60 sec.

Extended tests can run for a minimum of eight hours under load or can last up to a week. However, the exact length of time really depends on the application being tested. Such tests are useful for determining the stability of the application, uncovering memory leaks and monitoring application throughput over a longer period of sustained stress.

Create a Controlled Environment

In addition to determining the adequate number of browser connections and the optimal number of stress client machines, we advise you to conduct testing in a controlled environment. Controlled implies that there should be machines designated as stress clients and others as servers which host applications. These machines should be as close as possible in hardware specifications to your production environment. All systems should be connected on a private network with adequate bandwidth. Test results are of little value if performance issues cannot be reproduced. A controlled environment helps ensure test reliability by eliminating non-test related network noise.

The absence of a controlled environment can potentially skew test results by introducing dependencies on external servers for DNS lookups, authentication, and so on. Depending on the time of day that the tests are conducted, network delays can change based on network congestion, which could affect application throughput. If such dependencies on external servers do exist, the testing schedule can be affected due to problems with one or more of these servers, which delays the overall stress testing effort. With a controlled environment, you can also be certain that the entire load generated on the application servers is solely due to the stress clients.



Performance Testing Microsoft  .NET Web Applications
Performance Testing Microsoft .NET Web Applications
ISBN: 596157134
EAN: N/A
Year: 2002
Pages: 67

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net