8.2 Testing Tools

Besides the hardware required for a test system, software is necessary. In this regard, the tester can pursue a course anywhere along a continuum between two extreme strategies. One extreme is to merely take existing email programs and adapt them to the test environment. The other extreme is to write all test software from scratch. An organization must always do at least some work, no matter how much it would like to use only existing code. Even so, anyone writing a complete system can typically benefit by leveraging portions of existing tools or at least libraries to reduce the amount of effort necessary to produce a test system. This section will discuss some popular Open Source software packages available on the Internet that can be adapted for load testing, consider which pieces must be developed by the tester, and discuss places where home-built tools might provide considerable benefits compared to existing software.

8.2.1 sendmail as Source

Not surprisingly, the first software tool we'll consider is sendmail itself. The sendmail MTA can be used effectively as both a source and a sink for SMTP messages. A source server that generates email messages to send to the target must have software on it that can speak SMTP, or at least fake it well. The SMTP protocol, especially if one includes the ESMTP extensions, can require some fairly complex implementations. In most environments, it's not worthwhile to duplicate this effort. However, we don't want or need to run a normal sendmail implementation on these servers. As this effort isn't "real" email, we can safely cut some corners to improve performance. Following is a sample .mc file for use on a Linux-based source machine using sendmail version 8.12:

 OSTYPE(linux)  FEATURE('nocanonify')  FEATURE('use_cw_file')  define('QUEUE_DIR','/ramdisk/mqueue')  define('confDELIVERY_MODE','interactive')  define('confSAFE_QUEUE','False')  define('confDF_BUFFER_SIZE','110000')  define('confXF_BUFFER_SIZE','16384')  define('confCHECKPOINT_INTERVAL','0')  define('confQUEUE_SORT_ORDER','Filename')  define('confDEAD_LETTER_DROP','/var/mail/dead.letter')  define('confREFUSE_LA', '25')  define('confQUEUE_LA', '100')  define('confTO_IDENT','0')  MAILER(local)  MAILER(smtp) 

Before we go through this file, let's discuss some prerequisite machine configuration issues. In this example, we'll use a Linux server, although any operating system would suffice. We'll run a name daemon on this machine to speed DNS resolver lookups. When we send email messages from this machine, we will do so with a script that assembles a message and pipes the data to sendmail invoked from the command line. We want the queue to be as fast as possible, so we create a memory-based file system, which the Linux folks call a "RAM disk," and mount it on /ramdisk. We also create a directory called mqueue under /ramdisk that sendmail will use as our queue directory. We set appropriate permissions on these two directories. That is, we make /ramdisk readable, writable, and executable by root and readable and executable by everyone else. We make the mqueue subdirectory owned by the user root and the group mail, and make it readable, writable, and executable by the owner and inaccessible by everyone else.

In some cases, the source machines will mimic several users from several domains. In this circumstance, each domain should be set up on the name server, and each user name should be set up in the passwd or aliases files so that bounce messages can be recovered and inspected later. Each host or domain name that a source or sink is pretending to be should be recorded in /etc/mail/local-host-names. A regular sendmail daemon should be ready to receive mail on each source host specifically to receive DSNs from the target. After each test run, we inspect the mailboxes for the sending users to identify bounced messages. If any are found, the reasons for them should be well understood and be acceptable if they were generated in the production environment. Before running a new test, these mailboxes and the message queue should be cleaned out and the log file rotated to prevent statistical contamination from occurring between tests. In a Linux environment, email-related file names listed in the syslog.conf file should be preceded with a "-" sign so that the system won't become bogged down waiting for them to be fsync()ed.

Now that the server configuration is well defined, we can look through the source machine .mc file. The familiar OSTYPE macro appears at the top of the file. Next, we add the nocanonify feature. If this feature is turned on, then sendmail does not attempt to fully qualify domain names, but rather assumes they are already fully qualified. Therefore, sending email to [e-mail] will be handled properly but npc@host won't, even if the domain of the server is example.com or host.example.com is the local server's name. Because all the sending and receiving addresses are explicitly listed for the test software on the source machine, it's a simple matter to make sure they're always fully qualified, and so the extra resources canonification consumes on the server won't be needed. This gain will generally be very small, but a gain nonetheless. We also add the use-cw-file feature so that sendmail will consult the /etc/mail/local-host-names file.

Next, we see that the queue is defined so as to reside in our memory-based filesystem. This choice is made for speed reasons: We don't want disks moving for mail to be queued as we're sending email messages. If the operating system used on our source machines doesn't support a memory-based filesystem, or if it doesn't have enough memory to use one as a queue without swapping, then the queue should go on a separate disk, if possible. If a physical disk holds the queue, it would be appropriate to mount it using the async and noatime options because the data it will contain does not require protection.

To further reduce the amount of queue writing needed, we set the delivery mode to be interactive and turn the SuperSafe option off. We're writing messages to a memory-based filesystem, so not running fsync() doesn't reduce data integrity in the face of a system crash, and we want as much throughput as we can get. Note that this case is one of the very few occasions where it might be considered acceptable to set SuperSafe to False. As neither the queue nor the logs are being saved synchronously, accurate information about a test run may not be recoverable if a server reboots during the test. If this event happens, the test must simply be rerun, which shouldn't create a significant burden.

The DataFileBufferSize generally should be configured to be just a little bit larger than the largest message that will be sent from the source. If buffering the data causes the source machine to start to swap, the best solution is not to lower this value, but rather to add more memory to the source. Many people have an ill-advised tendency to acquire load generation machines as inexpensively as possible. A better strategy is to provide as much load on the target server for as little money as possible, which is an entirely different thing. Underpowering these machines tends to waste money. Even though these servers will never be put in production, they do need to be fast. There's no reason to buy gold-plated components, but one shouldn't skimp either. While an xf file is unlikely to grow as large as the 16,384 bytes buffered in this configuration file, setting the configuration parameter to this value won't hurt us. Again, if there might be a large number of recipients per message because one is simulating mailing list behavior, then it might be beneficial to raise this number. Because CheckpointInterval controls how often the qf file in the queue is updated, we set this value to zero to prevent updates from occurring.

We set the queue sorting algorithm to be Filename to require the least resources in processing the queue. In most tests, all messages in the queue will head to the same server anyway, so a more sophisticated sorting algorithm would consume resources to no useful purpose. We fully expect that queue runners won't be necessary to deliver messages unless the target server becomes saturated. In fact, we may not start a sendmail process using the -q flag at all.

The Dead Letter Drop serves as the repository for email that a server can't deliver and can't bounce. Such email messages shouldn't occur, but sometimes they do. They are more likely to be generated in a test environment, which explains why we list the repository's location explicitly. For example, we might forget to create an account or alias for the user name test38, which will send email from a source machine. We might also forget to configure the target machine's /etc/mail/local-host-names file to include target.example.com. If the source sends email from [e-mail] to [e-mail], the message will bounce from the target. The sending host will try to deliver it to the sender. If that attempt fails, it will try to deliver the message to the postmaster alias on the local host. If it still can't deliver the message, the email will go in the Dead Letter Drop. This file should also be inspected and cleaned out after every test run.

We don't care if the load generator becomes saturated, so we set QueueLA to an absurdly high value. We want to do so explicitly because this environment involves a Linux system. As has already been mentioned, Linux is one of the few operating systems where threads of execution blocked waiting for I/O count toward the system's load average. We also set RefuseLA to a high value, although we don't expect our source machine to receive a large number of messages at all. Finally, our source machine does not need to verify the real user name associated with any email it receives, so we turn IDENT checking off.

After setting up the server, we want to be able to send messages. The simplest tests involve a single sending address and domain and a single test message. The most naive attempt might look something like the following:

 #!/bin/sh  # sendmessage.sh  while [ 1 ]  do           /usr/sbin/sendmail -t -f [e-mail] \                < /tmp/messagefile  done 

where the file messagefile contains

 To: [e-mail]  Subject: Testing  This is a test message. 

This test script uses the -f flag to communicate the sending user to sendmail. The -t flag suggests that the recipient should be determined from the To: line in the header of the message itself. Usually, we'll want to send messages that are more representative of real traffic, but it's easy enough to invent and generate a larger message.

This method has several problems. First, it will send only one message at a time. Under no circumstances will this technique be able to maximize the source machine's throughput, because most of the time it takes for both ends to process this message will be dominated by the latency of the SMTP request/response conversation, even over a fast local network. Second, it doesn't exercise multiple users or multiple sender or recipient domains. Third, a single small message doesn't accurately represent the distribution of traffic flowing through the system. All of these problems can be remedied.

The easiest way to fix the "one message at a time" problem is to run several copies of the test script in parallel. How many should be run? The answer depends, but one can readily fire off, for example, two of them, and then keep launching more scripts in groups of two until some part of the load generator or target becomes saturated. Once the system has reached a "steady state," where the number of relevant processes such as sendmail, mail.local, test scripts, and so forth on all machines involved in the test stop varying significantly over time, then measurements can be taken.

One can simulate multiple users by using a randomizing function with a predefined list of names. For example, it is trivial to create a program in C, which we'll creatively call random, that will print out a random number between 1 and an integer argument (inclusive) given on its command line. Its usage might look like the following:

 % random 8  4 

This code could be used to select a random entry from a file containing a list of N entries:

 line='random N'  sender='head -n $line sender_list.txt | tail -n 1' 

After some minor extra work, the same process can be applied without needing to know the number of lines in the file list:

 maxlines='wc -l rcpt_list.txt'  line='random $maxlines'  recipient='head -n $line rcpt_list.txt | tail -n 1' 

Faster and more efficient ways to do this exist, but this example is about as simple as matters get. Often, the time taken by the person in writing the test scripts is at least as valuable as the amount of time saved by using fewer load generation resources. The simple method should be demonstrated to be inadequate before more complex mechanisms are pursued.

The same method used to select recipients can be used to select from a variety of message bodies. Suppose we want to test a new email gateway and to reproduce in our test environment both the aggregate amount of traffic per message and the distribution of various message sizes. After creating a histogram from analyzing all the size= fields from our sendmail message logs, we find that we can reasonably approximate the distribution of our messages as follows:

Size Range	Average	Frequency
<4KB	2.5KB	0.6
4 32KB	8.5KB	0.3
>32KB	128KB	0.1

First, we create a directory, called messages, in which to store our standard messages. Because 60% of our messages are less than 4KB, and the messages in this range average 2.5KB, we create six files in this directory with that size. This can be simply the first 2.5KB of the /usr/share/games/fortune, /usr/dict/words, or similar file on any available system. To make matters even simpler, we can call these files "1" through "6". The files "7", "8", and "9" would each be 8.5KB of data, and the file "10" would be 128KB in size. More or fewer files of different sizes or types could be substituted depending on what a histogram of the actual sizes suggests.

Putting this information together, we can alter our email submission script to look like the following:

 #!/bin/sh  # sendmessage.sh  # Determine number of recipients, senders, and messages.  # We have to do this only once, so it goes outside the loop.  maxrcpts='wc -l rcpt_list.txt'  maxsenders='wc -l send_list.txt'  maxmessages='ls messages | wc -l'  while [ 1 ]  do       # Generate parameters for email message to be sent.       messagefile='random $maxmessages'       rcpt_number='random $maxrcpts'       sender_number='random $maxsenders'       recipient='head -n $rcpt_number rcpt_list.txt | tail -n 1'       sender='head -n $sender_number send_list.txt | tail -n 1'       # Send the message             /usr/sbin/sendmail -f $sender $recipient < \                   messages/$messagefile  done 

It should be straightforward to modify the script to add the possibility (and probability distribution) of multiple recipients per message. It is trivial to write a script that would fire off some number of these load generators in parallel. Extending this script even further to more closely approximate the production load is left to the reader's imagination. (As an aside, the specific testing done earlier in this book was accomplished with scripts that are almost identical to the ones presented here.)

8.2.2 sendmail as Sink

Next, we examine the use of sendmail as an SMTP sink. In this case, we want to absorb messages as quickly as possible, but we really don't care about the integrity of those messages, nor would we even need to store them in most cases.

As far as basic machine setup, much as in the source example, we will run a name daemon on this server to speed DNS queries. We will also set up the queue, if possible, on a memory-based filesystem. All hosts and domains for which this server is supposed to receive email will be added to /etc/mail/local-host-names. We also want to add each recipient user name to whom email will be sent to the /etc/mail/aliases file as follows:

 user1:      /dev/null  user2:      /dev/null  user3:      /dev/null  ... 

Now the sink won't have to write all email messages it receives to disk, but rather will just eliminate them upon reception. Obviously, this method eliminates disk writes. Someone might want to verify that the messages are actually received intact, which would require writing the messages out to a message store. Generally, however, we would need to be this careful only if we were testing a new LDA or had modified some part of the code that processed the email messages along the way. For testing configurations, it usually suffices to compare the size of the email message on the sending machine with that on the recipient machine. Note that the size recorded by syslog in the mail logs considers the size of the body and the headers together. We want syslog to log its entries without fsync()ing them; in fact, we may want to not run it at all.

This same behavior could also be accomplished without the need to explicitly list each recipient by combining the aliases file with the virtusertable:

 FEATURE('virtusertable') 

We enable the virtusertable by including the previous line in the .mc file, and an entry like the following is added to /etc/mail/virtusertable for each domain in the local-host-names file:

 @example.com        devnull  @other.domain       devnull 

Now all messages for any user in the listed domains will be readdressed to the user devnull. Add the following line to the aliases file:

 devnull: /dev/null 

We no longer need to list every possible user name to which email might be sent. In fact, the source machines could even generate user names randomly.

In most cases, when sendmail writes email to a file it will obtain a lock on the file first, thereby avoiding the possibility of message corruption. However, if sendmail detects that the destination file is specifically /dev/null, not only will it not lock the file, but it won't write the message to the file. Instead, it will just acknowledge that the message was delivered successfully.

The configuration listed in the previous section is already set up to process email as efficiently as possible without regard to data integrity, which is what we want. If we can use neither sendmail 8.12 with its fast queueing nor a memory-based filesystem as a queue, it would be possible to use the DISCARD feature of the access database to reject the message even before it is queued.

The access database can be supported by adding the following line to the .mc file:

 FEATURE('access_db') 

Then, in the /etc/mail/access file, we can add an entry like the following:

 Connect:target.example.com          DISCARD 

To have it take effect, we create database files by running makemap hash /etc/ mail/access</etc/mail/access.

Any message coming from target.example.com will now be silently discarded.

Note that the DISCARD function of the access database is not as efficient as one might expect. Consider the CPU-bound test server that has been used in examples throughout this book; it runs sendmail 8.12.2 using interactive delivery mode and interactive queueing. It is capable of receiving and discarding around 600 messages/minute using the method combining the virtusertable with an alias pointing to /dev/null. When the system is reconfigured to use the DISCARD function of the access database, throughput drops to around 270 messages/minute. Of course, if we weren't using version 8.12 we would expect a less pronounced difference between these two methods, as using the virtusertable would require queue writes. Even so, on this test server the number of messages that can be discarded per minute using access is on the order of the number of messages per minute that can be delivered to mailboxes, which points out some significant inefficiencies in the discard mailer. At the present time, it would generally be better to use version 8.12, interactive mode, and /dev/null to receive and discard messages.

8.2.3 postfix Tools

Other tools besides sendmail can be used for SMTP sources and sinks. One set consists of the tools that come with the postfix distribution in the src/smtpst-one directory. These tools, which are called smtp-sink and smtp-source, are used just how their names would imply. Each program contains rough documentation, but these utilities aren't that difficult to figure out how to use. They make good substitutes for sendmail in the configurations listed previously. The shell scripts documented earlier could readily be modified to use these tools instead of sendmail.

8.2.4 popclient

Besides SMTP testing, an organization may want to do performance testing of POP access. To do so efficiently, one needs a tool that will perform a POP session in a scripted environment. One package that can accomplish this goal is popclient.

The popclient utility was written by Carl Harris, Jr., of Virginia Tech University, and has been floating around the Internet unmodified, as far as I can tell, since August 1995. The operating system OpenBSD makes popclient available as an installable package, but no central authoritative FTP or Web site for this software exists at the current time. Nonetheless, it's available via HTTP or anonymous FTP access at several software repositories. Most of the people who once used it to download their email have switched to Eric Raymond's improved fetchmail package; unfortunately, however, fetchmail is not well suited for use in email performance testing.

The command syntax for use in a test environment is simple. Retrieving and discarding all email for a given account on a target server can be accomplished as follows:

 #!/bin/sh  popclient -s -u username -p SeCrEt -o /dev/null \          popserver.example.org  logger -p "local3.debug"-t "popclient" \           "Exited with status: $?" 

In this example, we begin with the -s flag to suppress the status information ususally printed to STDERR. We then issue the user name and password on the command line using the -u and -p flags, respectively. Of course, we wouldn't issue a password on the command line in a real-world situation, but in a test environment with fake accounts and fake email this choice is a perfectly acceptable optimization. We list the server name as popserver.example.org; -o indicates that the messages popclient downloads should be written to /dev/null so we don't perform local disk I/O. Information on the number of messages and quantity of email transferred in bytes can be recovered from the POP daemon's log files on the target machine. A great deal of information is available via popclient's exit codes, which prove especially valuable when using the application in an automated mode. The codes and their meaning are well documented in the popclient manual page. We use the logger program to log this useful information to syslog along with the tag (-t) "popclient" using the facility local3 and the priority debug. If logger isn't already available on the operating system used for load testing, it can be written readily or the exit status of the popclient can just be appended to a file.

Other command-line flags that may prove useful in scripting include the -k flag, which causes popclient not to issue instructions to delete the read messages on the POP server, and the --protocol APOP flag, which allows connection using the APOP authentication mechanism. Many other options are available as well. Consult the manual page for more information.

As with the sendmail source and sink examples, we can populate mailboxes on the target machine for a wide number of users, keep a list of these users on the sink (client) machine, and make random selections from this list to exercise the server. The most common POP client behavior is to connect to the server, download all email, delete the messages from the server, and then disconnect. As popclient shares this default behavior, it makes the program useful for this testing. Again, a script should be written to invoke this process. To simulate a real-world load, the sink would run several of these scripts in parallel.

8.2.5 c-client

The variety of modes in which POP clients connect and download messages is really very restricted. IMAP, on the other hand, offers a great deal more freedom. Determining what a typical IMAP session looks like for a server is by no means an easy feat. Some IMAP daemons can be switched into a debugging mode, which will log the commands used by each session. Similarly, a utility such as tcpdump may be used to perform a packet capture of IMAP sessions. Either of these techniques may violate users' privacy, and neither should be undertaken without express permission.

Even armed with many sample sessions, there are probably as many IMAP profiles as there are IMAP clients multiplied by the number of IMAP users. This fact makes it extremely difficult to have any real confidence in performance numbers generated in an IMAP test environment. Nonetheless, some rough numbers are better than none, as long as their limitations are well understood.

Once one or more "typical" IMAP session profiles have been defined, the next step is to create an automated client that can simulate these usage patterns. No IMAP equivalent of popclient that can be used for scripting IMAP sessions exists. Instead, the best that can be hoped for is a toolkit that allows someone to write an application that mimics an IMAP client without having to write the entire client side of the IMAP protocol.

Fortunately, such a toolkit exists. Called c-client, it comes with the University of Washington IMAP server distribution. Its primary Internet repository is at ftp://ftp.cac.washington.edu/mail/imap.tar.Z. A C library, c-client.a, is created in the src/c-client directory when this distribution is unpacked and compiled. As the name suggests, it contains a set of library calls to perform IMAP functions that can be called from a C program that links with it. The documentation is sparse, but the file internal.txt in the docs directory and the source code itself provide enough information to allow a programmer to build something that will work. The often-used pine UNIX IMAP client links with the c-client library, so one can use its source code as an example. Even better examples are probably the utilities included in the associated imap-utils.tar.Z package found in the same directory of the previously mentioned FTP site. Despite the documentation and examples, creating an automated IMAP client will almost certainly represent a serious undertaking for even the most experienced email programmers.

8.2.6 Perl Modules

All of the tasks we sought to accomplish using the programs and scripts discussed earlier in this section can also be accomplished in other languages, including Perl, which is commonly used for writing test harnesses by system administrators. Certainly, a reader who is very familiar with Perl programming may already have thought about how to carry out some of the test tasks described earlier in this language, in some cases perhaps much more efficiently and thoroughly than the examples presented here.

The Perl language is supported by an online library of preexisting Perl software modules and documentation called CPAN (Comprehensive Perl Archive Network) [CPA]. The CPAN archive contains a number of modules that can assist those writing email test utilities in Perl, some of which will be discussed here. The contents of CPAN change often, so the information presented in this book inevitably gives an out-of-date snapshot of what is currently available. Anyone interested should browse the online repository to see whether updates or new modules are available.

Mail::Bulkmail

The Bulkmail module contains routines for sending email messages, including a Perl version of a stripped-down SMTP client. Its purpose is to efficiently send out large amounts of email, such as announcements. The module sends those messages to an SMTP server, which will relay them. For testing, the destination could easily be the target server. In essence, this code is an example of a simple SMTP-based injector, like those mentioned in Chapter 5.

The code in the module is less careful about timeouts than a complete SMTP client should be. For instance, it assumes that the server used to relay the messages is "well behaved" and will respond to queries reasonably quickly. This assumption shouldn't be a problem in most test environments. The module also doesn't handle all possible SMTP error codes that can be returned, but it does include cases for handling some of the most common ones. Again, this simplification will usually be acceptable in a test environment.

With all of these modules, the entire module itself may not be directly useful. Nevertheless, even if it's not, the module may contain subroutines that can be copied to save a programmer some extra effort.

Mail::POP3Client

As its name suggests, the POP3Client module contains code for accessing email servers via the POP3 protocol. Rather than being a complete POP client, it comprises a set of routines that can be used to build a POP session. Most MUAs that support the POP protocol tend to access email in the same sort of ways, but often some small differences crop up. If one type of POP client is commonly used in a given organization, then once the sequence and timing of commands for that client has been determined, they can be duplicated using the routines in this module without the need to write RFC 1939 compliant code.

As an example, suppose that the most widely used POP client in an organization performs the following default behavior: It authenticates with the server, asks for the UIDL of all messages, and retrieves the headers using the TOP command asking for the first 20 lines. Then, if new email has arrived, it prompts the user whether it should be retrieved. In taking a profile, we find that 85% of the time users answer "yes," and they wait an average of 45 seconds before making that response. The email is downloaded, the client sleeps for five minutes, and then it repeats the process. While this scenario may not be a very realistic or very well-designed POP process, it is easy to model, and an accurate simulation of such a client can be written using this module with little effort. It would take more work to try to get popclient to behave in this manner.

Mail::IMAPClient

The IMAPClient module contains a set of routines and data structures that may be helpful in creating an IMAP client. It will come as no surprise that this module is larger and has more features than the POPClient module. Nonetheless, once a profile has been created of an IMAP session that one wishes to automate, it will almost certainly take much less work to code it using this module than to write all of the client-side code from scratch.

At the time of this writing, this module seems to be fairly well documented within the code itself, and its implementation of the IMAP protocol is very thorough. Of course, turning even a well-considered IMAP profile into an automated client will not be easy. A lot of coding will be required, but using the IMAPClient module should simplify the process.

8.2.7 Mstone

While developing their Messaging Server product, Netscape programmers also developed a set of testing tools called Mailstone. They kindly provided these tools to their customers for the purpose of testing the installation of their new email systems before it was placed in production. When Netscape chose to release its browser technology under an Open Source license, the company also elected to release its Mailstone test suite under the same license, albeit under a slightly different name: Mstone. This test suite can be of great assistance to anyone who wants to load-test an email server.

Obtaining the Mstone package can be a bit tricky, as a tar bundle isn't available via FTP or HTTP on the Internet. Instead, one must use anonymous Concurrent Versions System (CVS) to download the package. Instructions on how to contact the Mozilla anonymous CVS server are available at the following URL: http://mozilla.org/cvs.html. Information on which modules to download is available at http://mozilla.org/projects/mstone/. Anyone with an Internet connection and a working CVS client should be able to download the package from the information provided in these two Web pages.

Building Mstone should be straightforward on any supported platform for Netscape Messaging Server version 4.15, including Solaris, Linux, and several others. Before building, the installer must have the GNU version of the make utility and a version of Perl at least as recent as version 5.005. The gnuplot and gd graphics utilities may also prove beneficial, although they are not strictly required. The system is well documented, although some of the URL links mentioned in the documentation are no longer valid, and some information hasn't been rewritten since the package underwent its name change and became widely distributed. A careful reading, especially of the doc/MailStone.html file, will still provide the information needed to create profiles and run SMTP, POP, and IMAP tests against a target machine.

8.2.8 Testing Tool Summary

None of the tools mentioned here represents the only, or even best, method for generating a load so as to measure the capability of an email server in a test environment. However, this section does cover a reasonable set of tools that can provide a useful starting point for further exploration. While one can easily write more sophisticated and efficient tools than those that can be obtained by cobbling together the components mentioned in this section, the items discussed here should at least provide some reasonable examples from which to draw, even if they do not give a complete load-testing solution.

One final note: Before venturing too far down the road of modifying or rewriting test tools to more specifically meet an organization's needs, remember the adage that "time is money." Time spent programming costs money just as surely as buying hardware does, and it's important that the money be spent most efficiently. It's often cheaper to buy more hardware to run an inefficient test suite than to spend a great deal of valuable time writing complex software.