8.1 Test System Setup

Before an email server or server configuration change can be tested, one must have a system to test. If new equipment has been purchased, then it can be assembled and tested. If, however, one wants to determine whether a given configuration change on an operating server is more or less likely to improve the performance of that server, then the change has to be made either on the real server or on another machine. The choice of whether an organization elects to purchase a test server, and how closely the test server will resemble the production server, comes down to economics. How does the cost of downtime for testing and iteration compare to the cost of purchasing extra hardware and potentially extra software licenses? Each site must evaluate this issue, but the final decision is best made explicitly by people who have both the authority to decide on the best course of action and the information necessary to properly evaluate the risks and rewards of all possibilities. This decision is often, if not usually, out of the hands of the people responsible for performing the upgrades. Nevertheless, the system administration staff have the responsibility to make sure that the decision makers are armed with all knowledge necessary to make an informed choice.

The decision on whether to set up a test environment is rarely as black and white as having nothing or replicating every piece of gear. Invariably, some compromise becomes necessary. For example, although an organization may choose not to purchase a complete test system that duplicates an expensive centralized IMAP server, that doesn't mean that the only alternative is to have nothing on hand. A less expensive, lower-powered machine can provide valuable information when used as a test platform. For example, if the production IMAP server is a Sun Enterprise 4500 server that is in danger of becoming memory bound, testing a recompilation of the IMAP daemon on a Sun Ultra 5 to see whether its memory footprint is smaller will yield results that could be expected to apply to the larger server. However, extrapolated test results are not always guaranteed to be valid.

In fact, results from any test environment are never guaranteed to be applicable in actual practice. Every test environment is merely an approximation of the real environment. The more effort expended to make sure the test environment resembles the real environment, the more likely the test results will resemble the behavior of the real system. Of course, no amount of effort expended on a test system can translate into absolute certainty about real-world behavior. Each organization needs to decide how much money it should spend on equipment and how much effort it should expend on configuring the system to mimic the production environment. The organization's resources, tolerance for risk, and applicability of the results of any particular test will all determine the appropriate amount of effort. This book cannot provide enough information to thoroughly evaluate the first two conditions. Some rough guidelines for determining how much confidence should be placed in certain test scenarios appear later in this chapter.

In an email test environment, we distinguish between three categories of servers involved in testing: source, sink, and target machines. The target is the server sometimes a set of servers to be tested. As stated earlier, the more closely this machine's configuration matches the production system, the more reliable the results of the test will be. Source machines, also known as load generators, provide data in this case, usually email messages to be processed by the target. We might want to test many situations, such as email relaying or POP3 server performance testing where the target machine will generate traffic that it will send to some other system. Machines that accept data from the target are called sink machines. Some tests will not require sources and some will not require sinks, but some tests will require both types of servers. All tests will, of course, require a target.

Each of the machines involved in a particular type of test must be able to communicate with the others, which means they're linked together via a network. This network should typically have the same properties as the network of the production server that the target is emulating. For example, if the production server is a well-tuned Intel-based server running FreeBSD on a 100 Mbps network, and if the target is an identically configured machine, then the test network should also be 100 Mbps. If the production network is switched, the test network should be switched, preferably using the identical networking equipment, including cabling.

If, however, the production server is a Sun E4500 connected to the outside world using Gigabit Ethernet and the test server is a Sun Ultra 5, it's probably not productive to run Gigabit Ethernet in the test environment. We do not expect the Ultra 5 to be able to handle the same throughput as the E4500, so going to the extra expense of matching networking technologies when such a great disparity in the performance capabilities between the production and test servers exists is usually not productive.

Remember, though, that production email servers with performance problems are usually I/O bound rather than CPU bound. An E4500 acting as an email gateway with a single SCSI disk connected to it as its queue may have a very similar throughput to an Ultra 10 with the same disk attached. If a bottleneck exists on a production server, it's usually safe to provide a test network sufficient to handle the same ratio of networking traffic, if the test network cannot support the same absolute throughput. For example, if I/O controller throughput acts as the bottleneck, and if the test target has a slower controller than the production server, then it's usually safe to provide a test network sufficient to handle the same ratio of networking traffic.

For example, a production server connected to the Internet via Gigabit Ethernet might be I/O controller bound on an 80 Mbps SCSI controller. At peak times, the production server may exchange 160 Mbps of traffic with the outside world. If a test server has only a 20 Mbps I/O controller, then a switched 100 Mbps test network might suffice to handle the projected 40 Mbps of network load during a scaled-down test. Of course, if the test server isn't I/O controller bound, then all bets are off. If this is the case, our test is unlikely to produce valid results no matter how the network is configured.

If the test network is connected without restriction to the general Internet, a simple configuration error could cause test email to interact with the rest of the world, which could prove disastrous. While it undoubtedly would be too restrictive to suggest that the test network be "air gapped" (that is, cut off completely) from the rest of the Internet, some precautions should be taken to ensure that any tests that run amok won't spew bogus email to the general Internet. One solution is to configure a packet-filtering router to block all commonly used email ports: SMTP (25/TCP), POP3 (110/TCP), IMAP (143/TCP), and message submission (587/TCP). This router could then provide connectivity to the test network. Alternatively, a dual-homed bastion host that doesn't route packets could be set up to screen the rest of the world from the test network. Any host with two network interfaces could serve this purpose as long as no routing daemon runs on the host (such as routed or gated), and the kernel is configured to not forward IP packets between interfaces.

To add another layer of protection against flooding the Internet with test email, one can use a block of private IP addresses, as discussed in RFC 1918 [RMK⁺96]. In this way, if a host on the test network tries to make a connection with external hosts, it will be prohibited from doing so because its IP address won't be routable across the Internet (or at least it shouldn't be). Some organizations use private IP addresses for many machines within their organization and network address translation (NAT) servers to proxy connections from these hosts to Internet servers. In such a case, the test network should use a different block of private IP addresses than the computers that are allowed to make these connections, and the NAT gateway should be explicitly set up not to translate the test network addresses. In many cases, using RFC 1918 private addresses is not a good idea; this instance is a case where using them most definitely is a good idea.

Another service that needs to be set up within the test network is DNS. As discussed earlier in this book, very good reasons support running a high-volume email server with a caching-only domain name service daemon. To replicate the production system as closely as possible, one can run a similarly configured name server on the test network target machine. If the test network uses private IP addresses, then the test machine will not be able to contact the root name servers. As a consequence, it won't be able to identify any of the machines attempting to connect to it, because a caching-only name server contains no information about any DNS zones. Further, an externally accessible DNS server may not be configured with accurate information about the machines on the test network.

There are two good ways to solve this problem, either of which may be the best option depending on the circumstances. First, instead of setting up the name server on the target machine to be a caching-only server, set it up to be a caching-forwarder pointing to a name server that knows about all domains used in the test network. If a bastion host is used to screen the test network from the rest of the Internet, it represents a good candidate to run this name server if it isn't practical to set up a machine in the test network for these purposes. The test network name server may or may not be able to query the root name servers. If it cannot, then the second solution to this problem is to create a fake root name server on the test network that contains the domains to be used. Again, it could consist of a machine on the test network dedicated to this task, or it might be combined with the bastion host if one is available. Whichever solution is chosen, I'd recommend not running either the forwarder or fake root name server on a machine used as a source, sink, or target host. Having one of these hosts perform this function might adversely affect the repeatability of certain tests.

Before running any test, make sure that DNS is working properly. Situations in which the target machine (or source or sink machines) delay connections waiting for DNS timeouts to occur will seriously skew test results. If the bastion host separating the test network from the outside world runs a name service daemon, this host does not have to have its resolver (typically via the /etc/resolv.conf file) point to itself. It can query another name server for its DNS information, which is important if it is running as a fake root name server.

Finally, having an inexpensive computer sitting on the test network that isn't directly involved in the testing can prove very valuable. This machine can be used for network monitoring, storing log files, processing data, and as a long-term repository for information in general. On test networks, the targets, sources, and sinks tend to get wiped out and rebuilt from scratch fairly frequently, so it's nice to have a nearby repository where one can put files and expect to find them again next month.