Section 6.4. Test Environments


6.4. Test Environments

This section examines some examples of different test environments that are currently available.

6.4.1. Shell Scripts and Batch Files

The simplest ad hoc test environments are just collections of a number of different test commands. Unix shell scripts and Windows batch files provide one way to collect these commands together. Scripting languages (such as Perl, Python, and Tcl) can also be used. Failure of a test written for these environments can often be detected by the value of each program's return code, with nonzero generally indicating a problem. Output is usually simple: the test name and pass or fail, with the error code. Single tests can be run by simply invoking them with the same arguments that the shell script uses.

Some aspects of testing that become more difficult if you use shell scripts and batch files are:

  • Checking for deadlocks when a test is not ending properly. This can be done by using a separate watchdog process to provide timeouts for tests, but a program can do this more efficiently by using threads.

  • Coordinating which regular expression patterns to look for in the output from each particular test. With a more general language, this information can be stored in the same place as each test definition.

  • Gathering statistics about different parts of each testfor example, startup times and response times. A program can offer a better structure for running different parts of tests separately.

  • Generating reports in HTML. This is just plain tedious using shell scripts.

For fewer than perhaps a hundred straightforward unit tests, this approach at least gets you started, and has very little overhead. However, the overall awkwardness of shell scripts explains why they are rarely used for complex products. The major drawback of this approach is that adding features to the test environment gets progressively harder as the shell script becomes more complex, which in turn discourages you from writing more tests or making changes to the test environment.

As a practical example of some of these issues, CVS (see Section 4.6.1) is released with a 34,000-line shell script named sanity.sh for running a number of tests on different platforms. The file TESTING, which is also part of the source code distribution for CVS, discusses various alternatives to sanity.sh, precisely because it is hard to understand such a monolithic shell script.

6.4.2. xUnit

The first of the series of xUnit test environments to become popular was JUnit (http://junit.sourceforge.net and http://junit.org), originally written by Erich Gamma and Kent Beck, which is a well-known open source test environment for Java classes. JUnit is now integrated into the Java build tool Ant and also into most IDEs for Java developers.

The xUnit test environments all use the same basic architecture as JUnit, but each one is intended for implementing tests for different languages. Each test inherits from a TestCase class and implements setUp and tearDown methods to prepare a class for testing and to clean up after testing, respectively. The author of the test class then defines methods whose names start with "test" to perform the tests, and uses a variety of assert methods to determine the result of each test. Generally, each test class should test a small part of the application (the "unit" part of xUnit). Multiple tests can be combined into TestSuite classes. You can also use a TestResult parameter within the tests for the collection of runtime statistics.

There is a long list of all the available xUnit test environments at http://c2.com/cgi/wiki?TestingFramework, and there is an even longer list at http://www.xprogramming.com/software.htm. The book Unit Test Frameworks, by Paul Hamill (O'Reilly), provides a good overview of the test frameworks for Java, C++, .NET, Python, and generated XML documents. The Pragmatic Unit Testing series, by Andy Hunt and Dave Thomas (Pragmatic Bookshelf), covers JUnit and NUnit in the authors' classic back-to-basics style.

One large advantage of the xUnit test environments is that since the architecture and classes are similar from one language to another, it becomes easier to write tests in different languages and easier to understand other people's tests. There are a number of command-line and GUI tools for running xUnit-based tests and collecting their results. However, the tests that you write are closely tied to the xUnit environment; it's harder to run such tests directly from the command line or by using a different test environment. Writing your tests in the same language that your project is using is a popular way to make developing tests easier.

6.4.3. DejaGnu

DejaGnu (http://www.gnu.org/software/dejagnu) is an open source test environment written in Expect that uses Tcl as the language for defining tests. Like the bug tracking system GNATS (see Section 7.2.4), DejaGnu was originally developed at Cygnus. It's now maintained by Rob Savoye and Ben Elliston, and it is the test environment used by gcc, gdb, and some other GNU projects.

Tcl and Expect

Tcl (http://tcl.sourceforge.net) is a scripting language commonly used by testing teams, though you can also use it to build larger applications. The graphical interface for BitKeeper (see Section 4.6.5) is one such application.

Expect (http://expect.nist.gov) is an open source tool written in Tcl that helps you automate interactive tasks such as FTP sessions, remote logins, and complex system tests.


DejaGnu's main focus is on running system tests, which are usually defined in Tcl, but you can also run existing tests in the same environment. There is strong support for running remote tests on many different platforms (though to run it on Windows, you must have the Cygwin environment) and there is good integration with the GNU Autotools suite (see Section 5.5.3). Running the tests automatically generates both a summary of the results and details of the tests' output. One benefit of using Expect is that DejaGnu can be used to test interactive applications such as gdb, the GNU debugger, which uses an interactive CLI by default. DejaGnu is also one of only a few available test environments that are POSIX-compliant.

The manual for DejaGnu is well written and contains an excellent step-by-step guide to getting started. Test debugging is mostly done by increasing the verbosity of various logfiles, but there is also more information on debugging remote tests available at http://kegel.com/crosstool/current/doc/dejagnu-remote-howto.html.

If you need to test your product on many platforms (basically anything that gcc runs on) or if you want a lot of text-based user interaction as part of your testing, then you should consider DejaGnu for your testing environment.

6.4.4. SilkTest

SilkTest is a commercial test environment from Segue Software (http://www.segue.com). First released in 1993 as QA Partner, the latest release of SilkTest was Version 7.1 in 2004. Pricing starts above $6,500 per user for the test design tool; the runtime part costs less.

SilkTest is one of the industry-standard test environments (the other one being WinRunner) that is commonly used for testing everything from GUIs to browsers to databases to network management applications. The design tool runs only on Windows platforms and allows you to record and play back test actions, and to extend them with different data. SilkTest has its own test-scripting language named 4Test, which has support for multiple platforms and many existing applications. There is support for testing Java applications on Unix platforms, and the separate runtime part of SilkTest can be used on multiple platforms in parallel.

SilkTest is highly customizable; hence it has a fairly steep learning curve. The quality of support from Segue seems to have gone through cycles, but is sometimes excellent. Alternative sources of information include the QA forums at BetaSoft (http://www.betasoft.com).




Practical Development Environments
Practical Development Environments
ISBN: 0596007965
EAN: 2147483647
Year: 2004
Pages: 150

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net