Automated Testing

According to Robert Binder, author of the definitive testing guide Testing Object-Oriented Systems, for tests to be effective and repeatable, they must be automated [Binder, 1999]. This is an added bonus: by adopting test automation techniques, you not only help to reduce the project timeframe but also apply best-practice testing methods.

The arguments for writing automated tests have close parallels with the arguments used to justify the creation of code generators during development:

Code generators are introduced in Chapter 6.

Accuracy and repeatability.

Test scripts are usually executed more than once during the course of a project. Even the most careful developers fail to prevent every defect from finding a way into the system. Although a single test cycle must be the goal, several test cycles are often required.

Accurate system testing relies on tests being repeatable between cycles. Repeatability is easier to achieve with test automation. By comparison, manual testing is labor-intensive and subject to both human error and oversight. A test strategy based completely on manual testing procedures carries the risk that errors introduced between testing cycles will escape detection. An automated test that is 100 percent repeatable avoids this danger.

Reduced timeframes.

The time available to fully test a complex enterprise system, with numerous interoperating subsystems and countless points of integration between collaborating components, may be such that test automation is the only plausible option for realizing an effective test strategy in a timeframe that is acceptable to the customer.

Improved test effectiveness.

Certain types of testing are difficult to achieve without the use of automated tools. Load and stress testing fall firmly within this category. Here, sophisticated tools are required that reproduce the same level of throughput upon a system as a large number of users all accessing the system concurrently. Unless the project's budget runs to hiring potentially thousands of testers, then the use of these tools is a necessity.

Removal of mundane tasks.

Testers, like developers, get bored if they are required to perform the same type of tests repeatedly. As with the use of code generators, automated test scripts remove the drudgery from the testing effort, leaving the QA specialist free to focus on other aspects of the system.

Note

The adoption of test automation does not completely remove the need for manual testing processes. Expert testers can, and should, continue to perform invasive manual tests in an effort to break the system. An effective test strategy is a combination of automated and manual tests that together comprehensively test a system prior to its delivery to the customer.

A question that often arises in relation to test automation is whether the time taken to write the automated test script is justifiable in terms of the effort saved in running an automated test. Despite the benefits, test automation does consume effort and hence incurs a cost to the project. Nevertheless, these costs are usually recoverable within two or three test cycles, especially when including the added benefits of improved test accuracy and consequent increase in software quality. Furthermore, the same test scripts are reusable in the long term, as the system enters a maintenance phase.

Tip

The creation of test scripts can start in the early stages of a project iteration by using a prototype as a starting point for the creation of automated test scripts.

The J2EE Testing Challenge

An enterprise-level distributed J2EE application presents some very unique challenges to the tester, and regardless of whether manual or automated tests are used, the testing of a distributed application is an involved and complex process.

First, an application built on the J2EE platform employs a variety of Java-based technologies for the development of system components, which are distributed over remotely distinct tiers. Thus, the tester immediately has a multimachine environment to contend with, plus the headaches a network brings in the form of firewalls and corporate security policies.

Chapter 4 examines some possible software architectures for creating J2EE applications that do not depend upon the remote method invocation services of Enterprise JavaBeans.

In addition to the distributed environment, by leveraging the full capabilities of the J2EE platform, the architecture of a J2EE enterprise application could potentially involve the use of asynchronous messaging between systems, long-running business transactions being fired between heterogeneous systems, and critical business functionality being exposed to other systems as part of a wider service-oriented architecture.

Besides the sophisticated architectures made possible by the J2EE platform, enterprise systems present their own special challenges to the tester, with system operational attributes such as security, performance, scalability, and robustness requiring intensive test coverage. The sensitivities revolving around security alone often demand the skills of specialized QA experts.

Collectively, these points make the task of testing an enterprise system as complex for the tester as the task of development is for the software engineer.

Test Automation Tools

Fortunately, test automation tools are available to help address the challenges presented in thoroughly testing J2EE solutions. These tools aim to both simplify the testing process and improve the quality of the tests conducted.

Tools that support the testing process fall into the following general categories:

Test coverage.

These tools analyze the entire code base and report on the depth and breadth of existing test cases.

Quality measurement.

Measurement tools operate either statically, analyzing such artifacts as the design model and the source code, or dynamically by inspecting the runtime state of the system under test. The output from these types of products enables metrics to be compiled on the application in terms of complexity, maintainability, performance, and memory usage. This information can then be used to drive the production of a suitable test strategy.

Test data generators.

One of the harder aspects of testing is the generation of suitable test data. This is especially the case where a new system is under development and no existing legacy data is available. Test data generators will produce test data from such artifacts as the design model, database schema, XML schemas, and the source code.

Test automation tools.

This general-purpose category covers the tools that execute prepared automated test scripts. Products are available that support the automation of the full range of testing types, including unit, integration, functional, load, and stress tests. Methods employed by these tools include record and playback of events for GUI testing and programmatic testing for specific GUI types.

Various testing tools are available as open source, although few offer the same level of functionality as the high-end commercial products. Table 15-1 lists some of the open source testing tools available to the Java enterprise tester for undertaking test automation.

Table 15-1. Open Source Java Testing Tools

Name	Description	Type	Reference
Cactus	Cactus is a unit/integration test framework from the Apache Software Foundation. It is used for the testing of server-side Java code and offers a means of undertaking in-container testing on J2EE components.	Unit/Integration	http://jakarta.apache.org/cactus/
Grinder	The Grinder orchestrates the activities of a test script in many processes and across many machines, using a graphical console application. Test scripts use client code embodied in Java plug-ins. The Grinder comes with a plug-in for testing HTTP services, as well as a tool that allows HTTP scripts to be automatically recorded.	Load/stress	http://grinder.sourceforge.net/
HttpUnit	HttpUnit offers a Java API for developing a suite of functional tests for Web applications.	Unit/Functional	http://sourceforge.net/projects/httpunit/
Jameleon	Jameleon is an acceptance-level automated testing tool. Jameleon claims to separate applications into features, which are then tied together as test cases.	Functional	http://jameleon.sourceforge.net/
JFCUnit	An extension to the JUnit framework that enables you to execute unit tests against code that presents a Swing GUIbased interface. JFCUnit offers a record and playback facility to enable novice GUI developers to generate and execute tests.	Unit/Functional	http://sourceforge.net/projects/jfcunit/
JMeter	Apache JMeter is a 100 percent pure Java desktop application designed to load-test functional behavior.	Load/stress	http://jakarta.apache.org/jmeter/
Solex	Solex is an Eclipse plug-in for testing Web applications. It provides functions to record client sessions, adjust parameters as required, and then replay later as part of a regression suite.	Functional	http://solex.sourceforge.net/

The next sections cover the use and application of two of these tools: HttpUnit and JMeter. We first look at HttpUnit, an API that provides support for functional testing.