Executable Tests

Writing executable tests requires the tests to be expressed in a form that can be interpreted by a machine. They don't necessarily have to be written in a programming language, but they do have to conform to a set of syntax rules and identify the information necessary to perform and evaluate the test. Usually a programming or scripting language works best, because it will provide ways to write tests at an understandable level while tucking the many details away in lower levels.

When picking a language in which to write the tests, look for the following:

• It's readily available on the platform(s) where your tests will run.

• You can hook it up to a user interface test driver, if applicable.

• An implementation of the xUnit framework exists in the language.

• You're reasonably fluent in it

• The system is being developed in the same language

We'll be using Java for our illustrations and examples, because it's available on all platforms, it's a common language used to develop the systems we test, it can be interfaced to just about any tool, and a Java implementation of the XUnit family of test frameworks is available at www.JUnit.org (JUnit for Java). If you aren't fluent in Java, don't panic. It's an easy language to use.

It's best to illustrate executable tests with an example. Take, for instance, a login story. We might have identified high-level acceptance tests for this as follows:

Attempts to login with a valid id and password succeed; attempts with invalid ids and/or passwords fail.

This captures the essence of the test but leaves out the details. The details can go into an executable test, as follows:

 assertTrue( login("bob","bobspassword") )  assertTrue( login("BOB","bobspassword") ) assertFalse( login("bob","") ) assertFalse( login("","bobspassword") ) assertFalse( login("bob","wrong") ) assertFalse( login("bob","Bobspassword") ) 

Understanding this test is straightforward: assertTrue() verifies that the operand (whatever is between the parentheses) has a true value, and if not, fails the test. assertFalse() works in an analogous way, requiring its operand to be false. login() attempts to log into the system using the specified id and password, returning a value of true when the login succeeds and false when it fails.

The test includes two cases of logging in with a valid id and password and four cases of logging in with an invalid id or password. It should be clear from these cases that the id is not case sensitive, since both bob and BOB are expected to work, but the password is case sensitive, since bobspassword should work and Bobspassword should not.

This shows how the details of the login story can be put into the test itself. As we discover additional details, we can include them by adding more cases. The rest of the team, with the possible exception of the customer, will have no trouble understanding the test and using it as a reference when coding. You may need to spend some extra time with the customer to accustom him to this format. In the worst case, you can provide a less geeky version for the customer, which we'll illustrate in the next chapter.

If you know the cases you want to test when you sit down to write, which at this point you should, then writing this test takes about five minutes. For the sake of clarity, we've omitted several lines of Java directives and declarations that are essentially the same for every test, but even if you typed those in each time (instead of copying and pasting, which is how we do it), it would still take just a few minutes to write.

To run this test, you need assertTrue(), assertFalse(), and login(). You could develop your own assertTrue() and assertFalse(), but that isn't necessary, because they've already been developed (by Kent Beck and Erich Gamma) and are available free at www.jUnit.org. login(), on the other hand, will have to come from your project in one form or another.

The simplest thing, always a good place to start, is for login() to be a piece of the system. Then this test is runnable at soon as the login() piece (and everything it relies on) is programmed. You simply put it all together into one big program (your tests, login(), and anything login() requires) and run it. Of course, that works only if the system is being developed in the same language as the test, which is one reason why this is a consideration when choosing the test language.

When the tests and system development languages are incompatible, someone on the team will need to write a login()bridge in the test language that passes the operands from the test to the system's login() and ferries the return values back.

In fact, you'll often need to use something similar, even when the system development and test language are the same, because the actual pieces of the system are likely to have different interfaces from the ones one you want in the test. You'll need some kind of intermediate module to convert from one to the other.

In addition, when you want to run the tests through the user interface, as opposed to calling pieces of the system directly, you'll need a bridge to carry operands and results between your test and whatever tool you're using to drive the user interface.

Let's look at another example, the searching story for our directory application. One of the high-level acceptance tests we defined was this:

A search with a category/location combination for which some businesses exist returns a list of those businesses; a search for a combination without businesses returns an empty list.

Here's a first cut at the executable test:

 assertTrue( search("woodworking","dublin ohio") )  assertTrue( search("horses","denver colorado") ) assertFalse( search("joy","mudville")) assertFalse( search("U","team") ) 

Like login() in the preceding example, search() performs a search with the specified category and location. It returns true if the search returned a list of businesses and false if it returned an empty list. The test contains two cases of searches that should return lists and two cases that should return empty lists.

Now suppose one of the details we uncover is that when the location is a major U.S. city, the state can be omitted. We could add a case:

 assertTrue( search("horses","denver") )  

But this doesn't really express what we want. It just says that a search with denver for the location will return a list of some businesses. We want it to test that it returns the same list as the search with denver colorado.

To write the test we want to write, we need to refactor our search() so it returns, instead of simply true or false, the list of businesses that match the location and category combination. Then we can write the test as follows:

 assertEquals( search("horses","denver colorado") ,                search("horses","denver")          ) 

The assertEquals() verifies that its two operands are equal, and if not, fails the test. As with assertTrue() and assertFalse(), this is a module you get with the Beck/Gamma jUnit framework.

This example illustrates how the interfaces to search() and login() can be changed (refactored) as you learn more details about the desired system behavior. This is a further reason to use an intermediate module between your tests and pieces of the system, because even if the interfaces are the same at the outset, they probably won't stay that way as the tests and system are refactored.

Writing executable acceptance tests in this manner and at this point in the iteration will put you in an excellent position to begin running tests as soon as the requisite pieces of the system are programmed. Some additional work will probably be required to bridge between the tests and the system before the tests are actually runnable, but this can usually be completed quickly once the pieces of the system are in place.

Writing the tests in this manner greatly increases the probability they can be automated. It doesn't necessarily preclude your executing them manually, but we'll have more to say about that in Chapters 19 and 20.

This has been a high-level introduction to writing the acceptance tests. In the next several chapters, we'll take a closer look at some of the details and problems that commonly come up in when you actually try to do it.