Software Test Automation | Lessons Learned in Software Testing

Although test automation is just another class of software testing tools, it's one that deserves special consideration. The software test tools that you've learned about so far are indeed effective, but they still must be operated or monitored manually. What if those tools could be combined, started, and run with little or no intervention from you? They could run your test cases, look for bugs, analyze what they see, and log the results. That's software test automation.

The next few sections of this chapter will walk you through the different types of automation, progressing from the simplest to the most complex.

Macro Recording and Playback

The most basic type of test automation is recording your keyboard and mouse actions as you run your tests for the first time and then playing them back when you need to run them again. If the software you're testing is for Windows or the Mac, recording and playing back macros is a fairly easy process. On the Mac you can use QuicKeys; on Windows the shareware program Macro Magic is a good choice. Many macro record and playback programs are available, so you might want to scan your favorite shareware supplier and find one that best fits your needs.

Macro recorders and players are a type of driver tool. As mentioned earlier, drivers are tools used to control and operate the software being tested. With a macro program you're doing just thatthe macros you record are played back, repeating the actions that you performed to test the software.

Figure 15.7 shows a screen from the Macro Magic Setup Wizard, which walks you through the steps necessary to configure and capture your macros.

Figure 15.7. The Macro Magic Setup Wizard allows you to configure how your recorded macros are triggered and played back. (Figure courtesy of Iolo Technologies, www.iolo.com.).

The Macro Magic Setup Wizard allows you to set the following options for your macros:

Name. Giving the macro a name provides a way to identify it later. Even for a small software project you could write hundreds of macros.
Repetitions. Repetition testing is a great way to find bugs. You can set the number of times the macro will repeat or loop when it runs.
Triggers. You can set how the macro is started. This can be by a hot key (for example, Ctrl+Shift+T), by a set of typed-in characters (maybe run macro 1), by clicking a shortcut, when a certain window is displayed (whenever Calculator is started, for example), or when the system has idled unused for a certain length of time.
What's captured. You can select to capture (record) just keystrokes or both keystrokes and mouse actions such as moving and clicking.
Playback speed. The macro can play back from up to 20% slower to 500% faster than how you originally recorded it. This is important if your software's performance can vary. What would happen if the software you're testing became a bit slower and the button the macro was to click on wasn't yet onscreen?
Playback position. This option determines if the mouse movements and clicks should be absolute or relative to a certain window onscreen. If you're testing an application that might change onscreen positions, making your movements relative to that application is a good idea; otherwise, the mouse may not click where you would expect.

Now's a good time to experiment with recording and playing back macros. Find and download some macro software, try it out on a few simple programs such as Calculator or Notepad, and see what you think. Think like a tester! What you'll find is that although macros can do some automated testing for you, making it much easier and faster to rerun your tests, they're not perfect. The biggest problem is lack of verification. The macros can't check that the software does what it's supposed to do. The macro could type 10099 into the Calculator, but it can't test that the result is 1you still need to do that. This is an issue, sure, but many testers are happy just eliminating all the repetitive typing and mouse moving. It's a much easier job to simply watch the macros run and confirm that the results are what's expected.

Playback speed can be another difficulty with macros. Even if you can adjust the speed of playback, it may not always be enough to keep the macros in sync. A web page may take 1 second or 10 seconds to load. You could slow down your macros to account for the expected worst case, but then they'd run slowly even if the software was running fast. And, if the web page unexpectedly took 15 seconds to load, your macros would still get confusedclicking the wrong things at the wrong time.

NOTE

Be careful if you use a macro recorder to capture mouse movements and clicks. Programs don't always start up or appear in the same place onscreen. Setting the playback position to be relative to the program's window rather than absolute to the screen can help, but even then just a slight change in the GUI could throw off your captured steps.

Despite these limitations, recording and playing back macros is a popular means to automate simple testing tasks. It's also a good place to start for testers learning how to automate their testing.

Programmed Macros

Programmed macros are a step up in evolution from the simple record and playback variety. Rather than create programmed macros by recording your actions as you run the test for the first time, create them by programming simple instructions for the playback system to follow. A very simple macro program might look like the one in Listing 15.1 (created with the Macro Magic Setup Wizard). This type of macro can be programmed by selecting individual actions from a menu of choicesyou don't even need to type in the commands.

Listing 15.1. A Simple Macro That Performs a Test on the Windows Calculator

 1: Calculator Test #2 2: <<EXECUTE:C:\WINDOWS\SYSTEM32\Calc.exe~~~~>> 3: <<LOOKFOR:Calculator~~SECS:5~~>> 4: 123-100= 5: <<PROMPT:The answer should be 23>> 6: <<CLOSE:Calculator>>

Line 1 is a comment line identifying the test. Line 2 executes calc.exe, the Windows calculator. Line 3 waits up to five seconds for Calculator to start. It does this by pausing until a window appears with the word Calculator in its title bar. Line 4 types the keys 123100=. Line 5 displays a message prompt stating that the answer should be 23. Line 6 closes the Calculator window and ends the test.

Notice that programmed macros such as this one have some real advantages over recorded macros. Although they still can't perform verification of the test results, they can pause their execution to prompt the tester (see Figure 15.8) with an expected result and a query for her to okay whether the test passed or failed.

Figure 15.8. Simple programmed macros can't verify the results of a test but can prompt the tester for confirmation. (Figure courtesy of Iolo Technologies, www.iolo.com.).

Programmed macros can also solve many timing problems of recorded macros by not relying on absolute delays but instead waiting for certain conditions to occur before they go on. In the Calculator example, the macro waits for the program to load before it continues with the testa much more reliable approach.

So far, so good. With programmed macros you can go a long way toward automating your testing. You have a simple macro language to use, generic commands for driving your software, and a means to prompt you for information. For many testing tasks, this is more than sufficient and you'll save a great deal of time automating your tests this way.

You're still missing two important pieces, though, to perform complex testing. First, programmed macros are limited to straight-line executionthey can only loop and repeat. Variables and decision statements that you'd find in a regular programming language aren't available. You also don't have the ability to automatically check the results of your test. For these, you need to move to a comprehensive automated testing tool.

Fully Programmable Automated Testing Tools

What if you had the power of a full-fledged programming language, coupled with macro commands that can drive the software being tested, with the additional capacity to perform verification? You'd have the ultimate bug-finding tool! Figure 15.9 shows an example of such a tool.

Figure 15.9. Visual Test, originally developed by Microsoft and now supported through IBM, is an example of a tool that provides a programming environment, macro commands, and verification capabilities in a single package.

Automated testing tools such as Visual Test provide the means for software testers to create very powerful tests. Many are based on the BASIC programming language, making it very easy for even non-programmers to write test code.

If you wanted to try typing the string Hello World! 10,000 times, you'd write a few lines of code such as this:

 FOR i=1 TO 10000 PLAY "Hello World!" NEXT I

If you wanted to move your mouse pointer from the upper left of your 640x480 screen to the lower right and then double-click, you could do it like this:

 PLAY "{MOVETO 0,0}" PLAY "{MOVETO 640,480}" PLAY "{DBLCLICK}"

A testing language can also give you better control features than just clicking specific screen areas or sending individual keystrokes. For example, to click an OK button, you could use the command

 wButtonClick ("OK")

You don't need to know where onscreen the OK button is located. The test software would look for it, find it, and click itjust like a user would. Similarly, there are commands for menus, check boxes, option buttons, list boxes, and so on. Commands such as these provide great flexibility in writing your tests, making them much more readable and reliable. The most important feature that comes with these automation tools is the ability to perform verification, actually checking that the software is doing what's expected. There are several ways to do this:

Screen captures. The first time you run your automated tests, you could capture and save screen images at key points that you know are correct. On future test runs, your automation could then compare the saved screens with the current screens. If they're different, something unexpected happened and the automation could flag it as a bug.
Be aware that tests using screen captures can be a huge effort to maintain. Even a single pixel change will cause the screen comparison to fail. Unless your software's user interface is unchanging, you could end up manually comparing and recapturing screens with each run of your tests. That defeats the whole purpose of automation.
Control values. Rather than capture screens, you could check the value of individual elements in the software's window. If you're testing Calculator, your automation could read the value out of the display field and compare it with what you expected. You could also determine if a button was pressed or a check box was selected. Automation tools provide the means to easily do this within your test program.
File and other output. Similarly, if your program saves data to a filefor example, a word processoryour automation could read it back after creating it and compare it to a known good file. The same techniques would apply if the software being tested sent data over a modem or a network. The automation could be configured to read the data back in and compare it with the data that it expects.
As with screen captures, file comparisons can have issues, too. If the file or the file format includes a date, a counter, or other changing values, the file comparison will fail. You'll need to program your automation tool to ignore these differences.

Verification is the last big hurdle to overcome with automated software testing. Once you have that, you can take nearly any test case and create automation that will make trying that case either much easier or completely automatic.

To get more information about several of the popular test automation products available, visit the following websites:

Software Development Technologies at www.sdtcorp.com
Mercury at www.mercury.com
Segue Software at www.segue.com

These packages can be a bit pricey for individuals since they're targeted mainly at corporate testing teams. But, if you're interested in gaining some experience with them, contact the company and ask for an evaluation copy or, if you're a student, ask for a student discount. Most software tool companies will help you out in hopes that you'll like their product and eventually recommend it to others. Another option is to seek out open source automation tools. There are many available but they vary widely in their capabilities and quality so you'll need to investigate which ones might work best for your testing needs.