Random Testing: Monkeys and Gorillas


The test automation tools and techniques that you've learned about so far have concentrated on making your job as a software tester easier and more efficient. They're designed to help you in running your test cases or, ideally, running your test cases automatically without the need for constant attention.

Using tools and automation for this purpose will help you find bugs; while the tools are busy doing regression testing, you'll have time to plan new tests and design new and interesting cases to try. Another type of automated testing, though, isn't designed to help run or automatically run test cases. Its goal is to simulate what your users might do. That type of automation tool is called a test monkey.

The term test monkey comes from the idea that if you had a million monkeys typing on a million keyboards for a million years, statistically, they might eventually write a Shakespearean play, The Adventures of Curious George, or some other great work. All that random pounding of keys could accidentally hit the right combination of letters and the monkeys would, for a moment, look brilliantmuch like the one in Figure 15.10.

Figure 15.10. Test monkeys will test forever as long as they have electricity and the occasional banana.


When your software is released to the public, it will have thousands or possibly millions of people using it. Despite your best efforts at designing test cases to find bugs, some bugs will slip by and be found by those users. What if you could supplement your test case approach with a simulation of what all those users would do, before you released your product? You could potentially find bugs that would have otherwise made it past your testing. That's what a test monkey can do.

NOTE

The use of a test monkey to simulate how your customers will use your software in no way insinuates that computer users are related to apes.


Dumb Monkeys

The easiest and most straightforward type of test monkey is a dumb monkey. A dumb monkey doesn't know anything about the software being tested; it just clicks or types randomly. Listing 15.2 shows an example of Visual Test code that will randomly click and type 10,000 times.

Listing 15.2. Just a Few Lines of Code Can Create a Dumb Monkey
 1: RANDOMIZE TIMER 2: FOR i=1 TO 10000 3: PLAY "{CLICK "+STR$(INT(RND*640))+", "+STR$(INT(RND*480))+" }" 4: PLAY CHR$(RND*256) 5: NEXT i 

Line 1 initializes the random numbers. Line 2 starts looping from 1 to 10,000 times. Line 3 selects a random point onscreen between 0,0 and 639,479 (VGA resolution) and clicks it. Line 4 picks a random character between 0 and 255 and types it in.

The software running on the PC doesn't know the difference between this program and a real personexcept that it happens much more quickly. On a reasonably speedy PC it will run in just a few seconds. Imagine how many random inputs you'd get if it ran all night!

Remember, this monkey is doing absolutely no verification. It just clicks and types until one of two things happenseither it finishes its loop or the software or the operating system crashes. If the software under test crashes, the monkey won't even know it, and will continue clicking and typing away.

NOTE

If you don't believe that a dumb monkey can possibly find a serious bug, try running one on your favorite computer game or multimedia program. It's very likely that it won't last more than a few hours before crashing.


It doesn't seem to make sense that simple random clicking and typing could find a bug, but it does for a couple reasons:

  • Given enough time and attempts, just like the monkeys writing Shakespeare, the random inputs will eventually stumble onto a magic sequence that the programmers and testers didn't think of. Maybe the monkey enters some data and immediately deletes it or types in a huge string where a short one was expected. Who knows? It will find it, though.

  • A dumb monkey, with its continuous repetition and use, can expose bugs such as memory leaks that might not occur until many hours or days of normal use. If you've ever used software that seemed to become less and less stable the longer you used it, you've seen a problem that could have been found with a dumb monkey.

Semi-Smart Monkeys

Dumb monkeys can be extremely effective. They're easy to write and can find serious, crashing bugs. They lack a few important features, though, that would make them even more effective. Adding these features raises your monkey's IQ a bit, making him semi-smart.

Say that your monkey ran for several hours, logging thousands of random inputs before the software crashed. You'd know there was a problem but you couldn't show the programmer exactly how to re-create it. You could rerun your monkey with the same random seed but if it took several hours again to fail, you'd be wasting a lot of time. The solution is to add logging to your monkey so that everything it does is recorded to a file. When the monkey finds a bug, you need only to look at the log file to see what it was doing before the failure.

TIP

Another solution to track what your monkey does is to set up a video camera to record what happens on the screen. When you notice that the software has failed, just rewind and replay the tape.


It's also a good idea to program your monkey to operate only on the software you're testing. If it's randomly clicking all over the screen, it could (and will eventually) click the exit command and stop the program. Since the monkey doesn't know that the program closed, it'll keep on going. Think about what would happen if the monkey was clicking all over your computer's screenouch! Most programmable automation tools provide a way to always target a specific application, or to stop working if the application is no longer present.

Another good feature to make your monkey semi-smart is crash recognition. If you started your monkey running for the night and it found a bug as soon as you walked out the door, you'd lose many hours of valuable test time. If you can add programming to your monkey to recognize that a crash has occurred, restart the computer, and start running again, you could potentially find several bugs each night.

Smart Monkeys

Moving up on the evolutionary scale is the smart monkey. Such a monkey takes the effectiveness of random testing from his less-intelligent brothers and adds to that an awareness of his surroundings. He doesn't just pound on the keyboard randomlyhe pounds on it with a purpose.

A true smart monkey knows

  • Where he is

  • What he can do there

  • Where he can go

  • Where he's been

  • If what he's seeing is correct

Does this list sound familiar? It should. A smart monkey can read the software's state transition mapthe type of map described in Chapter 5, "Testing the Software with Blinders On." If all the state information that describes the software can be read by the monkey, it could bounce around the software just like a user would, only much more quickly, and be able to verify things as it went.

A smart monkey testing the Windows Calculator (see Figure 15.11) would know what buttons are available to press, what menu items are present, and where to type in the numbers. If it clicked the Help menu's About Calculator option, it would know that the only ways out were to click OK or the Close button. It wouldn't randomly click all over the screen, eventually stumbling onto one of them.

Figure 15.11. A smart monkey would know how to close the About Calculator dialog box.


A smart monkey isn't limited to just looking for crashing bugs, either. It can examine data as it goes, checking the results of its actions and looking for differences from what it expects. If you programmed in your test cases, the smart monkey could randomly execute them, look for bugs, and log the results. Very cool!

Figure 15.12 shows a smart monkey called Koko, named after the gorilla that could speak in sign language. To program Koko, you feed it the state table that describes your software by defining each state, the actions that can be performed in that state, and claims that determine whether the result of executing an action is right or wrong.

Figure 15.12. The Koko smart monkey can be programmed to know where it is and what it can do.


When Koko runs, it drives the software to a known state, randomly selects an action based on a weighting that simulates real-world likelihood, performs that action, and then checks the result. If the action results in the software changing states, Koko knows that and uses a new set of actions that apply to that new state.

With such a system, you could simulate real-world usage of your software, compressing thousands of hours of use into just a few hours. Smart monkeys are truly bug-finding machines!

NOTE

Koko is a proprietary smart monkey developed by the author. It is not commercially available.




    Software Testing
    Lessons Learned in Software Testing
    ISBN: 0471081124
    EAN: 2147483647
    Year: 2005
    Pages: 233

    flylib.com © 2008-2017.
    If you may any questions please contact us: flylib@qtcs.net