Realities of Using Test Tools and Automation

Before you get all excited and want to run out and start using tools and automation on your tests, you need to read this section and take it to heart. Test automation isn't a panacea. When it is properly planned and executed it can make your testing immensely more efficient and find bugs that would have otherwise gone undiscovered. However, countless test automation efforts have been abandoned and cost their projects dearly when they went astray.

You should consider these important issues before you begin using the techniques described in this chapter:

• The software changes. Specifications are never fixed. New features are added late. The product name can change at the last minute. What if you recorded thousands of macros to run all your tests and a week before the product was to be released, the software was changed to display an extra screen when it started up? All of your recorded macros would fail to run because they wouldn't know the extra screen was there. You need to write your automation so that it's flexible and can easily and quickly be changed if necessary.

• There's no substitute for the human eye and intuition. Smart monkeys can be programmed to be only so smart. They can test only what you tell them to test. They can never see something and say, "Gee, that looks funny. I should do some more checking"at least, not yet.

• Verification is hard to do. If you're testing a user interface, the obvious and simplest method to verify your test results is capturing and comparing screens. But, captured screens are huge files and those screens can be constantly changing during the product's development. Make sure that your tools check only what they need to and can efficiently handle changes during product development.

• It's easy to rely on automation too much. Don't ever assume that because all your automation runs without finding a bug that there are no more bugs to find. They're still in there. It's the pesticide paradox.

• Don't spend so much time working on tools and automation that you fail to test the software. It's easy and fun to start writing macros or programming a smart monkey, but that's not testing. These tools may help you be more efficient, but you'll need to use them on the software and do some real testing to find bugs.

• If you're writing macros, developing a tool, or programming a monkey, you're doing development work. You should follow the same standards and guidelines that you ask of your programmers. Just because you're a tester doesn't mean you can break the rules.

• Some tools are invasive and can cause the software being tested to improperly fail. If you use a tool that finds a bug, try to re-create that bug by hand without using the tool. It might turn out to be a simple reproducible bug, or the tool might be the cause of the problem.