Exploratory Testing

As she contemplated the setting sun, its dying rays casting the last of their brilliant purple light on the red-gold waters of the lake, Debbie realized that she should never again buy her sunglasses from a guy parked by the side of the road.

— Malinda Lingwall

Introduction

The term "exploratory testing," coined by Cem Kaner in his book Testing Computer Software, refers to an approach to testing that is very different from scripted testing. Rather than a sequential examination of requirements, followed by the design and documentation of test cases, followed by the execution of those test cases, exploratory testing, as defined by James Bach, is "simultaneous learning, test design, and test execution." The tester designs and executes tests while exploring the product.

In an article for StickyMinds.com entitled "Exploratory Testing and the Planning Myth," Bach wrote, "Exploratory Testing, as I practice it, usually proceeds according to a conscious plan. But not a rigorous plan ... it's not scripted in detail." James adds, "Rigor requires certainty and implies completeness, but I perform exploratory testing precisely because there's so much I don't know about the product and I know my testing can never be fully complete." James continues, "To the extent that the next test we do is influenced by the result of the last test we did, we are doing exploratory testing. We become more exploratory when we can't tell what tests should be run, in advance of the test cycle."

Exploratory Testing

To the extent that the next test we do is influenced by the result of the last test we did, we are doing exploratory testing. We become more exploratory when we can't tell what tests should be run, in advance of the test cycle.

In exploratory testing, the tester controls the design of test cases as they are performed rather than days, weeks, or even months before. In addition, the information the tester gains from executing a set of tests then guides the tester in designing and executing the next set of tests.

Note this process is called exploratory testing to distinguish it from ad hoc testing which (by my definition, although others may disagree) often denotes sloppy, careless, unfocused, random, and unskilled testing. Anyone, no matter what their experience or skill level, can do ad hoc testing. That kind of testing is ineffective against all but the most defect-ridden systems, and even then may not find a substantial portion of the defects.

Bach suggests that in today's topsy-turvy world of incomplete, rapidly changing requirements and minimal time for testing, the classical sequential approach of Test Analysis followed by Test Design followed by Test Creation followed by Test Execution is like playing the game of "Twenty Questions" by writing out all the questions in advance. Consider the following discussion from a testing seminar discussing exploratory testing:

• Instructor: Let's play a game called "Twenty Questions." I am thinking about something in the universe. I'm giving you, the class, twenty questions to identify what I'm thinking about. Each question must be phrased in a way that it can be answered "Yes" or "No." (If I let you phrase the question in any form you could ask "What are you thinking about" and we would then call this game "One Question.") Ready? Brian, let's begin with you.

  Twenty Questions: The Game

  A game in which one person thinks of something and others ask up to 20 questions to determine what has been selected. The questions must be answerable "Yes" or "No."

  When played well, each question is based on the previous questions and their answers. Writing the questions out in advance prevents using the knowledge acquired from each answer.

• Brian: Does it have anything to do with software testing?
• Instructor: No, that would be too easy.
• Michael: Is it large?
• Instructor: No, it's not large.
• Rebecca: Is it an animal?
• Instructor: No.
• Rayanne: Is it a plant?
• Instructor: Yes, it is a plant.
• Henry: Is it a tree?
• Instructor: No, it is not a tree.
• Sree: Is it big?
• Instructor: No, I've already said it is not large.
• Eric: Is it green?
• Instructor: Yes, it is green.
• Cheryl: Does it have leaves?
• Instructor: Yes, it has leaves.
• Galina: Is it an outdoor plant?
• Instructor: Yes, it generally grows outdoors.
• Jae: Is it a flowering plant?
• Instructor: No, I don't believe so but I'm not a botanist.
• Melanie: Is it a shrub?
• Instructor: No.
• Patrick: Is it a cactus?
• Instructor: No, it is not a cactus.
• Angel: Is it a cucumber?
• Instructor: No, perhaps rather than guessing individual plants it would be more effective to identify categories.
• Sundari: Is it a weed?
• Instructor: No, good try though.
• Lynn: Is it a perennial?
• Instructor: No, I don't believe so. I think it must be replanted each year.
• Julie: Does it grow from bulbs?
• Instructor: No.
• Michelle: Is it in everyone's yard?
• Instructor: No, at least it's not in mine.
• Kristie: Is it illegal? (Laughter in the class)
• Instructor: No, it's quite legal. Well, we've gone through the class once. Brian, let's go back to you.
• Brian: Is it poisonous?
• Instructor: No, although my children think so.
• Michael: Is it eaten?
• Instructor: Yes, it is eaten.
• Rebecca: Is it lettuce?
• Instructor: No, not lettuce.
• Rayanne: Is it spinach?
• Instructor: Yes, it is spinach. Very good.

How successful would we be at this game if we had to write out all the questions in advance? When we play this game well, each question depends on the previous questions and their answers. So it is in exploratory testing. Each test provides us with information about the product. We may see evidence of the product's correctness; we may see evidence of its defects. We may see things that are curious; we're not sure what they mean, things that we wonder about and want to explore further. So, as we practice exploratory testing, we concurrently learn the product, design the tests, and execute these tests.

Description

In his classic time management book, How to Get Control of Your Time and Your Life, Alan Lakein suggests we should constantly ask ourselves: What is the most important thing I can do with my time right now? Exploratory testers ask an equivalent question: What is the most important test I can perform right now?

Key Question

What is the most important test I can perform right now?

A possible exploratory testing process is:

• Creating a conjecture (a mental model) of the proper functioning of the system
• Designing one or more tests that would disprove the conjecture
• Executing these tests and observing the outcomes
• Evaluating the outcomes against the conjecture
• Repeating this process until the conjecture is proved or disproved

Another process might be simply to explore and learn before forming conjectures of proper behavior.

Exploratory testing can be done within a "timebox," an uninterrupted block of time devoted to testing. These are typically between sixty and 120 minutes in length. This is long enough to perform solid testing but short enough so that the tester does not mentally wander. In addition, a timebox of this length is typically easier to schedule, easier to control, and easier to report.

When performing "chartered exploratory testing," a charter is first created to guide the tester within the timebox. This charter defines a clear mission for the testing session. The charter may define:

• What to test
• What documents (requirements, design, user manual, etc.) are available to the tester
• What tactics to use
• What kinds of defects to look for
• What risks are involved

This charter is a guideline to be used, not a script to be followed. Because of this approach, exploratory testing makes full use of the skills of testers. Bach writes, "The more we can make testing intellectually rich and fluid, the more likely we will hit upon the right tests at the right time."

Key Point

The charter is a guideline to be used, not a script to be followed.

Charters focus the exploratory tester's efforts within the timebox. Possible charters include:

• Thoroughly investigate a specific system function
• Define and then examine the system's workflows
• Identify and verify all the claims made in the user manual
• Understand the performance characteristics of the software
• Ensure that all input fields are properly validated
• Force all error conditions to verify each error message
• Check the design against user interface standards

It is possible to perform exploratory testing without a charter. This is called "freestyle exploratory testing." In this process testers use their skills to the utmost as they concurrently learn the product and design and execute tests.

Exploratory testers are skilled testers. (Of course, we want testers to be skilled no matter what testing process we are using!) The exploratory testing approach respects those skills and, in fact, depends on them. Good exploratory testers are:

• Good modelers, able to create mental models of the system and its proper behavior.
• Careful observers, able to see, hear, read, and comprehend.
• Skilled test designers, able to choose appropriate test design techniques in each situation. Bach emphasizes, "An exploratory tester is first and foremost a test designer."
• Able to evaluate risk and let it guide their testing.
• Critical thinkers, able to generate diverse ideas, integrate their observations, skills, and experiences to concurrently explore the product, design the tests, and execute the tests.
• Careful reporters, able to rigorously and effectively report to others what they have observed.
• Self managed, able to take the lead in testing rather than execute a plan devised by others.
• Not distracted by trivial matters.

Testers without these skills can still perform useful exploratory testing if they are properly supervised and coached.

In general, processes that have weak, slow, or nonexistent feedback mechanisms often do not perform well. Scripted testing is a prime example of a slow feedback loop. Exploratory testing provides a tight feedback loop between both test design and test execution. In addition, it provides tight feedback between testers and developers regarding the quality of the product being tested.

Advantages of Exploratory Testing

1. Exploratory testing is valuable in situations where choosing the next test case to be run cannot be determined in advance, but should be based on previous tests and their results.
2. Exploratory testing is useful when you are asked to provide rapid feedback on a product's quality on short notice, with little time, off the top of your head, when requirements are vague or even nonexistent, or early in the development process when the system may be unstable.
3. Exploratory testing is useful when, once a defect is detected, we want to explore the size, scope, and variations of that defect to provide better feedback to our developers.
4. Exploratory testing is a useful addition to scripted testing when the scripted tests become "tired," that is, they are not detecting many errors.

Disadvantages of Exploratory Testing

1. Exploratory testing has no ability to prevent defects. Because the design of scripted test cases begins during the requirements gathering and design phases, defects can be identified and corrected earlier.
2. If you are already sure exactly which tests must be executed, and in which order, there is no need to explore. Write and then execute scripted tests.
3. If you are required by contract, rule, or regulation to use scripted testing then do so. Consider adding exploratory tests as a complementary technique.

Summary

• Exploratory testing is defined as "simultaneous learning, test design, and test execution." The tester designs and executes tests while exploring the product.
• In exploratory testing, the tester controls the design of test cases as they are performed rather than days, weeks, or even months before. In addition, the information the tester gains from executing a set of tests then guides the tester in designing and executing the next set of tests.
• Exploratory testing is vital whenever choosing the next test case to be run cannot be determined in advance but should be chosen based on previous tests and their results.

References

Bach, James. "Exploratory Testing and the Planning Myth." http://www.stickyminds.com/r.asp?F=DART_2359, 19 March 2001.

Bach, James. "Exploratory Testing Explained." v.1.3 16 April 2003. http://www.satisfice.com/articles/et-article.pdf

Kaner, Cem,Jack Falk, and Hung Q. Nguyen (1999). Testing Computer Software. John Wiley & Sons.

Kaner, Cem,James Bach, and Bret Pettichord (2002). Lessons Learned in Software Testing: A Context-Driven Approach. John Wiley & Sons.

Weinberg, Gerald M. (1975). An Introduction to General Systems Thinking. John Wiley & Sons.