Choosing an Off-the-Shelf Tool

Your team isn't able to write a tool you need in-house, for whatever reason. You can't find open-source tools that do the job you need done. So you've decided to look at commercial tools. How do you know if you're selecting an appropriate tool?

In many large, traditional software development organizations, tool selection can take months. First they put together a large, detailed requirements definition. Then they evaluate various tools, spending weeks or months looking at each one. Finally. they compare the pros and cons and choose a tool.

With many projects, when you need a tool for something, you need it right now. You don't have a staff to help you select it. You can't afford "shelfware": fancy tools that sit shrink-wrapped because nobody has time to go to a week-long class to learn them. How do you get something useful quickly?

If open-source tools don't meet your needs and in-house development isn't feasible, do some research to see what off-the-shelf tools may already do what you require. There are lots of quality assurance and testing Web sites that list and even review tools as well as exploring various approaches to automation for example, Bret Pettichord's hotlist, www.pettichord.com, and Brian Marick's list of agile testing links at www.testing.com/agile and his index of automation techniques at www.testingcraft.com/techniques.html.

Also see our bibliography for books that may guide you on this subject. Ask coworkers and even other vendors you use if they know of anything that might fit the bill. Talk to people in your local quality assurance and testing user group or XP user group. Post a message on related Usenet groups (see www.xptester.org for links) to see what other people might be using for the same purpose.

If you've decided to try a vendor tool and have found one or more candidate products, you need a quick way to separate the winners from the also-rans. Check out the vendor. How easy is it to get a price quote? The ponderousness of a tool seems directly related to the difficulty of getting a straightforward price quote. Lightweight tools usually just have a price, and it's posted on the vendor's Web site. Can the salespeople answer technical questions? Is this tool the specialty of the vendor? This could be an indicator of how good their post-sales support will be.

Get references and see what kinds of businesses the other customers are. Recruit experts from your own organization to join you in calling the references.

If a tool seems to fit, install an evaluation copy and try it out for an iteration or two. Is it easy to install? How long does it take you to ramp up enough to make productive use of it? How good is the tech support? When you're moving at the speed of XP, these issues are critical.

Figure out the cost of the tool, including any training or implementation costs you might incur not just dollars but time spent. Cost obviously plays a big part in the risk. If you find a tool that seems to do what you want and is inexpensive, you can buy it without much worry. If it turns out to be a dud, you're not out much money, even if you get no further use out of it. Be extremely careful of expensive tools, because if they don't work out, not only will you be out the money, but the big investment will keep you trying to use it way past when you should have abandoned it.

Look past the bells and whistles. Don't fall into the trap of making the "safe" decision. You can be criticized for buying the best-selling tool if it adds too much overhead to your testing or doesn't do what you need.

Be careful about anticipating future needs. If you're making a huge investment, you do need to think about what the tool will do for you down the road. Avoid the huge investment if you can. Don't invest heavily in more tool than you need. Always do the simplest thing that could possibly work. The reason your team is using XP is because you can't anticipate what's around the next bend in the road.

Any tool you choose is likely to be used by the entire team. Involve others in the decision-making process. Everyone needs to buy off on the tool, or not everyone will use it.