QA and Testing | Coder to Developer: Tools and Strategies for Delivering Your Software

< Day Day Up >

Where do bugs come from? Or, more precisely, how do they get identified (since whether or not you want to admit it, you know quite well where they come from; they’re inserted into the product by your own efforts and those of the rest of your development team).

If you’re fortunate enough to be working in a medium or large organization, bugs are normally found by people working in the quality assurance (QA) department—the testers. As a developer, you shouldn’t view testers as the enemy. Rather, they’re your allies in making a better product. In this section, I’ll give you a brief overview of the activities of a good software tester, and talk about how to be your own tester.

Types of Software Testing

Here are some of the types of testing that might go on with any given application. This overview won’t give you enough information for you to actually work as a software tester, but it should be sufficient to help you speak the same language.

Unit testing In unit testing, a single component or function is tested to make sure that it’s bug free. Typically, unit testing is done by developers rather than by testers, as I discussed in Chapter 5, “Preventing Bugs with Unit Testing.”
Functional testing This involves testing the application to make sure that it conforms to the requirements and specifications. A tester can carry out thorough functional testing only if there is a written specification for the product, which is one more good reason for committing specifications to writing. Even if there is no written specification, though, testers can perform functional testing simply by exercising the application and deciding whether its behavior makes sense.
Conformance testing The purpose of conformance testing is to ensure that the application conforms to industry specifications. For example, if the application outputs Extensible Markup Language (XML) files, conformance testing would check whether these files conform to the World Wide Web Consortium (W3C) XML specification.
Compatibility testing The goal of compatibility testing is to check that the application runs in as wide a variety of computing environments as possible. This might involve using different sets of hardware, different versions of the operating system, different user interface languages, or different browsers. Compatibility testing is often quite challenging due to the sheer number of possibilities involved.
Performance testing This testing involves identifying performance issues or creating benchmarks under normal operating conditions. The goal of performance testing is to check that the application’s performance will be acceptable to users.
Stress testing Stress testing identifies how the application fails when it is subjected to excessive stress. Stress testing often involves using software to simulate a large number of users, low RAM, CPU contention, and other unusual conditions. The goal is to make sure that the application failure doesn’t result in damage to the operating system, data loss, or other undesirable consequences.
Regression testing When a new build of the application is made available to testers, one of the first things they normally do is run a suite of regression tests—tests that were passed by previous builds of the software. This is an easy way to make sure that new features didn’t break existing functionality. Developers can also perform their own regression testing by maintaining a suite of unit tests.
Smoke testing This involves running “quick and dirty” tests that exercise major features without which it’s pointless to continue testing. If the smoke tests fail, the build is considered completely broken and testers normally wait for the next build.
Black-box testing This testing of components is based strictly on their external interfaces, treating the implementation as a “black box.” Most testing done by QA departments is black-box testing.
White-box testing Here the internal behavior of components is tested, with full knowledge of the implementation. Unit testing by developers tends to be white-box testing.

Many books, tools, and other resources are available to help software testers do their jobs. This book isn’t primarily about testing, so I won’t try to review this literature here. One good starting point if you’d like to learn more is the Software QA/Test Resource Center (www.softwareqatest.com/index.html). A critical piece of testing software in many organizations is a test framework that allows saving and consistently executing a set of tests. One excellent .NET-aware framework is TestComplete, from AutomatedQA Corp. (www.automatedqa.com/).

QA for the Lone Wolf

At many software companies, the software development staff breaks down something like this:

One third management and architects
One third developers
One third testers

But when you’re working by yourself, it’s hard to saw off an arm and a leg and call them your QA department. You will likely need to spend time being your own tester. Unfortunately, the plain fact is that many good developers are lousy testers. It’s difficult to take code that you’ve just written and then systematically attempt to prove that you haven’t finished with it after all. Here are a few tips for the developer who doesn’t have a testing department to call on:

Use unit tests to keep bugs from creeping into your code in the first place.
Create a list of critical requirements so you can perform a “smoke test” on each new build.
Set code aside for a few days before performing functional testing on it.
If at all possible, get someone else to test your application. Perhaps you can send it to external beta testers, swap-test with another independent developer, or convince your teenagers that your application is a video game in disguise.
Keep a written list of requirements, and test to make sure the application meets each one.
Use a bug-tracking system to keep track of past bugs. This will help you learn which types of bugs you are most prone to introduce.
Be sure to check absurd input: 0 where you know only positive numbers should be entered, letters in place of numbers, dates with meaningless years, and so on. Even if your users don’t try to mess up your application this way, they’ll make typing mistakes with the same effect.
If you think of something that might go wrong while you’re in the middle of coding, enter it as a bug in your bug-tracking system. This will help you remember to test the problem when you’re not wearing your coding hat.

As the developer of the code, you do have one advantage: All of your testing can be white-box testing. You should make full use of the source code to help prevent bugs, by coding defensively (see Chapter 4, “Coding Defensively”) and by reading over your work to make sure you’ve thought of everything. When you do discover a bug in functional testing, take one step beyond fixing the bug and think about how you could have prevented it in the first place—more unit tests? Better checking of input conditions? More thorough code review? Over time, you should be able to identify the most common sources of bugs in your own code and eliminate them.

RULE

Treat bugs as a learning experience, not as a threat to your coding skills.

Finally, keep in mind that it is extremely difficult to be an effective tester of your own application. It’s quite common to develop blind spots where you simply don’t notice how an application is subtly failing, or to avoid testing something absurd because you know that it doesn’t make sense. Even if you have to do the majority of your own testing, try to find someone, whether it be spouse, best friend, or next-door neighbor, to at least try out your application a few times.

Building a Test Network

For the most part, books on software development treat software as pure stuff of the mind, sprung from the efforts of our intense concentration on some kind of ethereal being. In reality, of course, CPUs and RAM and monitors and lots of other hardware are involved. Although I’ve been ignoring the subject of hardware myself, this is a good point to bring grubby reality into the discussion. To test your software effectively, you need a network of computers. The more hardware resources you can put into testing, the more likely it is that you can duplicate peculiar configurations of hardware and software that shake out bugs in your product. For that reason, I’m going to put some thoughts about hardware for developers into this chapter.

Consider the sections that follow as guidelines, not hard-and-fast rules. What works for me in the hardware arena may not work quite as well for you.

Buy preassembled machines. Purchase preassembled machines, not a basket of parts. Yes, you’re a smart person and could build your own computer—but other people do that for a living, and they buy in bulk. You won’t save more than $100 or so by buying parts, and you’ll waste a lot of evenings. In the worst possible case, you’ll also blow up something expensive, like a CPU. I have.
Buy name brands. Buying from established vendors is always best. If you’re in a decent-sized city, it’s okay to buy from the local “white-box” shop if you get a good feeling when you walk in the door. (A “white box” is a no-name computer assembled from parts that have likely been imported from Taiwan.) If you’re in the hinterlands and working by mail, you don’t want to take that chance.
Buy serious development hardware. For your main development box, buy as much as you can afford: RAM, hard drive, CPU speed, video resolution, monitor. You should be billing a high enough hourly rate that the productivity is worth it. Of course, if you’re not doing hourly work, you may have to make some compromises here. In my own work, I’ve found that the most important factor is plenty of RAM, followed by a fast-enough CPU and a good-enough video card.
Skip the tape drive. You can safely skip the tape drive; most developers never find time to set up a proper tape rotation schedule, or to take the tapes offsite. Instead, buy fast hard drives and store your data on a pair of mirrored drives. This protects you against drive failure. It does not protect you against house fires. If you’re worried about that, use external USB 2.0 or FireWire drives for your data, and rotate them offsite frequently. I figure I’ll have so many other things to worry about if the farm burns down that it’s not worth the hassle for me. (I also back up critical files to a colocated server at my ISP’s facility, and have collaborators on most of my coding projects who have full copies of the source code in other states.)
Store data separately. I find that I repave (reformat and install everything from scratch) my development box about once every six months. This used to involve copying off all the data files, formatting, and putting on all the programs, then copying back the data files. I finally realized that at least some of that was pointless work. Now my main development box is actually two machines. One is the spiffy fast Athlon with all the RAM. This box has all of my programs installed but none of the data. The second box has a reasonable amount of RAM, a midrange CPU, and a ton of mirrored hard drive space. It stores all of my data: e-mail, source code, book chapters, articles, and so on. The two boxes are connected by a pair of fast network cards and a crossover cable, with addresses outside my main network space. Net result: I can completely reformat my program box without touching the data box. When I’m done reinstalling, I just map the drive back from the data box, and my data is there ready for me to get back to work. Of course, I don’t install anything other than the inevitable Windows patches on the data box in order to keep it stable.
Keep drive images. After you set up your computer the way you like it, but before you start working, consider using an application such as Norton Ghost (www.symantec.com/sabu/ghost/ghost_personal/) to store a snapshot of the drive, either to another hard drive or to a CD-ROM or DVD-ROM. This will speed up the inevitable rebuilds immensely. Of course, because they won’t have the latest security patches, such snapshots do age to the point of uselessness eventually, but if you have to rebuild relatively soon they’ll save you lots of time.
Use virtual machines. Virtual machines are your friend. As a developer, you’re likely need to test all sorts of crazy software that you don’t want to run on your main box. That’s what VMware Workstation (www.vmware.com/products/desktop/ws_features.html) or Microsoft Virtual PC 2004 (www.microsoft.com/windowsxp/virtualpc/) is for. With virtual machine software, you can create an entire computer inside a window on another computer, without installing anything on the real hardware. You can use virtual machines for setting up Bulgarian Windows 98 SE, or installing the latest Firebird beta so you can see how broken your CSS is, or testing at 640 480 screen resolution. My recommendation: Get another box just to run the virtual machines. Yes, you could run them on your main box, but these products suck down lots of CPU time, especially when you’re starting and snapshotting and stopping machines. I personally don’t care to wait for that.
Treat yourself to a good monitor or two. A good monitor is a great productivity booster—multiple monitors are even better. I like lots of pixels. LCD screens are sleek and sexy, but they don’t have enough pixels to suit me. I’m running a Sony Trinitron real CRT monitor on my development box. If I had enough physical desktop room, I’d probably put two of them side by side.
Use KVM switches. I have 21 computers in my home office network at the moment. I do not have 21 monitors. I use Terminal Services or Remote Desktop to manage some of them. Most, though, are on a keyboard-video-mouse (KVM) switch, so I can have a nice keyboard and monitor no matter which system I’m trying to work with. If you’re buying a KVM switch, make sure you get one that passes the mouse wheel signal through and that uses standard cables rather than hideously expensive proprietary ones.
Get your own domain. If you’re working with Internet technologies at all, I recommend you get your own domain routed to your house and learn to set up your own web server. This will give you a handy test bed, and teach you much about what will go wrong. If you’re working with databases, set up your own database server. If you’re developing against servers, you need to have ones that you can crash without upsetting anyone else.
Use a firewall. When you’re connected to the network full time, you must install a firewall. I personally prefer to use a hardware firewall, rather than complicate my software setup. Life is too short to spend a lot of time eradicating viruses and dealing with security holes.
Set aside test machines. If you can afford a spare computer or two (check eBay for deals on not-so-new business computers), set them aside for use in testing. You really shouldn’t test your own software, or new tools, on your main development computer. It’s much safer to use a computer without anything important installed, so that if something goes terribly wrong, you can just reformat and reinstall your disk image rather than struggling to recover critical data or reinstall the programs that you use every day.
Use a mix of machines for testing. It’s unlikely that all of your users will have brand-new hardware. Test on a variety of systems: new, old, and ancient. Fortunately, older computers can often be had for little or nothing from friends or relatives who are upgrading (though they may expect you to help set up their new system in return!).
Set aside a build machine. It’s also useful to set aside one computer, with a carefully controlled configuration, to act as the build machine for your project. By building every version of the application on a single machine, you can eliminate problems, such as newer libraries creeping in because you upgraded some piece of software on your development machine. Alternatively, you can configure and use a virtual machine for this process. I’ll talk more about the build process in Chapter 13, “Mastering the Build Process.”

< Day Day Up >