Performing a High-Level Review of the Specification

Defining a software product is a difficult process. The spec must deal with many unknowns, take a multitude of changing inputs, and attempt to pull them all together into a document that describes a new product. The process is an inexact science and is prone to having problems.

The first step in testing the specification isn't to jump in and look for specific bugs. The first step is to stand back and view it from a high level. Examine the spec for large fundamental problems, oversights, and omissions. You might consider this more research than testing, but ultimately the research is a means to better understand what the software should do. If you have a better understanding of the whys and hows behind the spec, you'll be much better at examining it in detail.

Pretend to Be the Customer

The easiest thing for a tester to do when he first receives a specification for review is to pretend to be the customer. Do some research about who the customers will be. Talk to your marketing or sales people to get hints on what they know about the end user. If the product is an internal software project, find out who will be using it and talk to them.

It's important to understand the customer's expectations. Remember that the definition of quality means "meeting the customer's needs." As a tester, you must understand those needs to test that the software meets them. To do this effectively doesn't mean that you must be an expert in the field of nuclear physics if you're testing software for a power plant, or that you must be a professional pilot if you're testing a flight simulator. But, gaining some familiarity with the field the software applies to would be a great help.

Above all else, assume nothing. If you review a portion of the spec and don't understand it, don't assume that it's correct and go on. Eventually, you'll have to use this specification to design your software tests, so you'll eventually have to understand it. There's no better time to learn than now. If you find bugs along the way (and you will), all the better.

TIP

Don't forget about software security when pretending to be the customer. Your users will assume that the software is secure, but you can't assume that the programmers will handle security issues properly. It must be specifiedin detail. Chapter 13, "Testing for Software Security," discusses what you should look for when reviewing the product design and specification for security issues.

Research Existing Standards and Guidelines

Back in the days before Microsoft Windows and the Apple Macintosh, nearly every software product had a different user interface. There were different colors, different menu structures, unlimited ways to open a file, and myriad cryptic commands to get the same tasks done. Moving from one software product to another required complete retraining.

Thankfully, there has been an effort to standardize the hardware and the software. There has also been extensive research done on how people use computers. The result is that we now have products reasonably similar in their look and feel that have been designed with ergonomics in mind. You may argue that the adopted standards and guidelines aren't perfect, that there may be better ways to get certain tasks done, but efficiency has greatly improved because of this commonality.

Chapter 11, "Usability Testing," will cover this topic in more detail, but for now you should think about what standards and guidelines might apply to your product.

NOTE

The difference between standards and guidelines is a matter of degree. A standard is much more firm than a guideline. Standards should be strictly adhered to if your team has decided that it's important to comply with them completely. Guidelines are optional but should be followed. It's not uncommon for a team to decide to use a standard as a guidelineas long as everyone knows that is the plan.

Here are several examples of standards and guidelines to consider. This list isn't definitive. You should research what might apply to your software:

• Corporate Terminology and Conventions. If this software is tailored for a specific company, it should adopt the common terms and conventions used by the employees of that company.

• Industry Requirements. The medical, pharmaceutical, industrial, and financial industries have very strict standards that their software must follow.

• Government Standards. The government, especially the military, has strict standards.

• Graphical User Interface (GUI). If your software runs under Microsoft Windows or Apple Macintosh operating systems, there are published standards and guidelines for how the software should look and feel to a user.

• Security Standards. Your software and its interfaces and protocols may need to meet certain security standards or levels. It may also need to be independently certified that it does, indeed, meet the necessary criteria.

As a tester, your job isn't to define what guidelines and standards should be applied to your software. That job lies with the project manager or whoever is writing the specification. You do, however, need to perform your own investigation to "test" that the correct standards are being used and that none are overlooked. You also have to be aware of these standards and test against them when you verify and validate the software. Consider them as part of the specification.

Review and Test Similar Software

One of the best methods for understanding what your product will become is to research similar software. This could be a competitor's product or something similar to what your team is creating. It's likely that the project manager or others who are specifying your product have already done this, so it should be relatively easy to get access to what products they used in their research. The software likely won't be an exact match (that's why you're creating new software, right?), but it should help you think about test situations and test approaches. It should also flag potential problems that may not have been considered.

Some things to look for when reviewing competitive products include

• Scale. Will there be fewer or greater features? Will there be less or more code? Will that size difference matter in your testing?

• Complexity. Will your software be more or less complex? Will this impact your testing?

• Testability. Will you have the resources, time, and expertise to test software such as this?

• Quality/Reliability. Is this software representative of the overall quality planned for your software? Will your software be more or less reliable?

• Security. How does the competitor's software security, both advertised and actual, compare to what you'll be offering?

There's no substitute for hands-on experience, so do whatever you can to get a hold of similar software, use it, bang on it, and put it through its paces. You'll gain a lot of experience that will help you when you review your specification in detail.

TIP

Don't forget about reading online and printed software reviews and articles about the competition. This can be especially helpful for security issues as you may not likely see the security flaws as you casually use the application. They will, though, be well known in the press.