| The reference to this appendix can be found in Chapter 12, "Build Verification Tests and Smoke Tests." This testing guide was created by the Developer Division (devdiv) at Microsoft and is intended for people who are interested in building good test plans. Some of the test conditions mentioned might be good for Smoke or BVTs, while others are just good test ideas regardless of what your code does. I hope you can find some good tips or ideas in it. The guide is broken down into the different resource types that a program might depend on. File Tests: Does Your Code Rely on Files? If your code relies on files (which the majority of code written does), you should test these scenarios: If a file that the application depends upon is removed or renamed, what would happen? Does the application crash or exit gracefully? (For example, in Visual Studio, the ToolBox depends on a .tbd file that is stored in the users/appdata directory. What would happen if we deleted that file?) What happens if the file doesn't have the expected structure or contents; is it corrupt? For example, the VS Start Page expects that a custom tab XML file will comply with the schema and will be correctly formatted XML. What happens if it doesn't, or if instead of XML, we have a simple text file or binary file? What if the file is in the expected format, but the values of the data that we get from the file are invalid? (For example, wrong data type, out-of-value bounds, containing invalid chars, and so on.) Does your feature expect a file to be ASCII? What happens if it has a different encoding, such as Unicode or UTF8? Does the application expect that the file it reads from will not exceed a certain size? What happens when we try working with big files? While you are doing this, what if the file is 0 length? What happens if you try to use a file that is in use by another process or user? What happens if the application depends on a file, but the permissions set on the file don't allow it to access the file? Try security versus read/write/execute permissions. What if the file is just hidden? If the disc is full, what is the result? Use a floppy disc, for example, and do reads/writes and open/close the application. You can use tools to emulate disc full conditions. Canned Heat is a good tool for that. Use FileMon to identify file system access. What happens if you can access the media at first but it becomes unavailable while the application is doing work? Errors when accessing media can happen if the hard drive, floppy drive, CD-ROM drive, and so on are unavailable or slow. Use Canned Heat to emulate the errors or delays. Just pop out the disc you are writing to or disconnect from the network share you were using and see what happens. Windows 2000 and above allow junction points. A junction point is basically a link from one point on disc to another. If your features do any type of recursive directory walking, you should try creating a loop and see how your feature handles it. For example, say that you map c:\myproject\linkedfolder to c:\myproject, and you have a file foo.txt in c:\myproject that contains the word foo in it. Now, if you do "Find in Files" from Visual Studio starting from c:\myproject and searching for "foo," you'll find 14 matches instead of 1. You can use linkd.exe to create junction points.
File Paths: Are File Paths Ever Given As Input to Your Program? Testing file paths is critical, so if you use them, check for these conditions: Invalid filename (invalid characters, and so on). Make sure your feature uses the operating system to handle invalid filenames, rather than writing its own interpretation of that code. Filename/path longer than the allowed max path. Be aware of when your feature adds something to the filename specified by the user. What if the user file path was already Max length and we still added an .aspx extension to the end of the path? It's important to test for the valid longest filename. Is this a hard-coded number? If so, why? Filename with spaces, dots, or semicolons; check these invalid or valid inputs. Using reserved names such as COM1 or AUX to test any functions that create or open files. The list of reserved names: CON, PRN, AUX, CLOCK$, NUL, COM1-COM9, LPT1-LPT9. Varied filenames, using all operating system allowed characters (`output^%$#@#.,!@#$%^)(& is a valid filename). Does your feature depend on the name of the file it uses? Try the same filename canonical representations: trailing white spaces, .\foo = foo, short file format (aka 8.3 format: <6chars>~<number>.<exte nsion>). (See Canonical Representation issues from Writing Secure Code for many more examples.) Check for paths of type: \\?\, file://? Try saving or opening by specifying a directory path rather than a file path. What is the default path that your features use when the user tries to open, save, or find a file? Does that default make sense? Are we changing the default according to the path that the user navigated to the previous time?
Input from Registry: Is the Registry Used to Store or Retrieve Information? Try these tests if your application is using the registry. If the registry keys are deleted (removed from system), what happens? What about registry data type changes (delete key, create same named key with different data type)? Try changing access control lists on folder that contains the key. (Read only, can't set value, and so on.) For example: Remove the key and then remove the right to create a key; see how the application responds. Delete a folder that contains the key. Data content changed. (See the API tests that follow.) Make sure no user-specific data is written in the HKLM tree (and vice versa). You can use RegMon to find the registry keys that your features are using.
Strings: Do You Input Strings into Your Application? Since most applications have at least one location where strings are used as input, you should test these various inputs: Input a null string. What happens? Enter a string without '/0' (if usig C or C++ code). Does the app crash? Try different extended ASCII characters (for example, Alt-0233). What about reserved strings or sequences that require special treatment? Or strings that mean something to the underlying code, such as "++", "&", "\n" (C++)? There are also special ASCII and UNICODE characters, such as Ctrl characters, that should be tested. Don't forget reserved device names, such as AUX and COM1. A good table enumerating special/troublesome ASCII characters can be found on page 29 of How to Break Software by James Whittaker. Using long strings that reach the maximum limit and over the limit of any functions that have string parameters. Input white space, carriage return, tab characters. Test International strings. There are lots of things you need to do here that I won't get into details on. Enable INTL character sets on your machine and have at it. Most INTL strings tend to take up 30% more space than the ENU versions. Look for overlaps and poor UI in your LOC versions. You can sometimes get around various UI verifications by pasting the text rather than typing it.
Numeric Values: Does Your Program Take Numeric Values for Input? After checking string inputs, you should test any numeric values that can input into your program: Try entering negative values, zero, or nothing (in VB, where applicable). Try least numeric value and greatest numeric value for type. Does your numeric value input have boundaries (from 1 to 50)? If yes, test those. Test at least 1, 50, 0, 51, and, of course, valid inputs. Test different data types. (For example, if the input expects int, enter a real number or a character.)
What Inputs Does Your Web Application Take? For web applications, all previous tests apply as well as testing these inputs: Escape sequences that are not allowed or checked to ensure that they do not allow something malicious to render. (For example, there are four ways to represent '\'. '\' = %5C = %255C = %%35%63 = %25%35%63'.) Look for HTML encoding check where applicable: '<' = < = < = &60. See http://www.w3.org/TR/REC-html40/sgml/entities.html. See Chapter 12 of Writing Secure Code for more checks.
Network Connectivity Tests: Are You Connected to a LAN or WAN? These tests are for all features that need network access. You can control the network speed/access by using tools, such as Canned Heat. The things to check for are hangs, crashes, and useless error messages. General Advice It's always bad to generalize, but this might be one exception. When testing your application, you should Get a list of all the errors that the code is supposed to handle and try causing all of them at least once. Are the invalid inputs handled correctly? Are there cases that you think the code should be handling, but you don't see an associated error message? Test them. Are all error messages easy to understand and descriptive?
Try using the default values or blank inputs (where applicable). Ask yourself whether the defaults are reasonable. Is that what the user would want to have as the default value? Try strings that have special meaning for the operating system, for Underlying Programming Language, or for the corresponding character set. Test for buffer overruns. Make sure that your feature doesn't allow code or script to be injected through input. Online features and features that use SQL are especially susceptible to this kind of problem. Test, repeating the same input or series of inputs many times. Don't test input only as individual fields. Find related inputs and test interesting combinations. For example, if you have month and date fields, test February and 1, 0, 28, and 29. Think about all the possible outputs that some input can result in. Try to cause all of them by changing the context/environment in which the code executes. For example, when you're trying to read a file from disc, here are some possible outputs: File is not found. We could not read the file. File is corrupted, so we can't read it. No permissions to read the file. File is locked, so can't read. Successfully read the file. The program hung because the file was too big.
References Here are the three books mentioned in this appendix that I feel should be on every developer's/tester's desk: Writing secure code Writing Secure Code, Second Edition, by Michael Howard and David C. LeBlanc How to break software How to Break Software: A Practical Guide to Testing, by James A. Whittaker Developing international software Developing International Software, Second Edition, by Dr. International
Tools You can download the tools mentioned from Canned Heat http://se.fit.edu/cannedheat FileMon http://www.sysinternals.com/ntw2k/source/filemon.shtml RegMon http://www.sysinternals.com/ntw2k/source/regmon.shtml
I do not see any reason why anyone involved in a software project would not benefit from going over this guide. Developers will write better code if they keep these test scenarios in mind, testers will have a good foundation to build more in-depth tests on, and Program Managers will gain a better understanding of the various scenarios that can come up. |
