Attack - Execute the Plan

Attack—Execute the Plan

Testing your application before you release it is how you will stay one or more steps ahead of an attacker who will later use your application. Up to this point, you’ve emulated what an attacker might do by determining potential points of entry based on a blueprint you’ve created, created scenarios that describe how you will test whether the potential points of entry are secure, and identified the tests needed to validate each scenario. Now it’s time to execute your plan and attack your application!

Testing Approaches

You can choose from several broad approaches to test your application. The advantages and disadvantages of each approach are shown in Table 9-2. The approaches shown in Table 9-2 are valid for general testing as well as for testing that your application is secure. The chief difference between general testing and testing for security is in the type of tests you create and the tools you’ll use to validate an application. In the case of testing for security, your goal is to emulate an attacker by creating tests that perform the same actions an attacker would take, and by running the same type of tools an attacker would run. For either security testing or general testing, your goal is the same: attempt to break the application.

Table 9-2: General Testing Approaches

Test Approach	Advantages	Disadvantages
Writing self-testing code	Catches regressions early Tests are integrated into the code	Requires running a separate debug build of the application
Ad hoc, or manual, testing	Cheap initial cost Good for scenarios that are difficult to automate Good approach for finding new bugs Effective for UI validation Effective for end-to-end validation of an application	High long-term cost if you run the same test repeatedly for each build of the application
Automated unit testing	Protects against regressions Ensures a certain level of quality based on the quality of the tests Relatively inexpensive to repeat all tests on each drop of the product	High initial cost of creating the tests High maintenance cost of updating specs if product features change significantly
Stress testing	Uncovers in high-stress environments application vulnerabilities such as low-memory condition and high-application load Good for assessing application vulnerability to denial of service attacks	Usually run over a long period of time—up to days— before any problems occur Difficult, when problem occurs, to re-create the conditions of failure without rerunning the test, which requires lots of time

Writing Self-Testing Code

Visual Basic .NET provides a rich set of debugging features, such as Debug.Assert and Debug.Fail. These statements are turned on only in debug builds of your application. You can use these statements to validate assumptions being made in your code without incurring a performance penalty in the retail build of the application. For example, if you are certain at a particular point in code that a user name string is not empty and has 20 characters or less (because it has previously been validated), you can add the following:

Debug.Assert(UserName <> "" AndAlso UserName.Length <= 20, _
"Invalid user name string")

This code is not included in the retail build of your application. If for some reason your assumption is false and the user name is invalid, the retail build of your application might try to use the invalid user name or it might silently fail. In either case, you might not be able to tell by running the retail application exactly where the problem is. However, if you run the same scenario using the debug build, an assert—a message box containing the warning “Invalid user name” string—will display. If you litter your code with Debug.Assert calls to check your assumption that the UserName is valid, you can more easily determine the point in code where the UserName value became invalid.

You can also include test code within subroutines where you add the Conditional(“DEBUG”) attribute, as shown here:

‘TODO: Add Imports System.Diagnostics at the top of the code module
<Conditional("DEBUG")> _
Sub TestHackTextControls()
 ‘TODO: Add code which tries passing invalid string
 ' to all text input controls in the application
 ' to validate the error handling for those controls.
End Sub

Any calls to TestHackTextControls will be included only if the symbol DEBUG is defined, which is the case when you build the application within Microsoft Visual Studio .NET with the debug configuration selected. You can use debug builds to perform self-validation of the application, which will slow down the performance of the application. The TestHackTextControls subroutine will not be included (nor will any calls to the subroutine) in retail builds, giving you deep verification checks in debug builds on the one hand and high performance in retail builds on the other.

Ad Hoc, or Manual, Testing

Ad hoc testing, also known as manual testing, is the most straightforward, free- form testing you can do. It simply involves running the application and trying to break it. In ad hoc testing scenarios, you try entering invalid data in all text- entry fields, for example, and see how the application responds. This form of testing gives you the best real-world view of the application and how it will perform. Ad hoc testing is best used with new features to find new bugs as quickly as possible. It also helps you to find bugs your customers would likely encounter when using the application.

The biggest drawback to this form of testing is it takes quite a bit of time to exercise all parts of your application. To verify that nothing has broken between builds, you need to repeat your steps every time a new build of the application is made.

Automated Unit Testing

Automated unit testing involves writing code or scripts that exercise a particular feature of your application and, more importantly, validate that the feature works as expected. You might be surprised to learn that you generally do not use automated tests to find new bugs—although if you decide to add new tests for an area that has not been tested, you will likely uncover an initial set of bugs. Ad hoc, or manual, testing is the best way to uncover new bugs.

An initial up-front cost is required to implement an automation test and any supporting functions needed to log and track its results. However, once you have your automated test system in place, adding new tests should come at a small incremental cost.

The main advantage of having automated tests that you run against every build of the application is that your tests establish a baseline quality measure for the application and help to prevent any regressions in quality. The ideal situation is to go through all known bugs you have fixed in your application, write test cases for every bug, and run those test cases against every build. If the test cases are correctly written and 100 percent of them pass, you can be certain the quality of your application did not regress.

For example, you could write a test to verify that the following CalculatePayments function works correctly when passed invalid values:

Public Function CalculatePayments(ByVal Amount As Decimal, _
 ByVal Duration As Decimal) As Decimal

If the function is documented to throw an InvalidDurationException (an application-defined custom exception derived from System.Exception) when the duration is 0 or less, you could write a test that called the function and passed 0 for the duration parameter as follows:

Try
 CalculatePayments(100, 0)
 LogTestFailed()
Catch exExpected As InvalidDurationException
 LogTestPassed()
Catch exExpected As Exception
 LogTestFailed()
End Try

There are a couple of ways that you could implement the functions LogTestPassed and LogTestFailed:

• The functions could write the passed or failed results to a database. If you want to record the test results for each drop, you can implement the functions LogTestPassed and LogTestFailed to write the name of the test being run, the build of the application being tested, and a pass or fail result to a database table named TestResults. After you have run all your automated tests against a build of your application, you can check the database and view the tests that failed.

• The functions could emit passed or failed status to a text file. Presume that the functions write either PASSED (for LogTestPassed) or FAILED (for LogTestFailed) to a text file named RESULTS.TXT. You could run the test once, inspect RESULTS.TXT to make sure it contained the expected result PASSED, and save the file as RESULTS.LOG, which will be your baseline file. You could run the same test for every new build of the application and compare the results by using a tool such as WinDiff.Exe—provided with Visual Basic .NET—to compare RESULTS.TXT against RESULTS.LOG. Any difference between the two files should be considered a failure, and a bug should be logged against the failing test.

To create and run automated tests, you can either create your own test system using Visual Basic .NET as shown here or use a commercial-provided testing framework. For example, you could use the free test framework called NUnit, which is designed specifically for creating and running tests for applications written in any .NET language. For more information and a copy of NUnit go to http://sourceforge.net/projects/nunit.

Stress Testing

Stress testing helps you identify problems or vulnerabilities in your application in various high-stress conditions. For example, you can use stress tools to see how your application responds in the following types of situations:

• When little or no memory is available

• When little or no disk space is available

• When multiple threads are executing various parts of your code simultaneously—useful for testing multithreaded components

• When multiple Web requests are made from one Web client or multiple Web clients simultaneously—useful for testing ASP.NET Web applications and Web services

  There are two main benefits to stress testing:

• Stress testing helps to identify security problems with your application. For example, if under a low-disk space condition your Web application returns an error revealing the path and filename of a file the Web application failed to save, an attacker might be able to use this information to gain knowledge of how your files are organized on the server and launch other forms of attack.

• Stress testing helps to identify performance and scalability bottlenecks. By identifying performance and scalability issues, you can make your application more resistant to denial of service (DoS) attacks. Although you often need to use a profiling tool to better understand and fix a performance or scalability problem, stress tools generally only alert you to the problem and don’t give you sufficient information on how to fix it. Profiling tools such as the Microsoft Application Center Test offer features to both stress and profile an application. Look for ACT (Microsoft Application Center Test) in the Visual Studio .NET online help index for more information.

Testing Tools

Test tools allow you to test or look at your application in different ways. Hackers use a variety of tools to create their own blueprint for your application— tools that allow them to obtain all the files that make up your application, the application-file contents, and external resources that your application uses (such as network locations, databases, or Web addresses). In addition, the hacker will use tools—such as input-manipulation, password-cracking, and stress tools—to penetrate your application’s defenses. You can use these same or similar tools to test your own application and ensure that it is secure—or you can at least run the tool (as a hacker would) on your application to see what the tool reveals. Table 9-3 lists some of the available tools.

Table 9-3: Test Tools

Category	Sample Tools	Description
Reverse-engineering to discover application logic	ILDasm, Anakrino, Link, DumpBin, FileMon, Teleport Pro	These are tools used to convert binaries to source or intermediate code to help uncover application logic, application dependencies, and stored secrets.
Web-page manipulation	NetMon, Netcat, Cookie Pal, Achilles	These tools are used to spy on and manipulate raw data such as HTML, HTTP headers, and cookies that are being sent between a client and server.
Network redirection	Netcat	These tools allow a hacker to hijack a network server, bypass firewalls, and give full control of a network computer to the hacker.
Password-cracking	JohnTheRipper, PWDump, LSADump2, L0phtCrack (in various incarnations, such as LC4 provided by @stake)	These tools are useful for cracking operating-system, database, or Web- application user passwords.
Automated Unit Testing	NUnit (unit testing framework for .NET)	These tools provide an environment where you can create and run automated tests.
Stress and profile test tools	Microsoft Application Center Test (ACT), Microsoft SQL Server Profiler, and ANTS, to name a few	These tools are useful for forcing stress failures in your application for the purpose of identifying application vulnerability to various security issues, including denial of service (DoS) and information-discovery attacks.

Create Your Own Test Tools

The beauty of Visual Basic .NET is that it’s a product development tool and test development tool all rolled into one. An advantage you have over developers who use most other programming languages is that you don’t need to look anywhere else to create your own test tools; you can create them in Visual Basic .NET. You can use Visual Basic .NET to create individual tests—known as test cases—as well. Visual Basic’s ease-of-use as a programming language results in it being easy to use as a test tool and test-case development tool. In fact, the Visual Basic .NET Test Team at Microsoft uses Visual Basic .NET to develop test tools for running test scripts, which are also written using Visual Basic .NET.

Example: Create a Test Tool for Testing Web Applications

A sample test application is available with the practice files in the CH09_Testing\WebTester directory. The sample is a simple example of what you can do with Visual Basic .NET to give yourself a hacker’s view of your own application. The WebTester application allows you to view and change hidden fields contained on a Web page. The application is composed of the following components:

• A WebTester Visual Basic .NET Windows client application

• A VBWebAppProxy ASP.NET Web application

• A MyWebApplication ASP.NET Web application that provides a simple Web page to be viewed

MyWebApplication contains a couple of hidden text fields: one is generated by ASP.NET to hold view state, and the other is a custom hidden text field containing the hidden message, “This is my hidden text”. Of course, when you view the page normally using Internet Explorer, you don’t see the hidden fields. The page appears as a normal, run-of-the-mill Web page, as shown in Figure 9-2.

Figure 9-2: The sample test page to be viewed by WebTester

The WebTester client application hosts the Windows Internet Explorer control and gives you a view of your Web page from a hacker’s perspective as shown in Figure 9-4. The WebTester client indirectly obtains the HTML page for MyWebApplication by calling the VBWebAppProxy Web application. The VBWebAppProxy application in turn requests the MyApplication page, processes the raw HTML (for the MyApplication page) by changing the hidden text fields to normal text fields, and changes the colors of the hidden fields for added effect. Figure 9-3 illustrates how the calls are made from the WebTester client to the VBWebAppProxy, which in turn requests the MyWebApplication Web page, processes it, and returns it to the client.

Figure 9-3: Five steps to get a hacker’s view of your Web page

A hacker will use tools similar to the WebTester client application—often command-line tools giving a view of the raw HTML without any UI—to view and change hidden information contained in the page.

As shown in Figure 9-4, the controls with dark backgrounds—displayed as blue on a color screen—are hidden controls. You can change the content of the hidden controls and send the results back to the server.

Figure 9-4: A hacker’s view of your ASP.NET-generated Web page

Run WebTester

You don’t have to take my word on how this works—try it for yourself! The following steps outline how to set up and then run the WebTester application on your test machine.

1. Run Visual Basic .NET, and open WebTester.SLN, which is located in the CH09_Testing\WebTester folder where you installed the practice files.

2. Press F5 to run the application.

3. Click the Browse button, and you should see the same results as shown in Figure 9-4.

Test in the Target Environment

To validate that your application works as expected and holds up to the various attacks you’ve identified in your test plan, you must test the application in the environment where it will be deployed. For example, you might be logged on with administrator privileges when creating the application and trying it out. You will not necessarily know, then, that there is a problem when the application is run by an employee named Betty who is logged on as a normal user. The application running with the operating system privileges of the logged-on normal user account might fail when attempting certain operations such as (mistakenly) writing to a registry key under HKEY_LOCAL_MACHINE that is off limits to everyone except administrative users. This would be an annoyance to Betty because the application would most likely show an error and not function properly. If you ran the application in a nonadministrative environment before giving it to Betty, you might quickly find this problem and determine it is a bug in the code—for example, you might decide that your code should be writing to a registry key under HKEY_CURRENT_USER (accessible to her) instead.

Another situation you need to test for is whether the application grants added permissions to Betty that she should otherwise not have when logged on as a nonadministrator. For example, if the application uses the privileges of an administrative account to save a file—such as a sales report—where the report file name is specified by the user, Betty might inadvertently overwrite a system file or another user’s personal file. By elevating the user’s privileges for certain operations such as saving files, the application opens itself up to a number of security risks. The only way to verify that Betty is restricted to certain activities—such as being able to save files only in folders where she is granted access—is to run the application logged on with the same nonadministrative permissions granted to Betty. You will be able to confirm whether the application elevates permissions and allows you to perform activities—such as the ability to save files in the System folder—that you should not be allowed to perform. An attacker can take advantage of such vulnerabilities by using your application to elevate her own privileges to wreak havoc with your application, database (or other data used by your application), or the operating system.

Make Testing for Security a Priority

If you’re not in the habit of incorporating security in your development or test plans, performing security reviews, or attacking your application like a hacker would, you need to find a way to get there, and quickly. If you don’t make security priority number one on your team, the chances you will do anything about it will be low because you’ll always find an excuse to do something— namely create new features, write code, and fix bugs—other than focus on security.

Adopting a security focus requires a social change in your organization. When it comes to improving security in your product, your focus should be on the people involved, not the security-testing technology employed. It requires raising awareness of the issue, making everyone realize its importance, and specifying each person’s responsibility. Specific steps—such as group-wide security training, a security-focused testing week, and security-focused process changes—are needed to accomplish these goals.