Acceptance Testing | Solaris System Management (New Riders Professional Library)

Acceptance testing is the final phase of testing before going live. It is a crucial process and eventually will determine whether a product is acceptable to the business.

When a product is deemed to be acceptable, it progresses through to implementation and becomes operational, but what happens if either the user acceptance or the system acceptance tests fail?

During acceptance testing, problem logs are created when either a bug is found or the application responds in an unexpected manner. The logs are passed to the developer(s) for resolution. An example of a problem log is shown in Figure 4.1. The tester often provides additional information by attaching a screen dump showing clearly the point at which the error occurs. Sometimes the resolution is procedural and may require only a trivial update to the documentation, but on other occasions, a fix may have to be supplied. For the latter, a patch is issued and installed on the system. However, it's not that simple because the fix could invalidate all the testing already done.

Figure 4.1. The problem log represents a formal document of the testing phase with a unique reference. It is often stored in a database using a program such as PVCS Tracker, allowing further analysis to be undertaken.

$graphics\04fig01.gif$

Consider the following example: While testing the application, a user receives the wrong results for a query operation. A problem log is raised and passed to the developers to fix. A patch is delivered containing the fixed module. The query module is shared by several forms and is also used to gather the information for the management and financial reports .

Clearly, in this example, the impact of the change is much wider than just the query being tested . The point is that any changes to a system or application under review must include a detailed impact assessment so that its effect is known and understood . In the example, the change is fundamental to the system and may require the testing to be restarted from the beginning. In reality, testing would probably be completed (without installing the patch) so that any other problems can be addressed before rerunning the tests.

Regression Testing

Regression testing is used as a mechanism of ensuring that there have been no unexpected side effects of a change. It allows tests to be rerun, knowing exactly what the result should be, thereby highlighting immediately any deviation. For example, consider an application that calculates the payroll for the employees of the company. Suppose that a new application is being developed to replace the existing one, which cannot handle the increased number of employees since the company expanded.

Using the existing system for calculating the payroll, a test would be devised, using specific test data (say, a dozen employees), and their payroll details calculated. When the new system is installed and tested, the same calculations would be carried out, using the same test data. The results should be exactly the same ”if not, then the new system is not functioning correctly.

For an application such as payroll, despite the fact that a new way of doing things has been developed, maybe using a new GUI front end, the functionality ”that is, the results of the calculations ”should be the same. Regression testing is used for this purpose, to be able to verify that the application still performs as expected. Without the benefit of regression testing, it would be extremely difficult to identify the effects of any changes on other parts of the application, leaving the potential for problems to be found only after the system is live.

System Testing

System testing involves testing all the features of a new application. The IT department carries out the system acceptance and fully tests the functionality against an agreed criteria, usually the functional specification. System testing differs from user acceptance testing in that the management and configuration of the application or system is tested in addition to the functionality.

Some of the types of tests are listed here:

Installation and configuration procedures, including a QA of the administration and installation documentation.
Creation and management of user accounts, and management of passwords (if applicable ) and permission assignments. For example, a user may be assigned a specific role that will require different permissions, such as query only, query and update, or full administrator.
Process of exercising each option that is available within the application to ensure that it works as expected. For GUI applications, command-line equivalents should be available for the core functionality of the application so that, for example, GUI sessions are not sent through firewalls when executing the application remotely across the network.
Configuration of printing from within the application. An example of this is the addition of a new printer or a printer queue.
Entering of invalid data to check that the error handling functions as expected. The testing looks for errors, so attempts will obviously be made to make the application or system crash.
Volume testing, to ensure that the system can work under conditions very close to that of the live environment.
Testing of the security of the system, as part of its security accreditation. The results of the security tests provide evidence of compliance (or otherwise ) with the security policy in force.
Performance of recovery testing, which is necessary particularly in larger systems involving hardware and software deployment. In this instance, the system is deliberately placed in a variety of fault conditions to establish how tolerant it is and how quickly processing can be resumed. For example, in a RAID storage configuration, a good test is to remove one of the disk modules (to simulate a disk failure) and check that a hot spare takes over the role of the failed module.

User Acceptance

The user community must have the opportunity to test the application before implementation. For larger projects, a user assurance coordinator (UAC) should be assigned to the project, normally from within the IT department. As a system manager, I have fulfilled this role. The UAC looks after the users' interests to ensure that the application meets their expectations. UAT should be carried out on a dedicated test environment in which changes to the application cannot be made without being carefully controlled and documented. At this stage of the process, changes are acceptable only when delivered as patches, and these would be as a result of problem logs being created by the UAT team.

This stage of testing is crucial to a project being successful for the following reasons:

The users run the system or the application as it will be used in the live environment, and exercise it fully so that any deficiencies can be identified as soon as possible. These problems subsequently are rectified before the final delivery. This includes the users entering invalid data, such as bad dates or reference numbers that don't conform to the standard, to check that they will be rejected. The user will determine whether the application will be accepted on behalf of the business. Normally a senior user representative will be a member of the project board.
The UAC role will be heavily involved in the creation of the acceptance test criteria, the associated acceptance test specifications, and test scripts. These are the documents that will be used during UAT execution. The test scripts that are produced should form a logical sequence that reflects the normal operational usage.

Recovery Testing

Recovery tests can sometimes be extremely difficult to accomplish, particularly with sealed units that can't be accessed directly. Certain tests likely may need the cooperation of the vendor so that the required scenarios can be fully tested.

UAT enables the user community to see the application running in its final state. It is an ideal opportunity to build user confidence in the product. This is especially vital if the application is new. People are naturally hostile to change, so the UAT provides the chance to gain familiarity and to see the benefits that the new product will offer. Conversely, a badly designed application or a poorly performing system can destroy user confidence. If this happens, then it will be extremely hard to get it accepted.
UAT will prove whether the user documentation is accurate and usable.
The users test the user interface as well as the application itself. This is particularly useful for GUI-based applications, in which color schemes and screen layouts are scrutinized.
Test cases are often executed by more than one person. The same result should be produced each time.

Performance Testing

The other areas of testing discussed so far verify, as much as possible, that the application functions as expected. Performance testing looks at a further aspect that is sometimes less obvious: the testing of the application when under load, or busy, conditions. Performance testing has become even more important since businesses started using the Internet, particularly with reference to Web servers. There might be a point at which the application, or the server, crashes if too many people try to use it at the same time, so the performance capability must be tested so that the anticipated demand can be met. It is extremely difficult to accurately predict the level of demand that a new Web site will generate ”companies are often victims of their own success. For example, when an advertising campaign has much greater success than predicted , the increase in demand can be more than the system can handle.

Producing test conditions and data for performance testing can be a time-consuming and expensive business, requiring a great deal of coordination. It often involves creating large amounts of test data, as well as having the relevant technical staff in attendance to accurately monitor the performance as the testing is carried out. From the system manager's point of view, however, the testing can be of great benefit because it might provide firm justification for upgrading a server that was proved to be inadequate, or the purchase of a new, more powerful machine.

Top