Verification and Validation

The concepts behind testing a service are basically the same as those behind testing components, except that an organization typically does not have visibility across the entire application. Interorganizational access to services requires that the service itself be certified without knowing how it is actually used by clients. This means that validation and verification of services is even more difficult and important, because it is used across organizations.

Testing the entire system is especially difficult for interorganizational systems, especially since a published service executes commands that change state. For instance, it is impossible to test the production version of the Withdraw method of the checking account service without actually removing money from an account. The service provider must supply test versions of the service to validate client access to the service.

Testing the QoS of a service is also important. A simple unit or regression test might not find that the service that once took two seconds to execute is now taking 20 seconds to execute. Even though the service is returning the correct result, the consumer might still time out because the request took too long. In addition, the service might work fine with one or two simultaneous requests, but when 20 simultaneous requests are made, the requests may fail the stated QoS for performance.

To thoroughly test services, functional testing, regression testing, load testing, and code inspections are necessary.

Unit Testing

Unit testing verifies all the objects within the service. A unit test is written before or during construction of the class and is used to validate that the class works correctly. Unit testing is usually performed with the support of testing and debugging tools such as JUnit. The same programmers who write the code for the classes write the unit tests.

Functional Testing

Functional testing, sometimes referred to as black box testing, tests the service interfaces, to ensure that they perform as expected. Functional tests test every possible state of a transaction. Invalid data is sent to the service, such as low limits (zero and negative numbers) and high limits. As with the unit test, the goal of the functional test is to provide maximum code coverage, executing all the code in the service.

Functional testing occurs on services that are new or have undergone change. If the functional tests are not time-consuming, they can be placed into the regression test suite in their entirety. Otherwise, a portion of them can be placed into the suite.

Regression Testing

Regression testing, a subset of functional testing, shows that previously passed tests still pass. These tests are performed on all services before the release of any change to any service in the service layer. Full functional testing is performed on services that have changed. Regression testing is performed on those that have not changed but that might be affected by changes in other services.

QoS Testing

QoS testing tests scalability, performance, and reliability. Load testing, one component of the QoS test suite, executes multiple requests of the same type at the same time, to guarantee a basic level of service under a specific system load. Performance testing finds the time it takes to execute a single service request. Reliability testing finds any correctness problems that might occur. For instance, these tests identify a potential for file corruption because two threads update a file at the same time. Reliability testing also verifies that the system can recover after a disaster.

Service Federation Testing

Service federation testing tests that multiple services work correctly together in a federation. For instance, consider a request that requires both the checking account service and customer service. The request requires that both these services communicate with each other and that both can roll back in the event of failure. Tests built to check for service-to-service coordination and communication are service federation tests.

System Tests

System tests include the clients in the testing process. This is the most difficult type of testing to perform in a service-oriented architecture, because the organization that provides the services is usually not the organization that consumes them. This means that multiple organizations must coordinate their testing efforts, to ensure that the service's clients can access it. If the consumer organization is not willing to participate in the tests or if the number of clients is too vast to perform full system tests, the service must go through a much more rigorous certification testing prior to release to ensure that it works correctly.

Inspections

Code inspections are good at finding defects of maintainability, reliability, and functionality. A code inspection checklist can find simple errors of omission and commission. An omission defect is when functionality is missing from the code that should have been there. A commission defect is when the code performs a function incorrectly. A maintainability defect is one that obviously negatively impacts the service's maintainability. This might be a defect in which a hard-coded value would be better stored in a file and read at runtime.

A reliability defect is one that might affect the running of the system. Reliability defects are best found by code inspections. A functionality defect is one in which the behavior of the service does not meet the requirements for the service. For example, if a service method was stubbed out during development and never implemented, it could pass all the functional tests, because it returns correct data, but it always returns the same data. Defects of this type are identified easily by code inspections.