Preparing for Compatibility Testing

Although the amount of preparation needed will vary based on a number of factors, certain steps should be followed in any organizationthe scope of the testing should be identified (what's in and what's out), the goals of the testing process should be clarified, and the process should be mapped out.

A significant advantage of following a phased design methodology, as presented in Chapter 2, "Planning, Prototyping, Migrating, and Deploying Windows Server 2003 Best Practices," is in the planning discussions that take place and in the resulting statements of work, design, and migration documents that are created as deliverables. Often, companies' contract with migration experts to help companies avoid classic mistakes in the upgrade process. By the end of this planning process, it will be very clear why the project is happening, which departments need which features and capabilities, and what budget is available to perform the work. The timeline and key milestones also will be defined.

If a phased discovery and design process hasn't been followed, this information needs to be gathered to ensure that the testing process addresses the goals of the project stakeholders, and that the right applications are in fact tested and verified by the appropriate people.

Determining the Scope for Application Testing

At this point in the process, a list should be put together that clarifies which Windows Server 2003 version is to be used, which version of server software will be used, which add-in features are required, and which third-party applications are needed. As discussed previously, Windows Server 2003 comes in Web, Standard, Enterprise, and Datacenter versions. Smaller companies may choose to use the Standard versions of Windows Server 2003 operating system, whereas larger organizations might require the Enterprise version on their server systems for more advanced scalability and fault tolerance.

A key issue to discuss at this point is whether it is acceptable to have multiple versions of the Windows Server operating system in the final solution. Some organizations want to control standards on both software and support services, and require just a single network operating system.

Note

Although the Standard Edition of Windows Server 2003 is significantly cheaper than the Enterprise Edition of the license, cost should not be the primary reason for choosing one version over another. It is not as simple to upgrade from the Standard to Enterprise Edition as just changing a software license key. It requires either setting up a brand-new server with the Windows 2003 Enterprise Edition and migrating applications from server to server, or a full upgrade of the Enterprise Edition over an existing Standard Edition license. An organization should seriously consider whether it needs the functionality of the Enterprise Edition before choosing to buy and install the Standard Edition and attempting to upgrade later.

Third-party applications should be identified as well. The applications most often used include tape-backup software modules or agents, antivirus software, fax software, and voicemail integration products. Additional third-party add-on products might include the following:

• Administration

• Antispam

• Backup and storage

• Customer Relationship Management (CRM)

• Log monitoring

• Migration

• Reporting

• Security and encryption

The hardware to be used should be listed as well, to ensure that it is available when needed. Ideally the exact hardware to be used in the upgrade will be ordered for the application testing process, but if that is not possible, hardware with specifications similar to that of the servers that will eventually be used should be allocated. Although processor speed and amount of RAM will most likely not make a difference to whether the application functions properly on the server platform, certain hardware devices should be as similar as possible. Tape drives, for example, should have the same features as the ones to be used in the production environment, because this is one of the most critical components. If an autoloader will be used in the production environment, one should be made available for the application testing process. If faxing from the Outlook inbox is required, the same faxing hardware should be allocated as well.

Some applications require clients to be present for the testing process, so at least one workstation class system should be available for this purpose. Connectivity to the Internet might also be necessary for testing the functionality of remote access products and antivirus software.

A sample checklist of requirements for summarizing the scope of the application testing phase is shown in Table 18.1.

Table 18.1. Checklist for Application Testing

Server #1	Details (include version #s)
Server specs required:
Processor RAM Hard drive configuration Other
Network OS and service packs:
Tape backup software version and agents:
Additional third-party apps required:
Additional hardware required:
SAN device Tape drive UPS Switch/hub Other
Internet access required?	Yes/No

This process should not take a great deal of time if previous planning has taken place. If the planning phase was skipped, some brainstorming will be required to ensure that the scope includes all the key ingredients required for the application testing. The goals for the application testing process will also affect the scope, which is covered in the following section.

Defining the Goals for Compatibility Testing

As with the previous step of defining the scope of the testing process, defining the goals might be a very quick process, or could require some discussions with the stakeholders involved in the project.

One useful way of looking at the goals for the project is to treat them as the checklist for successful completion of the testing. What conditions need to be met for the organization to confidently move forward with the next step in the Windows migration? The next step might be a more complete prototype testing phase. For smaller organizations, it might be a pilot rollout, where the new networking environment is offered to a select group of savvy users.

These goals are separate from the business goals the company might have, such as a more reliable network infrastructure or improved security. A more complete prototype phase could seek to address these goals while the application testing process stays focused on the performance of the specific combinations of the operating system and embedded and connected applications.

A convenient way to differentiate the goals of the project is to split them into key areas, as described in the following sections.

Timeframe for Testing

This goal can be defined with the statement "The testing must be completed in X days/weeks."

If there is very little time available to perform the testing, this limits how much time can be spent on each application and how many endusers can put each through its paces. It also necessitates a lesser degree of documentation. Remember to include time for researching the applications' compatibility with the vendors as part of the timeline. A quick project plan might be useful in this process as a way of verifying the assumptions and selling the timeline to the decisionmakers.

Estimating the Duration of the Application Testing Process

A good rule of thumb is to allow four hours per application to be tested for basic testing, and eight hours for a more thorough testing process. This allows time for the initial research with the vendors, configuration of the Windows 2003 operating system, and testing of the applications. Of course, the total time required will vary based on the types of applications to be tested.

For example, a Windows Server 2003 system with tape backup software and accounting software would take an estimated one or two days to test for basic compatibility and functionality, and potentially a week for more rigorous testing.

Note that if more than one resource is available to perform the testing, these configurations can be tested in parallel, shortening the duration of the process, but not the work effort.

It's always better to have some extra time during the testing phase. This time can be used for more extensive user testing, training, or documentation.

Contingency time should ideally be built into this goal. Resources assigned to the testing can get sick, or applications might require additional testing when problems are encountered. Vendors might not provide trial versions of the software as quickly as desired, or new versions of software or even the hardware itself can be delayed. With many companies seeking to consolidate the number of servers in use, it is not uncommon to see labs evolve through the testing process. Different versions of the Windows operating system are used, as are different versions of various application software programs.

Budget for the Testing

This goal can be defined with the statement "The testing must be completed within a budget of $X."

Of course, there might be no budget allocated for testing, but it's better to know this as soon as possible. A lack of budget means that no new hardware can be ordered, evaluation copies of the software (both Microsoft and the third-party applications) need to be used, and no external resources will be brought in. If the budget is available or can be accessed in advance of the production upgrade, a subset of the production hardware should be ordered for this phase. Testing on the exact hardware that will be used in the actual upgrade rather than a cast-off server will yield more valuable results.

Resources to Be Used

This goal can be defined with the statement "The testing will be completed by in-house resources and/or external consultants."

Often, the internal network administration staff is too busy with daily tasks or tackling emergencies that spring up (which might be the reason for the upgrade in the first place), and staff personnel should not be expected to dedicate 100% of their time to the testing process.

If an outside consulting firm with expertise in Windows Server 2003 is going to be used in the testing process, it can be a good leverage point to have already created and decided upon an internal budget for the testing process. This cuts down on the time it takes to debate the approaches from competing firms.

Extent of the Testing

The extent of compatibility testing can be defined with the statement "Each application will be tested for basic, mid-level, or complete compatibility and feature sets."

This goal might be set for different types of applications where some mission-critical applications would need to have extensive testing, whereas less critical applications might have more basic testing performed. A short timeframe with a tightly limited budget won't allow extensive testing, so basic compatibility will most likely be the goal.

Defining the Different Levels of Compatibility Testing

Basic compatibility testing, as used in this chapter, essentially means that the mission-critical applications are tested to verify that they load without errors and perform their primary functions properly with Windows Server 2003. Often the goal with basic testing is to simply see whether the application works, without spending a lot of time or money on hardware and resources, and with a minimum amount of documentation and training. Note that this level of testing reduces but does not eliminate the risks involved in the production rollout.

Mid-level testing is defined as a process whereby Windows Server 2003 is configured with all the applications that will be present in the eventual implementation, so that the test configuration matches the production configuration as closely as possible to reduce the chance of surprise behavior during the rollout. This level of testing requires more preparation to understand the configuration and more involvement from testing resources, and should include endusers. Some training should take place during the process, and documentation is created to record the server configurations and details of the testing process. Although this level of testing greatly reduces the risks of problems during the production migration or upgrade, the migration process of moving data between servers and training the resources on this process hasn't been covered, so some uncertainty still exists.

Complete testing adds additional resource training and possibly end-user training during the process, and should include testing of the actual migration process. Complete training requires more documentation to record the processes required to build or image servers and perform the migration steps. Complete testing is what is typically defined as a prototype phase.

Training Requirements During Testing

This goal can be defined with the statement "Company IT resources will/will not receive training during the application testing process."

Although the IT resources performing the testing will learn a great deal by going through the testing process, the organization might want to provide additional training to these individuals, especially if new functionality and applications are being tested. If external consultants are brought in, it is important that the organization's own resources are still involved in the testing process for training and validation purposes. The application testing phase might be an excellent time to have help desk personnel or departmental managers in the user community learn more about new features that will soon be offered so they can help support the user community and generate excitement for the project.

Documentation Required

This goal can be defined with the statement "Documentation will/will not be generated to summarize the process and results."

Again, the budget and timeline for the testing will affect the answer to this question. Many organizations require a paper trail for all testing procedures, especially when the Windows infrastructure will have an impact on the viability of the business itself. For other organizations, the networking environment is not as critical, and less or no documentation may be required.

The application testing phase is a great opportunity to document the steps required for application installations or upgrades if time permits, and this level of instruction can greatly facilitate the production rollout of the upgraded networking components.

Extent of User Community Involvement

This goal can be defined with the statement "Endusers will be included/not included in the testing process."

If there are applications such as Customer Relationship Management (CRM), document routing, voicemail or paging add-ons, or connectivity to PDAs and mobile devices, a higher level of user testing (at least from the power users and executives) should be considered.

Fate of the Testing Lab

This goal can be defined with the statement "The application testing lab will/will not remain in place after the testing is complete."

There are a number of reasons that organizations decide to keep labs in place after their primary purpose has been served. Whenever a patch or upgrade to Windows Server 2003 or to a third-party application integrates with Windows Server 2003, it is advisable to test it in a nonproduction environment. Even seemingly innocent patches to antivirus products can crash a production server. Other updates might require user testing to see whether they should be rolled out to the production servers.

Documenting the Compatibility Testing Plan

The information discussed and gathered through the previous exercises needs to be gathered and distributed to the stakeholders to assure that the members of the team are working toward the same goals. These components are the scope and the goals of the application testing process, and should include timeline, budget, extent of the testing (basic, mid-level, complete), training requirements, documentation requirements, and the fate of the testing lab. This step is even more important if a formal discovery and design phase was not completed.

By taking the time to document these constraints, the testing process will be more structured and less likely to miss a key step or get bogged down on one application. The individuals performing the testing will essentially have a checklist of the exact testing process, and are less likely to spend an inordinate amount of time on one application, or "get creative" and try products that are not within the scope of work. After the testing is complete, the stakeholders will also have made it clear what is expected in terms of documentation so the results of the testing can be presented and reviewed efficiently.

This summary document should be presented to the stakeholders of the project for review and approval. The organization will then be ready to proceed with the research and testing process for Windows Server 2003 compatibility.