Preparing for Compatibility Testing

Although the amount of preparation needed will vary based on a number of factors, certain steps should be followed in any organization: The scope of the testing should be identified (what's in and what's out), the goals of the testing process should be clarified, and the process should be mapped out.

A significant advantage of following a phased design methodology, as presented in Chapter 2, "Planning, Prototyping, Migrating, and Deploying Exchange Server 2003 Best Practices," is in the planning discussions that take place and in the resulting Statement of Work and Design and Migration documents that are created as deliverables. Often, companies contract messaging experts, who help companies avoid classic mistakes, to assist in the process. By the end of this planning process, it will be very clear why the project is happening, which departments need which features and capabilities, and what budget is available to perform the work. The timeline and key milestones also will be defined.

If a phased discovery and design process hasn't been followed, this information needs to be gathered to ensure that the testing process addresses the goals of the project stakeholders and that the right applications are in fact tested and verified by the appropriate people.

Determining the Scope for Application Testing

At this point in the process, a list should be put together that clarifies which Exchange Server 2003 version is to be used, which version of server software will be used, which add-in features are required, and which third-party applications are needed. As discussed previously, Exchange Server 2003 can be installed on either Windows 2000 Server or Windows Server 2003, and on the Standard or Enterprise versions of the NOS. Smaller companies might choose to use the Standard versions of Windows Server 2003 and Exchange Server 2003, while larger organizations might require the Enterprise versions of each.

A key issue to discuss at this point is whether it is acceptable to have multiple versions of the Windows Server operating system and of Exchange Server in the final solution. Some organizations want to control costs of both software and support services and require a single NOS and single version of Exchange Server. These organizations would rather choose a different messaging application than keep an older application in place that isn't compatible with the newest NOS and version of Exchange.

Besides the core Exchange Server 2003 software, additional components can be installed that extend the functionality of the software, such as Outlook Mobile Access (OMA), Mobile Information Server (MIS), or Live Communications Server (LCS).

NOTE

Although the Standard Edition of Exchange Server 2003 is significantly cheaper than the Enterprise Edition of the license, cost should not be the primary reason for choosing one version over another. It is not as simple to upgrade from the Standard to Enterprise Edition as just changing a software license key. It typically requires setting up a brand new server with the Exchange 2003 Enterprise Edition (on top of Windows Server 2003 Enterprise Edition) and migrating mailboxes from server to server. An organization should seriously consider whether it needs the functionality of the Enterprise Edition before choosing to buy and install the Standard Edition to upgrade easily later.

Third-party applications should be identified as well. The applications most often used include tape backup software modules or agents , antivirus software, fax software, and voicemail integration products. Additional third-party add-on products may include

• Administration

• Antispam

• Backup and storage

• Collaboration

• Customer Relationship Management (CRM)

• Content checking

• Disclaimers

• Email antivirus

• Fax connectors

• List server software

• Log monitoring

• Migration

• POP 3 downloaders

• Reporting

• Security and encryption

• SMS and paging

The hardware to be used should be listed as well, to ensure that it is available when needed. Ideally the exact hardware to be used in the upgrade will be ordered for the application testing process, but if that is not possible, hardware with specifications similar to that of the servers that will eventually be used should be allocated. Although processor speed and amount of RAM will most likely not make a difference to whether the application functions properly on the server platform, certain hardware devices should be as similar as possible. Tape drives , for example, should have the same features as the ones to be used in the production environment, since this is one of the most critical components. If an autoloader will be used in the production environment, one should be made available for the application testing process. If faxing from the Outlook in-box is required, the same faxing hardware should be allocated as well.

Some applications require clients to be present for the testing process, so at least one workstation class system should be available for this purpose. Connectivity to the Internet may also be needed for testing the functionality of remote access products and antivirus software.

A sample checklist of requirements for summarizing the scope of the application testing phase is shown in Table 17.1.

Table 17.1. Checklist for Application Testing

Server #1	Details (include version #s)
Server specs required:
Processor
RAM
Hard drive configuration
Other
Network OS and service packs :
Exchange version and service packs:
Tape backup software version and agents:
Antivirus software and related modules:
Additional third-party apps required:
Additional hardware required:
SAN device
Tape drive
UPS
Switch/hub
Other
Internet access required?	Yes / No

This process should not take a great deal of time if previous planning has taken place. If the planning phase was skipped , some brainstorming will be required to ensure that the scope includes all of the key ingredients required for the application testing. The goals for the application testing process will also affect the scope, which is covered in the following section.

Defining the Goals for Compatibility Testing

As with the previous step of defining the scope of the testing process, defining the goals might be a very quick process, or might require some discussions with the stakeholders involved in the project.

One useful way of looking at the goals for the project is to treat them as the checklist for successful completion of the testing. What conditions need to be met for the organization to confidently move forward with the next step in the Exchange upgrade? The next step might be a more complete prototype testing phase, or for smaller organizations it might be a pilot rollout, where the new messaging environment is offered to a select group of savvy users.

These goals are separate from the business goals the company may have, such as "more reliable messaging infrastructure," or "improved feature set of email client." A more complete prototype phase could seek to address these goals, while the application testing process stays focused on the performance of the specific combinations of operating system, Exchange Server 2003, and imbedded and connected applications.

A convenient way to differentiate the goals of the project is to split them into key areas, as described in the following sections.

Time Frame for Testing

This goal can be defined with the statement "The testing must be completed in X days/weeks."

If there is very little time available to perform the testing, this limits how much time can be spent on each application and how many end-users can put each through its paces. It also necessitates a lesser degree of documentation. Remember to include time for researching the applications' compatibility with the vendors as part of the timeline. A quick project plan might be useful in this process as a way of verifying the assumptions and selling the timeline to the decision- makers .

Estimating the Duration of the Application Testing Process

A good rule of thumb is to allow four hours per application for basic testing, and eight hours for a more thorough testing process. This allows time for the initial research with the vendors, configuration of the NOS and Exchange Server 2003 software, and testing of the applications. Of course the total time required will vary based on the types of applications to be tested.

For example, a Windows Server 2003 system with Exchange Server 2003, tape backup software, antivirus software, fax software, and voicemail connectivity (six total applications) would take an estimated three days to test for basic compatibility and functionality and six days for more rigorous testing.

If another system configuration is to be tested in the same lab ”which has Windows 2000 Server SP3, tape backup software, antivirus software, fax software, and voicemail connectivity ”allocate the same amount of time even though only one component is different: three to six days. Note that if more than one resource is available to perform the testing, these configurations can be tested in parallel, shortening the duration of the process, but not the work effort.

It's always better to have some extra time during the testing phase. This time can be used for more extensive user testing, training, or documentation.

Contingency time should ideally be built in to this goal. Resources assigned to the testing can get sick, or applications might require additional testing when problems are encountered . Vendors might not provide trial versions of the software as quickly as desired, or new versions of software or even the hardware itself can be delayed. With many companies seeking to consolidate the number of servers in use, it is not uncommon to see labs evolve through the testing process. Different versions of the Windows operating system are used, as are different versions of the Exchange Server 2003 software.

Budget for the Testing

This goal can be defined with the statement "The testing must be completed within a budget of $X ."

Of course, there might be no budget allocated for testing, but it's better to know this as soon as possible. A lack of budget means that no new hardware can be ordered, that evaluation copies of the software (both Microsoft and the third-party applications) need to be used, and that no external resources will be brought in. If budget is available or can be accessed in advance of the production upgrade, a subset of the production hardware should be ordered for this phase. Testing on the exact hardware that will be used in the actual upgrade, rather than a cast-off server will yield more valuable results.

Resources to Be Used

This goal can be defined with the statement "The testing will be completed by in-house resources and/or external consultants ."

Often, the internal Exchange Admin staff is too busy with daily tasks or tackling emergencies that spring up (which might be the reason for the upgrade in the first place), and staff personnel should not be expected to dedicate 100% of their time to the testing process.

If an outside consulting firm with expertise in Exchange Server 2003 is going to be used in the testing process, it can be a good leverage point to have already created and decided upon an internal budget for the testing process. This cuts down on the time it takes to debate the approaches from competing firms.

Extent of the Testing

The extent of compatibility testing can be defined with the statement "Each application will be tested for basic, mid-level, or complete compatibility and feature sets."

This goal might be different for different types of applications ”for example, mission-critical applications need extensive testing, whereas less critical applications might have more basic testing performed. A short timeframe with a tightly limited budget won't allow extensive testing, so basic compatibility will most likely be the goal.

Defining the Different Levels of Compatibility Testing

Basic compatibility testing , as used in this chapter, essentially means that the mission-critical applications are tested to verify that they load without errors and perform their primary functions properly with Exchange Server 2003. Often the goal with basic testing is to simply see if the application works, without spending a lot of time or money on hardware and resources, and with a minimum amount of documentation and training. Note that this level of testing reduces but does not eliminate the risks involved in the production roll-out.

Mid-level testing is defined as a process whereby Exchange Server 2003 is configured with all of the applications that will be present in the eventual implementation, so that the test configuration matches the production configuration as closely as possible to reduce the chance of surprise behavior during the rollout. This level of testing requires more preparation to understand the configuration and more involvement from testing resources, and should include end-users. Some training should take place during the process, and documentation is created to record the server configurations and details of the testing process. Although this level of testing greatly reduces the risks of problems during the production migration or upgrade, the migration process of moving data between servers and training the resources on this process hasn't been covered, so some uncertainty still exists.

Complete testing adds additional resource training and possibly end-user training during the process, and should include testing of the actual migration process. Complete training requires more documentation to record the processes required to build or image servers and perform the migration steps. Complete testing is what is typically defined as a prototype phase.

Training Requirements During Testing

This goal can be defined with the statement "Company IT resources will/will not receive training during the application testing process."

Although the IT resources performing the testing will learn a great deal by going through the testing process, the organization might want to provide additional training to these individuals, especially if new functionality and applications are being tested. If external consultants are brought in, it is important that the organization's own resources are still involved in the testing process, for training and validation purposes. The application testing phase might be an excellent time to have helpdesk personnel or departmental managers in the user community learn more about new features that will soon be offered so they can help support the user community and generate excitement for the project.

Documentation Required

This goal can be defined with the statement "Documentation will/will not be generated to summarize the process and results."

Again, the budget and timeline for the testing will affect the answer to this question. Many organizations require a paper trail for all testing procedures, especially when the Exchange infrastructure will have an impact on the viability of the business itself. For other organizations, the messaging environment is not as critical, and less or no documentation may be required.

The application testing phase is a great opportunity to document the steps required for application installations or upgrades if time permits , and this level of instruction can greatly facilitate the production rollout of the upgraded messaging components.

Extent of User Community Involvement

This goal can be defined with the statement "End-users will be included/not included in the testing process."

If there are applications such as Customer Relationship Management (CRM), document routing, voicemail or paging add-ons, or connectivity to PDAs and mobile devices, a higher level of user testing (at least from the power users and Executives) should be considered .

Fate of the Testing Lab

This goal can be defined with the statement "The application testing lab will/will not remain in place after the testing is complete."

There are a number of reasons that organizations decide to keep labs in place after their primary purpose has been served . Whenever a patch or upgrade to Exchange Server 2003 or to a third-party application integrates with Windows Server 2003, it is advisable to test it in a nonproduction environment. Even seemingly innocent patches to antivirus products can crash a production Exchange Server. Other updates might require user testing to see whether they should be rolled out to the production servers. Databases can also be taken offline and copied to the lab for grooming and maintenance. So although this might seem like a trivial question, it is important to clarify at this stage.

Documenting the Compatibility Testing Plan

The information discussed and gathered through the previous exercises needs to be gathered and distributed to the stakeholders to assure that the members of the team are working toward the same goals. These components are the scope and the goals of the application testing process, and should include timeline, budget, extent of the testing (basic, mid-level, complete), training requirements, documentation requirements, and fate of the testing lab. This step is even more important if a formal discovery and design phase was not completed.

By taking the time to document these constraints the testing process will be more structured and less likely to miss a key step or get bogged down on one application. The individuals performing the testing will essentially have a checklist of the exact testing process so are less likely to spend an inordinate amount of time on one application, or "get creative" and try products that are not within the scope of work. Once the testing is complete the stakeholders will also have made it clear what is expected in terms of documentation so the results of the testing can be presented and reviewed efficiently .

This summary document should be presented to the stakeholders of the project for review and approval; then the organization will be ready to proceed with the research and testing process for Exchange Server 2003 compatibility.