Project Tasks in the Transition Phase

As the project heads into the Transition phase, a number of development tasks remain. These include deploying the IOC, online help, installation scripts, final change requests, and data migration. Other tasks, such as training and acceptance testing, remain as well. I will discuss each of these separately.

Deploying the IOC

When the IOC milestone is reached and the Transition phase begins, the goal is to make the deployment of the new system as painless and smooth as possible. A major risk-reduction activity during the Transition phase is an early deployment of the IOC. This is often called a beta release. The following summarizes the goals and purposes of deploying a beta release:

• The primary purpose of early deployment is risk reduction. Early deployment provides the opportunity to discover problems that may not be found in the development environment. Issues relating to the network environment, operating system differences, hardware differences, and so on are just some of the variables encountered when moving from one environment to another. Early deployment means early detection of unexpected problems, which lets you detect and correct the problem before users depend on the system for their work.

• Early deployment allows the users to gain experience and familiarity with the system, which increases the likelihood of user acceptance.

• Early deployment permits the users to verify that the system meets their functional needs. If it is discovered that certain functionality is incorrect or missing, some time is still available to correct it before the system is placed into full production.

Although these are clearly important advantages, early deployment as a beta release does present some risks. If you plan an early deployment, be careful to monitor and plan to mitigate the following risks:

• The release must be used and exercised. Since early deployments are not "full" deployments, meaning the users do not yet depend on the product for their jobs, the users may not have a sense of urgency to try the system. In other words, using the system requires a time commitment from the users. To mitigate this risk, you should select a designated pool of interested users, which represents a subset of the user community, to participate in the testing. Preferably, they should be users who have expressed continued interest in the project and who together will exercise most of the product's functionality. The contractor should meet with the outsourcing organization's management and stress the importance of the early deployment as a risk-reduction activity that is vital to a smooth transition to production status. The customer should make sure the users have time allocated in their schedules for this activity.

• The customer may not understand and appreciate the importance of the early deployment. Be sure to explain the purpose and goals of the testing. The focus should be on finding last-minute bugs and to be sure the existing requirements have been interpreted correctly.

• Be sure to document and present to the users any functionality that is not yet implemented, such as help files and installation scripts. You do not want the customer to spend time looking for functionality that is known to be missing. It is also important to set their expectations up front by informing them of any features not yet ready.

• One risk that occasionally surfaces is users getting carried away and flooding the project team with change requests. These need to be evaluated carefully. Certainly, if the change requests pertain to valid bugs in the system, they should be corrected as soon as possible. But if the change requests are actually enhancement requests or new requirement requests, try to resist implementing them at this time unless they are absolutely required.

• Try to avoid risking production data during the early deployment. Until proper system behavior is proven, don't run the risk of disrupting the customer's business until the system is ready for full production. This may mean if the new system is replacing a legacy system, both systems will need to run in parallel for a time, until people gain confidence in the new system. Of course, if a separate copy of the production data were available, this would be ideal, as long as the users are aware that any permanent changes should still be made in the legacy system. Use of a separate copy of the legacy data should increase interest in using the early deployment, because users will be better able to observe the system behavior using data with which they are familiar.

Despite the challenges, early deployment testing is an important risk-reduction activity that should be performed.

Online Help

The first step in producing online help is to examine your system's requirements. Do the requirements specifically ask for online help? If so, what do they state? Even if you have no requirements for online help, the reality is that most users expect some form of online help to guide them in their usage of the system. Hopefully, the analysts on the project have uncovered and documented these requirements. Accordingly, the creation of online help would then be scheduled for an iteration in the Transition phase.

The Iteration phase is not the appropriate place to discover for the first time the requirements for online help. Although "online help" sounds simple enough, it can range from simply making an online copy of a user manual available from the product's user interface, to a fully functional tutorial involving sample exercises and canned demonstrations illustrating how the user interacts with and uses the product. Creating online help can, therefore, be a major task and should be planned in an iteration just like other major portions of functionality. Remember that online help is likely to be heavily used initially, until the end users become comfortable using the product. As discussed previously, this feature is highly visible to end users and will shape their experience with the product.

Installation Scripts

Installation scripts are another example of a task that is conceptually simple yet can be extremely complex to implement. Here are some considerations for determining how to structure the installation procedures:

• Who will be installing the product? If the product is a Web-based enterprise system, chances are the system administrators will perform the installation. The installation may be a one-time event, or it may take place only occasionally. If the system is a client/server-based system, the product may need to be installed by each client. If this is the case, the installation may be performed by less-knowledgeable end users, and the installation process will be repeated frequently. In the former case, it may be reasonable to make the installation less automated and involve some manual steps. In the latter case, this may be totally unacceptable. The installation process for system administrators could be a relatively simple installation procedure involving copying files into place. The installation process for client users may have to be completely automated.

• For installation procedures that are performed frequently, such as for client/server systems, what is the consistency of the end-user environment? For example, will all clients have the same kind of PC, with the same operating system, or will there be a lot of variation in the environment from client to client? If there is a lot of variation, the installation procedure should check whether the PC has enough disk space, memory, and processor speed before proceeding with the installation. Also, the installation procedure itself may have to vary, depending on the client's exact version of the operating system.

• It is strongly recommended for client workstation installations that any automated installation scripts and procedures be tested on several typical client machines in the actual environment. An infinite number of unknowns relate to the environment and are difficult to discover or anticipate. Testing in the actual environment is the only way to ensure proper operation.

Change Requests

Up until the Transition phase, the team has focused primarily on high-priority change requests. This means that lower-priority change requests, such as typographical errors and minor formatting changes on the user interface, have been accumulating. The Transition phase is the time to complete as many of these lower-priority requests as possible. These change requests are often items very visible to end users. An investment of time to correct these change requests will pay dividends in increased customer satisfaction.

Exercise caution in expending resources on change requests involving requirements that significantly alter the system's behavior or its business rules. Chances are when the system is fielded for the first time, there may be high-priority change requests, which are much higher in importance. The focus in Transition should be to make the system rollout as successful as possible, which means that a stable, bug-free release is the goal.

Data Migration

Data migration is another example of a task that is conceptually simple but that in practice is complex and tedious. Simply stated, the purpose of data migration is to move data that currently resides in legacy systems into a new system. In some cases, more than one legacy system may exist, such as when a newly developed system replaces multiple legacy systems. In most cases, this data represents core business knowledge or information that must be preserved. Often, this data represents an asset that has been built and accumulated over many years. As a consequence, the organization places a very high value on this data, and its integrity must be preserved when it is migrated to the new system.

Overview of the Steps in Migrating Data

Data migration involves a series of steps. These are summarized next and in Figure 12-1, followed by a detailed discussion of each step.

Figure 12-1. Overview of data migration steps

1.	The data migration process is planned and prepared.
2.	The legacy systems (the source of the data being migrated) must be shut down so that the data does not change during the migration. As a result, neither the legacy systems nor the new system can be used during this process. Therefore, data migration usually occurs over weekends or holidays.
3.	The data is extracted, or exported, from the legacy system. The data is stored in files so that it can be viewed and possibly edited.
4.	The data is examined to search for corrupt data and duplicate data and to make corrections to the data. This is known as data cleansing.
5.	The data is analyzed to determine to which fields in the new destination system the data should be mapped.
6.	Scripts are written to automate the transfer of the legacy data from the extracted, cleansed files into the destination database.
7.	Testing is performed to verify the accuracy of the data migration process.

Details of Data Migration Steps

As just mentioned, data migration is a tedious process. The key to success in a data migration effort is careful planning and starting long before the "official" migration begins. Because of the importance of this activity, I will discuss each step in detail.

Planning a Successful Data Migration Effort

Data migration by itself is a significant project. Like any other project, it requires resources, careful planning, and the setting of client expectations. It also involves significant risks that must be mitigated. The following are important points to consider when planning a data migration effort:

• Although the steps just listed are accurate, it is vital that the planning begin during the project's Inception phase. "Planning" includes conducting at least one, and possibly several, trial runs of analyzing and migrating the data before placing the new system into production with the migrated data. This is an important risk-reduction exercise that must not be omitted. I have yet to see a data migration effort proceed without unexpected issues arising. In most cases, these issues were unforeseen and would not have been discovered without conducting trial migration efforts.

• In general, the schedule of data migration proceeds in parallel with the effort to develop the new system. But there are some points where data migration can help you develop the new system. For example, analyzing the legacy data may lead you to discover aspects of functionality not identified in the new system. For example, if a field of data in the cleansed data set cannot be mapped to a field maintained by the new system, functionality is missing in the new system. Either the functionality should be analyzed and developed in the new system, or the extraneous data should be eliminated from the cleansed data set.

• Results from successful trial runs of migrated data can be useful to test the new system with actual data that will ultimately be used when the system is placed into production. This can save the test team time when creating sample data for testing.

• Client and user expectations must be managed carefully. The client and users often do not understand the complexity of the data migration effort. It is also vital that users familiar with the legacy data be quickly available, or on call, during the data cleansing effort. During the data cleansing process, questionable data records will be discovered. This requires the assistance of users knowledgeable about the data so that a proper decision can be made on the disposition of the data.

• During the data migration effort, any problems noted in system testing using the legacy data should be captured and processed, just as with all change requests.

Shutting Down Legacy Systems to Extract Data

During off-hours, or during maintenance, a recent snapshot of the data is sufficient in most cases for trial runs. Of course, for the final migration, the legacy system needs to remain down so that changes will not be made to the data during the switchover to the new system. Depending on the use of the system, this may require months of advance notice, especially for a mission-critical system. It may also be necessary to conduct the final migration over a holiday weekend. Prepare the staff conducting the data migration accordingly. If several trial runs using recent snapshots of the legacy system have been conducted, and the system has been tested with this data, there is an excellent chance that the migration effort will proceed successfully. However, be prepared to place the legacy system back into production if the migration effort fails. The goal, of course, is to place the new system into production; however, preserve the option of returning the legacy system into production as a fallback position.

As mentioned previously, trial runs of the data migration are mandatory. Another reason for conducting trial runs is that it helps produce a reliable estimate of the total amount of time the final migration effort will take. This is key, because neither the legacy system nor the new system will be available during the final migration effort. Because this is not a trivial situation for most organizations, it is vital to know ahead of time how much time needs to be allocated for the final migration.

Extracting the Legacy Data

Depending on the type of system in which the legacy data resides, methods for extracting the data will vary. If possible, automated scripts should be developed for this purpose, because extraction of the data will probably occur multiple times. The data should be stored in files that can be viewed and examined. Note that depending on the customer and the application, significant security concerns might be involved. Be sure to discuss any concerns about security and access to the data during the migration process with the customer ahead of time. Addressing these concerns beforehand minimizes disruptions during the process.

Depending on the amount and types of data stored in the legacy database, it may be necessary to extract the data in phases.

Data Cleansing

Data cleansing involves a thorough check of the data extracted from the legacy system. The data is modified when data meeting certain attributes is located. Examples include eliminating duplicate data, deleting obsolete or unused data, and even repairing corrupted data records. Most of this effort is done from the perspective of the legacy system. In other words, no consideration is given to the ultimate destination of the final migrated data. It is simply a "cleanup" effort that yields a "purified" set of data that still works perfectly in the legacy system. You must perform this step in close collaboration with users who are familiar with the data.

Sometimes, depending on circumstances, it may make sense to perform the cleansing step before you extract the data from the legacy system. The point is to examine all data records and purge any that are unnecessary.

Analyzing the Cleansed Data

Analyzing the cleansed data is perhaps the most painstaking step of the data migration effort. Each field of data must be traced to a corresponding field in the destination database. If a data model has been created, it will become very useful in this step. During this step, the data is also analyzed for consistency.

One example of inconsistent data is illustrated in the following example, which occurred on an actual project. A Web-based system was developed that was to contain a centralized database with a Web interface that could be accessed over the organization's internal network, which reached nationwide. The system was replacing a former client/server system in which the user community was divided into separate regions. Each region contained its own database. The format and layout of the database were the same for each database across all the regions. These databases were to be consolidated into a single database for the new system. The issue happened over the disposition of a field indicating which region the data belonged to. It turned out that two different regions were using the same value to indicate the region they belonged to. Previously, this had not caused any problems, because each region had its own separate database. But when consolidating the databases, it was definitely a problem. There was no way to distinguish between the two regions after the data was consolidated.

This is a good example of data consistency. In this example, the data in both databases is perfectly valid. But during the analysis, the problem is discovered. In this situation, the solution, of course, was to change the value for one of the regions so that each region was using a unique value.

Another scenario that requires extra diligence frequently occurs when you replace a legacy system that has been running for many years. In one case, analyzing the cleansed data revealed inconsistencies in the format of the data records prior to a certain date. It turned out that the legacy system being replaced had previously replaced yet another legacy system. This occurred before most of the users joined the organization, so no one recalled this event. The lesson learned was to be sure to investigate each instance of the data that does not conform to the expected parameters.

Yet another example encountered on a project involved replacing a mainframe-based legacy system. In this system, the database contained a customer name. The mainframe database defined a single string to contain the customer's entire name. The new system defined separate fields for each component in the customer name. The data migration effort involved writing scripts to parse the string from the legacy system to place it in separate fields in the new database.

These are typical examples of issues that can arise. Of course, each migration effort is different. Be prepared to discover these sorts of scenarios.

Transferring the Legacy Data

For simple systems, cleansed, analyzed data can be imported directly into the final destination database. For most systems, the data's complexity may not permit this. In addition, consider the original system managing the legacy data. If the legacy system uses an earlier release of Oracle, and the new system also uses Oracle (albeit a newer release), you are less likely to run into data representation problems. On the other hand, if the legacy system is a database management system from a different vendor running on a completely different type of system, you should consider setting up an intermediate database to "stage" the data. For example, suppose the legacy system runs on a mainframe system running MVS and the new database system runs on UNIX. In this example, you are much more likely to run into data representation problems than in the Oracle example. Two significant variables are involved: the change in platform and the change in database vendors.

Figure 12-1 showed a staging database being used to assemble the data after it is cleansed. Each extracted table can be individually imported into the staging database, which is created on the same platform and using the same database vendor and version as the final production system. After it is loaded into the database, the data is examined to ensure that it is consistent and that the new system will interpret it correctly.

Automated scripts and utilities are recommended to load the data into the destination databases. This allows multiple runs for importing the data, which is necessary to ensure a smooth migration effort when the final migration to production occurs.

Testing the Migrated Data

At each step, the data should be examined for consistency and to ensure that it has no corrupted records or missing data. The scripts that perform the data transfer must also be tested, like any other developed code.

When the data is transferred to the final database, the database engineer should meet with a user or group of users who are familiar with the data. Together, they should spot-check several meaningful scenarios to gain confidence in the migration's success and to ensure readiness for production.

Points to Remember for the Data Migration Effort

You should now appreciate the complexity of most data migration efforts. Remember the following key points to mitigate risks during data migration:

• The database engineer performing the migration needs very fast turnaround time for answers from users concerning issues found during data cleansing and consistency checks.

• Manage user expectations carefully. Most users do not appreciate the challenges involved in the migration effort.

• Expect to encounter corrupted data, missing data, and inconsistent data. Each instance must be pursued to resolution.

• Start the migration effort as early as possible. Be sure the staff working on the migration effort communicates with the analysts and testers when developing the new system.

• Be prepared to expect the unexpected. The schedule for the migration effort should include extra time to accommodate unexpected and unforeseen problems.

Following and heeding the steps described here should ensure that your data migration effort is successful.

Training

Most projects that produce a completely new system also require training to be developed. Training development and delivery warrants its own book, so it isn't discussed here. You should refer to the many excellent books on this topic. However, you should consider some ideas specifically in the context of delivering a new software system:

• Training should involve hands-on interaction with the product. Most users simply want to be comfortable using the system to accomplish their jobs.

• Consider having separate training for administrators, users, and system maintainers. The needs and focus of each of these groups are sufficiently different to warrant separate training for each.

• For the general user population, consider having one of the project analysts present during training. The trainer should be knowledgeable about how the new product operates, but it is equally important to have someone who can tie the product's functionality to the business requirements and business process. The project requirements analysts can provide this perspective.

• To train the system's administrators and maintainers, consider having someone from the development group there to field in-depth technical questions that may arise during training.

• Be aware that you may receive questions and comments during the training about functionality that was expected in the product but not delivered. Or the functionality that was delivered might work differently than what the users had been led to believe. The end users are sometimes left out of the loop when these decisions are made. Be prepared to field these questions and, if necessary, take extended discussion of these issues offline. You do not want these discussions to derail the purpose of the training.

Acceptance Testing

Most outsourced projects have some kind of official event where a representative of the outsourcing organization's management reviews the new system's functionality and officially signs off, indicating that the contractor has fulfilled each of the new system's requirements. This is usually called acceptance testing.

Like many joint events between customer and contractor, expectations need to be set carefully ahead of time. Several considerations exist, from both the contractor and the customer's perspectives.

Contractor's Checklist for the Acceptance Test

The contractor should make the following preparations for the acceptance test:

• It is useful to have as many workstations as necessary to fully exercise the product's functionality. Depending on the size of the group attending from the outsourcing organization, consider obtaining projection equipment rather than having people crowd around the monitors. For offshoring situations where the customer and the contractor cannot be in the same location, teleconferencing software can be used to remotely display the screen contents.

• Have a workstation available in the same room with someone from the contracting team dedicated to capturing any defect reports or change requests.

• Walking through the system's use cases is an excellent way to demonstrate the system's functionality. The use cases describe all the functional requirements, and users should already have some familiarity with them.

• Be aware that with a system that has lots of functionality, it may take several days to completely walk through the acceptance test. A dry run of the acceptance test helps you estimate how long it will take, in addition to catching any last-minute bugs. Create a schedule and publish for the customer a list of which use cases will be covered each day. This will help the customer identify the proper attendees.

• Have development people available on call to assist if any technical glitches occur.

• It is vital to ask the customer to have someone in attendance at the acceptance test with sign-off authority. In some cases, this may be your best opportunity to get the customer to agree that you have met the requirements. One risk that occurs occasionally is "the project that never ends." The customer will not commit to agreeing that the project is done, but instead drags it out as long as possible to get the contractor to continually add or change functionality.

• It is also helpful to have other personnel from the project team available, especially the project manager, requirements analysts, and someone who can simply observe the reactions and general demeanor of the customer representatives.

Preparation is the key to a successful acceptance test. The end of the acceptance test should result in one of three situations: acceptance of the system, acceptance with conditions (usually conditional on any bugs discovered being corrected), or rejection of the system. Hopefully, your system will achieve one of the first two.

Outsourcing Organization's Checklist for the Acceptance Test

The customer should make the following preparations for the acceptance test:

• Be sure to bring along experts from the user community who are familiar with the functionality to be demonstrated.

• Be inquisitive about the behavior of the demonstrated functionality. If something does not seem quite right, ask questions, or exercise the functionality until you are confident it is functioning correctly.

• Don't forget to verify that any nonfunctional (supplemental) requirements have been met. This includes proof that the system can support the required number of simultaneous users, failover and reliability requirements, and so on.

Most contractors are happy to work with their customers to achieve final system sign-off. Be prepared to sign off on items that have been demonstrated successfully so that the contractor can focus on correcting any final items.