CASE DESCRIPTION

A Mainframe Legacy

In 1970, a team of developers was contracted to develop a mainframe-based patient accounting system to manage admissions and billing functions at RRMC. Over the next 25 years, this system would serve as the backbone of RRMC Healthcare Information System (HIS) architecture. Hardware was upgraded, then replaced, as performance needs or application requirements dictated.

In the early 1990s, serial terminals were retired in favor of PCs equipped with terminal emulation software. Code was modified to keep pace with the ever-changing mass of insurance, Medicare and state Medicaid regulations. New point solutions for departments such as laboratory and radiology spawned small departmental LANs. Some of the applications running on these isolated LANs were interfaced to the mainframe. These interfaces were custom-written and typically transferred only Admissions, Discharge and Transfer (ADT) data.

ADTs are the most basic level of data relayed by an HIS interface. When a patient is admitted to the hospital, an Admission transaction is generated. Depending on the interfaces available, these transactions will be passed to other systems, such as a laboratory management system, to eliminate the need for reentry of patient demographic information. If a patient is later transferred to a different room or unit, a Transfer record will be created and passed to all downstream applications. Similarly, when a patient is released from the hospital, a Discharge record will be processed to close out the patient's visit, also known as an encounter. Since medical history data is vital to proper patient care, the patient's record is usually never removed from the system. Additional encounter records will be linked to the patient record and some financial transactions may be purged, but the core demographic record remains. These retention requirements provide additional difficulties when replacing a legacy patient management system, as will be seen later.

It must be noted that ADTs alone, do not constitute all of the interfaces required to meet the needs of a modern HIS environment. Interfaces for patient orders, test results and billing information also play a part. These interfaces were lacking in the original system; thus, much manual effort was still required. In many cases, the lack of patient order interfaces caused the medical center to lose potential revenue, due to charges lost due to manual processing. Figure 1 details, albeit in a greatly simplified fashion, the primary data flows utilized in the RRMC's new healthcare information system.

Figure 1: Primary Data Flows in the RRMC Healthcare Information System

By 1995, the core patient accounting application remained largely intact. While other, less automated, medical centers were beginning to take advantage of the more standards-based systems being offered by HIS vendors, RRMC was becoming increasingly aware of the constraints imposed by its highly customized environment. The medical center administration team realized that the mainframe's days were numbered and formed a search committee to recommend a course of action.

Building the Team

Comprised of departmental managers and supervisory level staff from critical clinical areas such as nursing, laboratory and radiology, as well as ancillary services such as business office, medical records and purchasing, the search committee began evaluating the HIS landscape in early 1995. The CFO of the medical center served as champion for the project and the leader for the search committee. While some members of the hospital staff and management had grown quite comfortable with the existing system and questioned the need for change, no one openly opposed the project. Having dealt with the legacy of customized applications, the decision was made that the new system would be implemented with a "no-custom" directive. Therefore, the chosen application would need to support the majority of the RRMC's requirements without specially tailored code.

Since RRMC had not undertaken such a massive revamping of its operating environment in many years, a consulting firm with experience in HIS acquisitions was brought in to "fill in the gaps" for the team. One of the first things noted was that most, if not all, of the available solutions required a more experienced IT support staff than was currently available in the medical center's Data Management department.

During this time, the Data Management (DM) department was actually an adjunct of the Business Office and reported to manager of that department. Most of the staff within DM was familiar with basic system operations, reporting and data-entry tasks. More complex issues with the mainframe were handled by the vendor. While the supervisor of the department performed basic PC support tasks, the majority of the microcomputer hardware and software issues were handled by the medical center's biomedical engineering service. It was clear that no members of the existing staff possessed the breadth of IT experience to oversee such a project. While the search committee proceeded with the onslaught of vendor presentations, site visits and marketing literature, the CFO shifted her focus to finding an experienced manager for the Data Management department.

After a three-month search, a suitable candidate for the DM manager's position was found. Soon after his arrival in July of 1995, he began working closely with the search committee and the consulting group to select a vendor from the remaining candidates. The prospective list had been whittled down from nine to three. Two heavyweights in the HIS industry were part of this list as was the original vendor of the mainframe solution, now offering a client-server platform. Early on in the process, the group had decided against a "best of breed" approach in favor of a single vendor solution. The driving force for this approach was the perception that a single vendor solution would be more integrated, therefore easier to use and support. Many vendors were dropped from initial consideration since they only provided solutions for particular clinical areas.

Further, more detailed, site visits were conducted during the final quarter of 1995. These visits focused on customers of the three vendor finalists. By December of that year, the medical center made their choice and initiated contractual negotiations with healthcare systems vendor McKesson/HBOC that lasted two and a half months. In February 1996, the contract was signed. The project was underway.

Integration Commences

Over the next several months, new staff members including a technical project lead, a decision support specialist and a network supervisor were added to the DM department. Existing staff members continued to support the production environment, while learning skills required for supporting the new environment. Some staff members in DM and other departments found themselves away for weeks at a time attending training sessions at locations across the country. In some cases, project timelines were compromised because critical personnel were off-site at these sessions. Figure 2 details the key IT personnel involved in the implementation effort.

Figure 2: Key IT Personnel Involved in the Implementation

Many tasks had to be completed before the first server to house the new applications could be brought on-site. One such project was the construction of a new data center to house the collection of Unix, Novell and NT servers that would make up the new system. The existing mainframe computer room was environmentally inadequate to meet these needs. Plans were made to convert several existing DM offices into a new computer room. Work on this phase of the project required four months to complete. Some small delays were introduced when it was discovered that the floor could not support the weight of the new systems. Additional supports had to be installed before the new data center could be utilized.

A new network infrastructure was also required to support the new environment. RRMC's existing LAN was a collection of cable of varying capacities and installation quality, placed by a number of vendors and in-house personnel. A network services vendor was brought in to assess the current cabling plant and develop a new infrastructure plan. This portion of the project ultimately required 15 months to complete and expanded the network from 90 nodes to over 700. Priority for new cabling installations was given to the new data center, DM offices and members of the Systems Implementation Team (SIT).

The SIT was an outgrowth of the systems selection committee, charged with the build and testing phases of the project. The manager of DM asked each area affected by the project to select one or more persons (depending on the size of the department) to act as the "power users" for that area. It was determined that this group would not be comprised of management staff, since managers would be unable to focus all of their time to one project. By August of 1996, the systems and applications were available for the SIT to begin the build process. The project plan called for a one year build and test phase. The system, actually a collection of applications and hardware, was scheduled to enter production in October 1997.

While chosen vendor's history prior to the early 1990s was mainframe-centric, they had purchased several distributed-system development firms during a burst of acquisitions in the mid 1990s. It was this "growth by acquisition" approach that caused many of the challenges for RRMC in terms of integration. At the time that RRMC purchased an "integrated solution," the vendor was still trying to finalize their own integration strategy.

At the core of this strategy was Health Level 7 (HL7), which is a vendor-independent protocol for passing data from one HIS application to another. HL7 is to healthcare applications what ANSI X.12 is to Electronic Data Interchange (EDI). It is an attempt to create a "Lingua Franca" for healthcare-specific data.

During the selection process, HL7 compliance was a requirement of RRMCs RFP. Companies acquired by the vendor were at different stages of implementing HL7. Some applications, such as the laboratory management system, supported HL7 v2.3, while the clinical documentation and patient accounting systems supported v2.2. Additionally, some of these products changed HL7 versions as a result of upgrades applied during the testing phase of the project.

As has been noted previously, historical patient records must be maintained for long periods of time in the interest of quality patient care. Thus, some conversion of data from the mainframe system to the new client-server platform was inevitable. Since the legacy patient accounting system did not contain much historical clinical data, it was decided that only a patient's demographic and insurance information would be converted. It was this dependence on legacy data and the complexities of conversion that forced RRMC to take an allor-nothing approach to installation of the new system. Rather than bringing the systems online in phases, the six core systems were slated to enter production status at one time. As a result, all interfaces between the systems had to be functional by October 1997.

McKesson/HBOC had chosen the DataGate interface engine by Software Technologies Corporation (now SeeBeyond Technologies) as their primary method of tying their disparate systems together into an integrated solution. STC was not acquired by the vendor, but rather worked as a business partner. STC was selected due to their early focus on the healthcare market (Schulte & Altman, 2000). The DataGate application was rebranded as Interface Manager by the McKesson/HBOC, but aside from the inclusion of the HBOC-specific interface libraries, the product was identical to the STC release.

HIS Integration

DataGate (recently rebranded as eGate) is a very powerful application that allows a trained user or programmer to move data from one application to another through a variety of protocols and data formatting standards. At the time of RRMC's installation, the DataGate server was supported by Unix variants only, although client machines could run any OS.

Within DataGate, data flows (messages) are processed in a FIFO sequence (unless reprioritized within the DataGate application) against authentication rules (message identifications and/or message translations). Figure 3 illustrates generically how DataGate processes these flows:

Figure 3: DataGate Interface Engine

While DataGate greatly simplified the management and administration of HIS interfaces, it did not, by itself, provide all the tools necessary for their construction. Metadata, in the form of documentation which described the various inbound/outbound sockets built into the core applications, was required from the vendor in order to build the appropriate communications to and translations within DataGate. Difficulties in obtaining accurate metadata, provided the seeds for delays which crept into the HIS implementation timeline.

Project Delays

In the first few months of the build/test process, the DataGate engine was used very little. Before any meaningful testing could occur, numerous tables such as: department master, unit master, physician's master and item master had to be populated. Without these tables in place, no test admissions or orders could take place. This was not to say that EAI problems were non-existent during this phase. The applications that comprised the new system were, in most cases, originally developed with text-based interfaces. The vendor was in the process of building Graphical User Interfaces (GUIs) for these applications and RRMC found themselves an unwitting beta-test site. While this required extra effort on the part of the implementation team, it resulted in minimal impact to the project timeline.

As the build process neared completion, data conversion and interface tasks began to take more of the spotlight. The conversion process began in the fall of 1996, but problems caused the process to extend to September 1997, the month immediately preceding the planned transition to production status. Most of the problems resulted from data integrity issues with the source files. Literally, millions of records had to be processed and then reprocessed in cases where the conversion routines aborted due to data inconsistencies. A single conversion run might take days, depending on the file in question. Many days were spent tracking down the source of conversion terminations and re-running the processes. These conversion issues did not directly impact the testing of the new system, but did add impetus to delay the launch from October 1997 to December 1997. What did impact the latter stages of the test phase were difficulties encountered with the implementation of the interface engine.

As noted earlier, there was a lack of reliable metadata from which to construct the validations and translations that would be used by the DataGate engine. Since the HIS vendor was working internally on its own integration efforts, it was frequently difficult to get support from them for RRMC issues. By August of 1997, it was clear that too many interface issues and related testing remained to make an October launch feasible. Faced with PR issues due to a delayed implementation of their product suite, the HIS vendor redoubled their efforts to insure the success of interface efforts. Final testing was completed in mid-November.

Moving to Production

The completion of testing by no means meant that project team members could relax. There was still planning to be done for December 1, the date on which RRMC would see if the efforts of the past two years had paid off. The Administrative Team of the medical center was charged with acting as "goodwill ambassadors" during the migration to production and canvassed the medical center with light snacks for staff members.

To insure accessibility, cellular telephones were issued to all key project members for the duration of the launch. Additionally, a classroom used for system training was converted to a "war room" and staffed with those most familiar with the applications and product representatives from the HIS vendor. These personnel would act as a first level help desk.

Patients and visitors to the medical center were kept informed of the changes and potential impact of the conversion by way of posters and placards placed in key areas such as admitting. Admissions clerks reviewed their screens and instructions for a final time, as did the nursing staff. By the weekend of November 29, all was ready.

The process of moving to production on the new system turned out to be a busy time for the project team, though no major problems were noted. A few interface issues arose, but these were quickly resolved by the combined RRMC and vendor staffs. Problems that were not immediately resolved were placed on an issues list. Over the days and weeks to follow, this list was reviewed by the project team, which, after the transition to production status, was renamed the Systems Integration Team. The project manager provided by the vendor was kept apprised of the status of items on the list and any new issues that arose. Items that remained on the list for more than a couple of weeks or were of an especially critical nature were escalated by the DM manager to the vendor's upper management.

Post Production

By January 1998, the institution had settled into a regular maintenance mode with the new HIS, but a new round of EAI changes was just around the corner. The initial interface configuration which had served for initial production was showing its inflexibility at meeting new system and workflow demands. Converting these interfaces into a format that allowed for more flexible processing was hindered by the lack of personnel. As the technical project lead had moved to the manager's chair during the course of the implementation, a new team member who could be dedicated to the critical task of interface management was needed. The unique set of talents required made the newly created position of Interface Analyst especially difficult to fill. Several months passed before a suitable candidate could be found.

With the new analyst on board, work began on the interface conversion process. RRMC's decision to rebuild its HIS interfaces was driven by several factors. Many new systems, such as mammography tracking and fetal monitoring, that interfaced to the HIS were planned. Regulatory changes also dictated the changes. For example, vendor delays in meeting guidelines for outpatient billing were scheduled forced RRMC to develop their own solution using the DataGate engine.

The largest challenge to date at RRMC with respect to interfaces was the installation of an emergency department (ED) tracking and nurse charting system. The package was designed to fully track a patient's visit through the emergency department from the second they present at the door to the minute they are transferred or discharged. The process to retrieve the data required by the ED tracking system was handled through the Data Gate interface engine. Once again, EAI techniques freed the institution from having to resort to custom programming.

Lessons Learned from HIS Implementation

It would be easy to categorize an HIS implementation as a purely technical challenge, given the IT expertise required to make such an effort feasible. While having the necessary knowledge in-house is certainly critical, IT personnel must not discount the "people factor." RRMC realized early in the project that gaining buy-in from senior management, clinicians, ancillary staff and possibly most critical, physicians, was concomitant with the project's success or failure.

This "buy-in" is not merely an acceptance of the inevitability of the project, but also a feeling of ownership and participation in the final product. Implementation teams and target users alike must understand the goals of the organization in committing resources for an HIS implementation. In a business sector where budgets are extremely tight, end users and indeed, members of management in some cases may not see the "big picture." The IT department may be viewed as an ivory tower at best and a money pit at worst. The business case for an HIS revamping must be made clear to all.

Before undertaking a task that can easily overcome an institution's financial and personnel resources, it is vital to understand the following points:

1. Know the difference between an interfaced system and an integrated system.

   Vendors often use the term "integrated," when in fact their application is actually a collection of smaller point solutions that are tied together via point-to-point interfaces. A truly integrated system may be comprised of several smaller applications tied together via an interface engine such as DataGate, but these applications are presented to the user in such as way as to appear seamless. A common look and feel is apparent throughout the environment.

   On the other hand, an interfaced, but little-integrated HIS application will appear fragmented to the user. Training staff to use such as system will be more difficult since they may have to learn several different user interfaces to switch between components effectively. An interface engine may provide the back end data sharing, but the users, in effect, must provide the front-end integration.

2. Have "champions" for the project at all levels of the organization.

   Even in small health care centers, the task of moving to a new HIS environment is staggering. In any organization, the longer a project takes to complete, the more likely it is to suffer from the perception that it is a never-ending process. Enthusiasm wanes for the original goals. Personnel changes, inevitable in any dynamic institution, further hinder the efforts to "keep the fire" under project members and the organization as a whole. For these reasons it is essential that the project team includes key staff members from senior and middle management, as well as influential clinical staff members to act as PR "agents" and further the cause

3. Insure that the proper skill sets are available before the project begins.

   In many instances, organizations proceed too far down a project plan before realizing that sufficient expertise does not exist to complete the task. Usually this is the result of poor communication between the project steering committee and front-line personnel. Too often, management feels that once the project is complete, the need for specialized knowledge is reduced. As a result, they may turn to short-term expertise as might be found through consulting firms, only to find out that once the consultants leave, so does the knowledge.

4. Negotiate vendor contracts carefully.

   Caveat emptor. While the process of contract negotiations may seem like so much administrative busywork as compared to other, more concrete project tasks, possibly no single item has more impact on long term success. Make sure that project deliverables are clearly specified along with a specifically stated timeline. This greatly reduces the chance that the institution will be bitten by the "vaporware" bug. The vendor will be contractually obligated to provide what they promise or face various legal remedies. It is also very important to include on-going support and maintenance into the agreement.

   Software vendors normally increase maintenance fees for their applications between 6% and 8% per year. Lesser (or at least fixed) increases can be negotiated upfront to ease the budget planning process in future years. Frequently, new major releases of and application are not included as part of the maintenance and licensing fees. These, too, may be negotiable, depending on how eager the vendor is to complete the deal. RRMC spent three months in intense negotiations with the vendor to lay the contractual foundation of the project. Over the course of the next four years, the contract documents were referenced to iron out misunderstandings between the vendor and the institution.

5. Develop stringent system monitoring procedures.

   One of the greatest post production challenges was insuring that all critical systems and applications were properly monitored. In the original mainframe environment, system monitoring was greatly simplified since the applications resided on one central system. In the era of distributed systems, however, it is very difficult to impossible to provide the monitoring necessary without exceeding staffing constraints. Monitoring toolsets and clearly written policies are vital.

   RRMC used a variety of means to keep track of system performance. Most important was the interface status monitor provided with the DataGate product. The monitor was a prominent feature of the operations team office and was consistently checked for errors. Additionally, a number of scripts and procedures were written by the staff to provide additional alerts. A "red book" of critical systems procedures was kept constantly updated and available both on-line and in hard copy in the data center.

6. Understand the workflow impacts of new systems.

   Some organizations may be tempted to consider new systems as a means of bringing efficiencies to existing processes. This shortsighted approach can lead to frustration after the system is placed into production, which can spell doom for the project and its backers.

   Unless an application is custom written for an enterprise, some workflows and processes that have stood the test of time must be augmented or entirely scrapped. It is all too easy to fall into the trap of modifying the application, when it is the old way of doing things that should be revised. There are situations that may necessitate customizations, but each proposal must be reviewed carefully. Otherwise, the organization may find itself in a position of being unable to apply applications upgrades in a timely fashion due to the required retrofitting of customizations. This was the situation that RRMC found itself in with the original mainframe system. As a result of RRMC's decision to avoid "custom code," the institution has successfully applied two major system upgrades since 1997 with little difficulty.