Scaling at Medcinsoft

Scrum was used at Medcinsoft to manage a Y2K project aimed at making Medcinsoft ‚ s products Y2K-compliant, providing releases of this software to over 350 major healthcare organizations and teaching hospitals (customers), and stabilizing these releases before October 1, 1999. Medcinsoft ‚ s software ran the administrative aspects of healthcare organizations, including admissions, discharges, billing, insurance, patient records, and receivables. The failure of such software would have had catastrophic consequences for these organizations and the populations they serve. Until Medcinsoft adopted Scrum, it had been using Gantt charts to manage the project, and they had proven woefully inadequate to the task. Customers were dissatisfied because releases regularly arrived months late, were horrifically buggy , and didn ‚ t contain desired features.

The project was complex and required scaling in a number of dimensions simultaneously . The software development, Y2K remediation , and bug fixing of over 300 developers of various product lines had to be prioritized and coordinated. Over 400 field support staff needed to have their work and communications with Medcinsoft customers prioritized and coordinated. Over 600 employees at the 350 customer organizations needed to be able to communicate plans and changes in these plans to Medcinsoft. Customers were operating various releases of Medcinsoft software, most of which were customized to their specific needs. All of the customers had different timetables for the implementation of the Y2K upgrade of Medcinsoft software, each of which took into account their plans to upgrade other software they used on site. Customers also had specific schedules for executing extensive testing of their new systems. In sum, customer timetables, knowledge bases, and skill levels varied widely.

Scrum had been used successfully elsewhere within Medcinsoft, so management asked Jack Hart, the manager of the Y2K project, whether he could use Scrum to help the Y2K project along. The most pressing problems that Jack had to address were the complexity of coordinating all of the work being done and the variability of timing in the different parts of that work. To coordinate the work being done, he needed accurate, timely information. Planning information from the customers came in sporadically and was oftentimes contradictory. Release status information was unreliable, and releases often weren ‚ t ready until weeks after they were supposed to be shipped. Customers and field service personnel weren ‚ t communicating with each other at all or weren ‚ t communicating effectively with each other.

Each customer ‚ s schedule for testing Medcinsoft ‚ s software was tied to the testing of other software and software packages, and those plans were in turn tied to training and rollout plans, all of which had to be completed before the start of Y2K. Medcinsoft had to deliver its new release on time, but it also had to be able to change its delivery date if customers experienced any changes in their schedule. Each Medcinsoft release had to include software that was completely tested , with all of the known Y2K remediation work completed, with critical and high-priority bugs resolved, and with any outstanding defects documented. This release had to be implemented at the customer site, and any previously customized code would be identified and implemented in the release by Medcinsoft field personnel. Finally, the Medcinsoft release had to be integrated with and interfaced to other software in use at the customer site.

As though this weren ‚ t enough, Jack had further complications. Despite having previously conducted extensive searches for Y2K defects in accordance with industry and internal guidelines, new Y2K defects were still being found on a regular basis. Also, Medcinsoft planned to integrate new Web access functionality into the Y2K release. Operational defects resulting from the Web enhancement proved difficult to detect, and as defects were corrected, other bugs were often created. Parts of the software were practically ancient; because yesterday ‚ s developers ‚ the ones who had written the code ‚ were no longer around to help rewrite it, today ‚ s developers had to learn the code as they upgraded it. Furthermore, the base software was over 2500 function-points in size , large by almost any standard. There were additional customer complexities in the mix, too. Many Medcinsoft customers had never performed such an extensive upgrade of their software and systems and were largely unprepared to do so. A surprising number didn ‚ t have testing and quality assurance environments or processes with which to systematically test a new release.

Approach

Scrum provides a degree of regularity or predictability in a complex or chaotic environment like that of the Medcinsoft Y2K project. Jack decided to apply Scrum to all aspects of the project, even activities in the field and at customer sites. Jack synchronized releases with each Sprint. Every 30 days, Medcinsoft would provide specific customers with a new release of software. Each Sprint would create a release that addressed top-priority Product Backlog and any critical bugs found during the Sprint. However, Jack found it difficult to acquire accurate information regarding customer priorities, implementation dates, and critical fixes in a timely manner. Field support personnel provided information about each customer ‚ s needs, and different people at the customer organization communicated with them. Jack needed a normalizing filter for this information, a way to make it timely, predictable, and accurate. To this end, he instituted Daily Scrums for customers who were within three months of requiring a Y2K release from Medcinsoft, and he instituted weekly versions of the Daily Scrum for customers whose release date was outside the three-month window. At each Daily Scrum meeting, the customer and the Medcinsoft field support personnel discussed status and issues. They maintained an updated, prioritized list of the required date for the Medcinsoft software release, customization required for that customer, customer-specific enhancements or functionality that had to be included in the release, and outstanding critical and high-priority bugs and defects at that customer site. (See Figure 9-2.) Each customer had its own list, or Product Backlog, that evolved across time.

Figure 9-2: Customer Product Backlog

Whenever the customer Product Backlog changed, it was rolled into Medcinsoft district and divisional Product Backlogs in the Medcinsoft field service organization. This combined Product Backlog was then prioritized to reflect scheduling among all of the customers. This combined Product Backlog was known as the ‚“field service Product Backlog. ‚½ Field service management used it to plan and allocate field service personnel to perform customizations and assist in the implementation, testing, and rollout of the software.

The Medcinsoft development Product Backlog consisted of Y2K fixes, product enhancements, bugs found in testing, and other high-priority development environment work. When the field service Product Backlog was merged with the development Product Backlog, the result was an overall Medcinsoft Y2K Product Backlog, which was used to prioritize and direct the overall Y2K efforts. Development work was prioritized based on Y2K objectives and specific customer requirements, with all priorities driven by dates. This Product Backlog supported the various customer implementations and directed all Medcinsoft and customer work. (See Figure 9-3.) The Medcinsoft Y2K Product Backlog was updated every day.

Figure 9-3: Hierarchy of Product Backlogs

The customer, district, division, development, and overall Y2K Product Backlogs were maintained on spreadsheets on a public server and could be accessed through the Web by everyone involved in the project. This required quite a lot of cooperation and communication, as the spreadsheet could only be updated by one workstation at a time. However, the people at Medcinsoft felt that this solution worked well enough, and they appreciated that it made their Y2K commitments and priorities visible. Everyone appreciated being able to plainly see how work was allocated and how release content was scheduled.

The field service organization was responsible for coordinating and conducting all work at customer sites. Judy was a district manager who had been brought to headquarters to help with the Y2K project. She assumed the Scrum Product Owner role and maintained the overall Y2K Product Backlog of work and kept it prioritized. She maintained the list by asking the following question: ‚“When do customers need a release with Y2K defects fixed and any other requested functionality? ‚½ For instance, the subset of top-priority Product Backlog at the start of a Sprint might look like this:

• Defect of report header printing date incorrectly in module or report.

• Defect of screen in module patient demographic displaying year incorrectly.

• Module patient demographic freezes when year 2010 entered in date field.

• New plug-in module from software vendor to fix rollover date problem.

• Bug ‚ screen address change of module patient demographic doesn ‚ t return to correct prior page.

• Customer MediLife requires release for implementation (currently running release 8.1).

• Customer MedClinic requires release for implementation (currently running release 7.2).

Development teams were grouped by functionality, such as BAR (Billing, Accounts Receivable), SCHED (Scheduling), and so on. (See Figure 9-4.) At Sprint planning meetings, the team in question worked with Judy to select for its next Sprint the Product Backlog that was within its functional domain. The team members had worked at Medcinsoft long enough so that who was responsible was rarely in dispute. If not enough work was available to fully load the team, the team would allocate only an estimated percentage of its time to the Y2K project during that Sprint and would work on other projects during its remaining time. Jack used this approach to keep work parsed cleanly, since up to 20 teams of up to 10 members could be simultaneously working on the Product Backlog. These teams included functional teams, build teams, Y2K-defect detection teams, quality assurance (QA) teams, and release teams.

Figure 9-4: Sprint planning for multiple teams

The same scaling mechanisms were used for the Web access project, which consisted of completely new functionality and code that was to be included in the Y2K release. This project began with rather extensive system architecture and design Sprints. The result of each Sprint was some working functionality as well as an increasingly detailed system design that was used to keep the various Web access teams from tripping over each other.

Bug Fixing

During each Sprint, bugs were entered into a defect tracking system. Some of these bugs were caught as a testing team regression-tested the ‚“being built ‚½ increment. Some of the bugs were Y2K bugs that were detected during further Y2K remediation testing. Some of the bugs cropped up as customers tested and implemented their own Y2K releases. Medcinsoft ‚ s initial intent was to fix all bugs and defects prior to the final Y2K release. This proved unrealistic , however, as many customers wanted to ‚“ batten down the hatches ‚½ well before the start of the year 2000. To accommodate these earlier implementation dates and improve the integrity of each release, Judy reviewed new bugs prior to Daily Scrums and added all critical and high-priority bugs to the Sprint Backlog for the various teams working on that release. This was a violation of the Scrum rule that no outside work will be added to a team ‚ s Sprint once it is under way. However, the Scrum practice of following common sense won out, because unless these defects were addressed immediately, pushing the new releases out to the field would just be asking for trouble and wasting customers ‚ time.

Jack gave the Scrum teams the following rules for managing their own time and work. He told them that Y2K work was their highest priority. If they had allocated only 50 percent of their time to this Sprint and were working on other things the remaining 50 percent of their time, they were to drain the other 50 percent and reallocate it to working on Y2K bugs and defects until they were all resolved. Or, if the team was 100 percent ‚ allocated to the Y2K Sprint, team members were to reach outside the team and include other engineers to help address the Y2K bugs and defects until they were all resolved.

Lessons Learned

Medcinsoft successfully navigated the shoals of this project ‚ s complexities and met its customers ‚ needs through constant inspection, analysis, devising of solutions, and adaptation. Many of the teams involved in this project didn ‚ t develop software; they tested software, implemented software, or managed organizational activities. Yet they all used the inspection and adaptation mechanisms of Scrum to stay on top of every change.

You can get through almost anything if you don ‚ t try to impose rigid solutions before problems even arise, instead devising solutions as necessary when problems crop up. Such solutions as the hierarchical Daily Scrum of Scrums worked, and the regularity and brevity of these solutions made them easy to bear. However, it wasn ‚ t the solutions Jack devised as much as it was the simple rules that the teams followed that caused the increased, synchronized communications that ultimately saved the project. Insisting on creating a shippable release every 30 days, regardless of what happens, and making certain to frequently synchronize customer realities with Product Backlogs always kept Medcinsoft on the straight and true.