Testing Under Live Conditions


Lead Advocacy Group: Test

It is amazing that no matter how good the requirements are and how much user feedback is solicited, some users will always surprise the team with how they approach and use a solution. As such, there is no substitute for putting a solution out in front of a broad array of users to observe and measure their experience. Commonly called piloting a solution, this process involves deploying a limited release to production or to a production-like environment (e.g., staging environment) to validate that a solution works as expected under live conditions with "real" users.

As outlined in the Pilot Plan, successive pilots might be needed. Typically, each successive pilot broadens user involvement (e.g., pilot first to a team, then to a department, and then to a division). As per the Pilot Plan, the pilot is concluded after a fixed period or after specific goals and objectives have been satisfied.

In many ways, a pilot is a project within a project. It needs a plan that details what, when, why, who, and how the pilot is to be conducted. It needs to be well planned, executed, and concluded. It involves piloting other solution elements to assess such as support and training of pilot users as well as end-user communications (e.g., launch preparedness announcements). The next few sections discuss these topics.

Goal

The goals of a pilot are to stabilize a solution further using feedback from a broad representation of users and to reach consensus with the stakeholders and users that a solution satisfies their needs and meets their expectations.

Focus

A pilot should be a well-orchestrated review of a solution to accomplish specific goals and objectivesnot a random walk by users through a solution. Conversely, it should not be a tightly controlled guided tour through a solution either. A team should make the purpose of the pilot and the measurement of feedback clear but let users go where they may within the context of what needs to be reviewedthis is necessary to surface any unexpected behaviors. Otherwise, if a team makes pilot users go down the exact same paths that testers did, users are unlikely to turn up anything new.

As part of the goal of using a pilot to stabilize a solution further, a pilot is a great opportunity to stabilize the support and training aspects of a solution as well as solution deployment. As mentioned previously, pilot users are real users that typically have not been involved with a solution, and as such, they will need more support and assistance. This necessitates that the help desk and training aspects of a solution be up and operational. It is also a good opportunity for a team to engage more with Operations to have them attempt a solution deployment and to support Operations as needed.

Preparing for a Pilot

To do it right, many things need to be accomplished and prepared before a pilot commences. Preparing for a pilot basically involves three areas: user involvement, solution evaluation, and solution deployment. User involvement focuses on identifying, soliciting, engaging, supporting, and maybe training users. Solution evaluation focuses on identifying what needs to be reviewed and gathering user feedback, including putting processes and tools in place to gather feedback. Solution deployment focuses on getting a solution deployed to the environment(s) used for pilot, including support aspects of a solution (e.g., online training).

As if it were a production deployment, a project team and operations team need to verify readiness of all aspects of a solution. It should be used as a rehearsal to work out any last-minute issues and to hone everyone's skills.

Conducting the Pilot

The trick to conducting a successful pilot is to engage with the pilot users just enough to keep them productive and providing feedback while not being bothersome. As part of engaging with pilot users, the team follows up with them on suspected issues and handles the issues accordingly.

During the pilot, the team is continually collecting, compiling, and evaluating pilot feedback and operations data.

Concluding the Pilot

Once the pilot completion criteria have been satisfied, the team concludes the pilot and regroups to address all outstanding issues and to refine a solution and supporting material based on the results of the pilot. If the schedule allows, a team might choose to address frequently experienced user errors through changes in a solution or through improved support and training.




MicrosoftR Solutions Framework Essentials. Building Successful Technology Solutions
Microsoft Solutions Framework Essentials: Building Successful Technology Solutions
ISBN: 0735623538
EAN: 2147483647
Year: 2006
Pages: 137

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net