3.3 The Beta Plan

The team will decide what type of beta to conduct. This decision will be based on the type of project and the needs of end users. Betas are public, closed, or a hybrid of the two. A public beta is when the company opens the beta up to the general public. A closed beta is when the company gives the project to a select group of internal users for testing. A hybrid beta is a closed beta that turns into a public beta.

An Internet- related project might be a candidate for a public beta. For example, if your company is creating an E-Learning portal for the general marketplace , you may want to have a sample or a small selection of courses available for the general public, customers, or distributors . Beta is used to work out any kinks in your training site. New applications or services that need to be tested may include a program that customers use to purchase a course, a database that manages the course, or training services that go along with the course. A hybrid beta is when you start out with a closed beta and at a predetermined time move to a public beta. For example, a company with a three-month beta will have a closed beta for two months and then have a public beta for one month.

There are pros and cons to public betas. One of the pros is potential users can now take your project for a spin and figure out if they like it. Some of the cons are that beta applications may not be stable and may not be feature-rich. Giving buggy code or services that are not fully available to the masses may cause bad press and support headaches . Public betas tend to reflect a company s philosophical outlook. Some companies like giving new services to their customers. They approach the competition with the attitude: ˜ ˜See the cool stuff we re doing; we re going to run rings around you. They approach their customers with the attitude: ˜ ˜Stay with us, and you will be the first kid on the block to get this kind of functionality. Another benefit to a public beta is it is typically difficult to find users willing to test-drive beta applications. Running a public beta provides you with a potentially large group of people to test your project. If your company decides to have a public beta you have to be sure the people in the company philosophically understand and feel comfortable with a public beta. Some projects work better as public betas than other projects. If a project (or its beta) requires a lot of hand-holding or support or you have very temperamental or impatient customers, it is better to use a closed beta. If the project is a new service, you have an established customer base, and you have a backup method of delivery, an open beta may be worth a try. For example, your project is creating an online billing application to augment your current paper billing. Currently, you send customers a paper bill; the online bill provides customers with just-in-time information. If the online site is down or working slowly due to unanticipated demand, your beta customers who can t get on the site will still have their paper bill. With a hybrid beta you can get the best of both a closed and a public beta. The benefit of a hybrid beta is you can limit the risk at the beginning of the beta when the solution is most vulnerable. Once you have had an opportunity for more people to test the project and the team has identified and fixed any major bugs , the project can be deployed to a wide audience. This will get more features tested and lead to wider acceptance.

Note: The project launch process outlined in this book is for a closed beta. For a public beta, the project launch starts once the beta is released. In Chapter 4, we introduce you to Communications and begin planning the launch process. If you are going to run a public beta, begin planning for your launch during Project Development. An effective launch will take four months to plan. Using your integrated schedule, figure out when you will need to begin planning for your launch. Since beta will probably last between two and three months, you will announce your plans for the project s general availability at the beginning of public beta. This will give you two to three months to actualize your announcement.

Regardless of whether the beta is public or closed, during development, marketing and QA meet and develop the Beta Plan and Beta Questionnaire. For public betas it is helpful to include a simple survey on your Web site. It has been found that simply giving away a T-shirt dramatically increases the number of people who take a survey.

The beta period takes place after the QA phase is complete. During the QA phase, the QA organization tests the project and confirms that it meets the specifications outlined in the PRD and Design Document and that the documentation is adequate and meets the requirements specified in the Doc Plan. After QA declares the project acceptable to enter beta, the project is either sent to a selected group of users or placed on the Web site for users to test and confirm that the project works as specified. Beta is an extension of quality assurance. Beta provides quality testing by allowing users to confirm that the project meets their specifications in a non-lab environment. The team needs to agree on what the beta period will entail. An example of a beta specification for a closed beta created by a project team is as follows :

It will be necessary to beta test this project at a minimum of three sites to a maximum of five sites over a twelve-week beta cycle. Sites will start on consecutive weeks. Each site will begin its test cycle when a technical representative from the company installs hardware, software, and documentation to the beta site. The representative will present a short training class to explain the new enhancements and will work with users to assure installation. The Beta committee, led by QA, is responsible for creating a list of features that the beta site will review. The QA employee responsible for managing the beta site will call the site weekly to ensure the beta is progressing properly.

Regardless of whether the beta is public or closed, marketing and QA should define a Beta Plan. The Beta Plan consists of a list of tasks, identifies who in the company is responsible for them, and assigns a start and completion date for the tasks so the beta will begin and end on time. Items that may be covered in a Beta Plan are:

Tasks	Group
Identify potential beta sites or create a public beta site	BD
Decide minimum configuration and features to be tested	QA/IT
Draft Beta Questionnaire a. User information b. Site environment information	BD and QA
Prepare checklist of technologies to be tested	QA
Fill in Beta Questionnaire for each potential site	PM
Bring beta site information to project team	PM
Prepare Beta Contract (for commercial projects)	Legal
Set up install date with user	PM/QA
Establish beta committee	QA
Prepare beta site test plan	QA
Prepare beta site cover letter	BD
Ship project/place project on Web server	QA
Install project	Help Desk
Monitor beta progress	Help Desk or QA
Ship beta updates/update Web server	QA or Help Desk
Beta sign-off call and sheet (review tested features)	QA
Review and send sign-off sheet and approve sign-off	Team
Prepare beta sign-off letter (closed site)	PM
Send beta site a gift (closed site)	PM

The Beta Questionnaire is a document reviewed by potential beta users to confirm that their site meets the needs of the project, that the site is qualified and interested in testing the project, and that the project team has the necessary information on each beta site. For public beta, the questionnaire should be a survey users take before they get access to the new service. Examples of questions asked in a Beta Questionnaire are:

Information necessary to qualify a potential beta site:

1. Company or department name and address along with the key contact name , phone number, fax number, and e-mail address.

2. System configuration and operating release level.

   • The beta machine must have (minimum hardware configuration prior to the test).

   • The beta user must have (minimum software configuration) prior to the test.

3. The user understands that problems with a beta project may interrupt normal operations. We therefore request the beta take place in a noncritical projection environment.

4. The user must provide status updates to (company name) (beta coordinator name) on a weekly basis.

5. The user must complete the beta evaluation questionnaire after the testing is over.

6. The user must commit to beta testing this project for a minimum of thirty days.

7. The user will be provided with preliminary documentation, which may be incomplete or contain errors. If the user elects to purchase the project, this documentation will be replaced .

Evaluating Vendors

Not all projects will require a company to evaluate different vendors. Many projects add on to existing projects where the vendor has already been chosen . For projects that need a vendor solution, the Project Manager, Business Development Manager, and IT Manager will need to team up and review different vendors solutions. Each member of this group brings a different skill and different idea of what makes a good solution. These members will need to evaluate different vendor solutions. There are a couple of simple guidelines to making a successful vendor choice.

1. Create a list of companies that provide the solution you are looking for. To find companies:

   1. Do a Web search.

   2. Ask businesses similar to you who they use.

   3. Check trade journals.

2. Keep an open mind. A vendor s salesperson may have been working with your IT department for a long time. They may have been spearheading the project. The teams responsibility is to the company and the project, not to the vendor. Keep an open mind; review multiple solutions; you may be surprised what you find. Salespeople have a way of disappearing after they sell a solution. Your job is to ensure that the company receives the best solution.

3. Use your Design Document. Vendors will provide you with their list of features. This is your project. You need to be sure that the solution meets all of your ˜ ˜A requirements.

4. Check references. You should talk to at least three customers who have installed and used the solution. If the vendor can t provide three customer references ”run! Make sure the referred customers are using the solution in a production environment. During the Internet boom many new companies came on the market. They got big funding, impressive offices, and lots of press. Salespeople gave presentations showing all the Fortune 500 customers using their solution. Established IT folks bought these new solutions thinking they were complete applications only to find that it was Christmas morning. The solution came in one hundred different pieces that didn t work together, and IT had to make them do so. Have the vendor provide you with a list of customers that you call and talk to. For large projects, have the Project Manager talk to the person who managed the solution at one of the referral companies, have the Business Development Manager talk to her peer at the company and the IT manager talk to his peer at the company. For smaller projects you will need to talk to only one person at the referral company. The goal of these conversations is to ensure that the solution works as promised and that customer support continues after the product has been purchased.

3.3 in a Nutshell

The Beta Plan is a document that outlines how the test will be conducted , who will receive the test product, and how the team will manage the test.

• The team decides what type of beta the project will need.

• Betas can be public ”open to everyone ”closed ”open to a select group of people ”or hybrid ”start out closed and eventually be made open to everyone.

• QA drafts a document outlining the requirements of the beta user.

• QA also drafts a questionnaire that beta sites will answer to ensure their site meets the team s needs.

It s true: Beta is when a select group of users try out the product.