Managing the Development Environment

Because today's systems are complex and critical, the development team must have a solid environment in which to work. Managing the application development environment is often just as important as managing the project itself. This environment should be isolated from the working environment of the organization, which we call the production environment, because of the potential for negative impact both on the organization's daily work and on the direction of the application development.

As a critical aspect of management, a change control system must be in place during both the development and maintenance periods to ensure that changes to a new application don't adversely affect the organization's production environment and the users that rely on its stability for daily operations. For example, in the early days of Web development, when Web pages usually consisted of only text and pictures, it was easy to make changes to them. However, today's Web pages are much more complex, and many are full-fledged application environments that must be maintained and available at all times. Downtime on a Web site can mean lost revenues or can cause bad will when customers are unable to access necessary resources. It's no longer acceptable for a Web site to be unavailable while someone tracks down the change that caused the site to stop working properly, ascertains who made the change and why, and fixes the problem.

A change control system that is not enforced is useless. The development team must put policies in place that prevent control procedures from being circumvented. For example, no one should be able to make direct changes to an application in the production environment without first applying that change within a testing and certification environment. The process for making changes must be well thought-out, well understood, and as automated as possible.

Production Channel

Application projects can go through at least four distinct stages, which comprise a life cycle called the production channel. These stages are:

• Development
• Testing
• Certification
• Production

Each of these stages is supported by a specified team of people and computer systems. Neither the people nor the systems should overlap between stages. A set of rules should be developed to determine when and how specific changes to the application progress through the production channel. By following these procedures, any problems will be discovered in the testing or certification stage, rather than in the production stage.

Development

First and most important, all changes occur on the development systems. If this basic policy is not established in the beginning, the entire process is likely to fail. This policy should not be compromised and should be enforced throughout the system. Developers will often be tempted to skirt around the policy when a critical change is needed or when going through the entire rollover process is too inconvenient. However, even the smallest and most innocent change can have unexpected and destructive consequences. Creating a policy and sticking to it ensures that the entire process is successful.

Testing

Each project should have a comprehensive test plan that encompasses the major functionality of the application and ensures that everything works properly. After the application is modified on the development systems, the changes are sent to the testing systems. This "sending" process may be nothing more than a batch file or a button on a Web page activated by whoever has been assigned this responsibility and has the proper access rights to initiate this "rollover." On the testing systems, a series of tests determine whether the application is still working properly after the changes.

The testing stage concentrates on functionality and usability issues, and can be carried out on a wide variety of machines and environments.

Certification

When basic testing is complete, the application is rolled over to the certification systems. A person or group of people then test it thoroughly to determine whether any bugs or problems remain to be fixed before the application is sent to the production server. This certification testing is much more thorough than the tests carried out on the testing systems and should include coverage, usage, and performance validation, which we discuss in detail later in this chapter. The certification team may perform integration checks with other systems that interact with the application, such as applications that share business COM objects or the production database. In addition, the certification team may check that the application can handle normal working stress by simulating the loads that are expected in the production environment.

The certification stage concentrates on performance and integration with existing systems. The certification environment should be fully functional, and be configured to resemble the production configuration as closely as possible. In particular, it should reproduce any multiple-network protocols, COM transports, domains, or firewalls that restrict access to particular computers. Items such as database connectivity must be configured on the certification servers the same way they are on the production servers.

NOTE

---

Testing and certification require people as well as systems. There is no reason to have a certification system if no one is available to test the application while it is on this system.

Production

The project is rolled over to the production server or servers only after the test plan has been completed and the application is proven stable. The advantages of a smoothly operating production channel are greater flexibility and scalability. For example, if certification tests determined that another production server is required to meet demand, the additional server can be configured and added. The only modification needed is to configure the application's systems on the new server.

In the meantime, the development team may already be working on the next version of the application on the development server.

Change Control Process

Developing a change control process and establishing policies are relatively easy compared with the task of enforcing them. Fortunately, Microsoft Windows NT and Microsoft Windows 2000 have built-in features to ensure that everyone follows the rules. Figure 10.1 shows the NT file system (NTFS) permissions that are appropriate for developers through the four production-channel stages. Obviously, the developers need full access to the source files on the Development server because they make the changes there. However, they are given limited access to the same files on the Testing, Certification, and Production servers so that they can't change the files without going through the change control process.

Figure 10.1 Developers' system permissions in the production channel stages

Figure 10.2 shows a simplified view of the production channel that culminates in rollover to three production servers. To illustrate the potential for problems if a change control process is not followed, suppose a company has a working Web site for which it is designing a Web-based application using Active Server Pages (ASP), COM business objects, and an SQL data server. The application has several informational Web pages containing text and graphics, including a picture of a lion.

Figure 10.2 Simplified production channel rollover process

Suppose that a developer decides that this picture is no longer needed, connects to the production Web servers, removes the link to the picture from the page, and deletes the picture from the servers, without realizing that another developer has linked to the same lion picture from a different page. Because the lion picture is no longer available, that page and any other pages linked to the picture display an error message.

Taking this scenario one step further, suppose the developer removes the picture from only the Production1 and Production2 servers, not from Production3. Now the production environment is inconsistent. A user who connects to the Web site via the Production3 server will see the lion picture, whereas a user who connects via Production1 or Production2 will not.

In the above example, removing the picture causes a minor inconvenience for a few people. The impact is potentially much greater if the same kind of unregulated change results in the deletion of a compiled business object. Any pages that use the business object will be out of service, and critical business processes may be affected. Even if the developer changes the business object instead of deleting it, other portions of the Web application might be adversely affected.

Sizing the Production Channel

When many developers are working on the same project, it quickly becomes obvious why a staged production channel is important, especially with Web-based applications. However, setting up a production channel is just as important if only one person is responsible for the entire application. The production channel enforces discipline, requires that changes to the application be tested before it is deployed in the production environment, and can provide the means to roll back changes if necessary.

The production channel example we have provided is very basic. Often an application will have multiple components such as Web pages, business objects, and data, and will be deployed on multiple servers, such as Web servers, indexing servers, transaction and queuing servers, and database servers. For the most accurate testing, each server should be emulated in the certification stage of the production channel so that the certification environment exactly mimics the production environment. As Figure 10.3 shows, the certification environment can also be scaled down for smaller applications or organizations.

Figure 10.3 Scaling down the certification environment

In small environments, the certification database may be physically stored on the production database server, but it should be a separate database that mimics the live data used in the production system. Production data should never be manipulated during certification.

In some circumstances, the entire certification environment might need to be located on a single computer. However, if the production environment consists of multiple computers, inter-computer connectivity won't be adequately tested by this certification configuration. Running multiple certification servers that are configured the same way as the production servers is more likely to identify and resolve connectivity issues, such as database connections, so that the application can be rolled smoothly to production.

CAUTION

---

Conducting performance tests that generate large amounts of network traffic on production networks can interrupt normal operation. Before executing these tests, the feasibility of isolating the testing and certification networks from the production network should be seriously considered.