11.5 Runbooks

< Day Day Up >

You may know this documentation by some other name. I have come across it at enough different client sites to assume you know that before a centralized support operation such as a data center or network control center will take over the day-to-day monitoring and fault management of your project deliverables, they expect you to produce specific documentation. The purpose of this documentation is to provide the nitty-gritty that someone who gets called in during off-hours can review and get a feel for how to go about troubleshooting the down server or intermittently crashing application, database, or network element.

Perhaps this is a trifle beneath your lot in life, but I can assure you that on every large project in which I have been involved, I have spent more time than I care to recount participating in the generation of this documentation. Requiring that the project create runbooks makes sense, but I know of few project managers who feel compiling a runbook themselves is a good use of their time. Here is the issue. At some point, the nonexistence of a runbook is going to give operations the chance to inform you that you cannot play in their sandbox until their runbook process, whatever that might be, is completed to their liking. This might sound a little bit like the arbitrariness of an income tax audit, and it very well may be, but it is something that has to get done.

Exhibit 1 lists the elements commonly found in these runbooks. However, the contents, format, and process may vary from company to company. Besides, chances are the average project manager has little idea what is involved, so Exhibit 1 is a great place to start.

Exhibit 1: Runbook Contents

Hardware information - including server make and model, network interface cards (NICs), memory, internal or attached storage, tape backup units, serial numbers.
Software information - including specific version levels for operating system (OS), tools, database or Web server brand, monitoring tools, and homegrown applications.
Application overview - a basic description of the application's business functionality; this should include user demographics or profiles (i.e., United States only).
Application architecture - how data is gathered, how it moves through the system and interacts with users and possibly other systems or platforms.
Hardware architecture - how the servers are cabled to the data center LAN, as well as to each other or peripheral devices, if applicable.
Addressing information - such as IP addresses and Domain Name System (DNS) names.
Start-up processes - when a server is powered up, the sequence of software loads leading to its production ready state.
Shutdown processes - what is the preferred method for downing the server?
Monitoring points - what systems or processes should be monitored automatically or manually to ascertain the current health of the system?
Escalation - detailed list of individuals to be called, based on the problem's severity, or the volume of end-user calls into the help desk.
Backup - if backups are required, schedule and type (e.g., incremental versus full), how to rotate tapes, offsite tape storage requirements, etc.
Disaster recovery (DR) - if this is a requirement, how it is implemented must be documented.

Chances are, your operation team will have a very specific runbook process. In most places I have worked, it is basically a word processor template with sections or chapters divided up along the lines of Exhibit 1, although at one site I had to enter the data into an online database. It is generally a requirement that the runbook process is complete and accepted by the support team before your applications can be brought into production. It is generally not the case that operations will assist very much in the creation of this documentation.

< Day Day Up >