Section 2.10. Unrelated Miscellanea

2.10. Unrelated Miscellanea

We were going to call this section "Oh, and by the way," but that seemed like a really weird heading.

2.10.1. Protect Your Career

One of the reasons that backups are unpopular is that people are worried that they might get fired if they do them wrong. People do get in trouble when restores don't go right, but following the suggestions in this section will help you protect yourself from "recovery failure fallout."

2.10.1.1. Self-preservation: Document, document, document

Have you ever tried to go on vacation? If you're the only one who understands the restore process or the organization of your media, you can bet that you will be called if a big restore is required. Backups are one area of system administration in which inadequate documentation can really get you in trouble. It's hard to go on vacation, get promoted, or do anything that would pull you away from the magical area that only you know. Your backups and restores should be documented to the point that any system administrator can follow them step by step in your absence. That is actually a good way to test your documentation: have someone else try to use it.

The opposite of good documentation is, of course, bad or nonexistent documentation. Bad documentation is the surest way to help you find a new job. If you do ever manage to take a real vacation in which you don't carry a beeper, check your voice mail, or check your emailin short, watch out: Murphy's Law governs vacations as well. You can guarantee that you, or more accurately, your coworkers, will have a major outage that week. If they crash and burn because you left them no guidelines for how to perform a restore, they will be looking for you when you return. You will not be a popular person, and you just might find yourself combing through the want ads.

You Can Run, But You Can't Hide

On a number of occassions, a lack of documentation caused me to lose personal time. I remember one vacation during which I spent two to three hours on the phone every day. I remember spending long nights in computer rooms because no one knew which button to press next. But none of those memories is as strong as the time when my wonderful daughter, Nina, was born. Right about now, you're probably saying "Aaah, that's sweet." It's not what you think. Yes, she and my other daughter, Marissa, have given me a whole other reason to get up every day, but that's not what this story's about.

The hospital in which my wife gave birth was about two blocks from my office building. I knew that. My coworkers knew that. (Anybody who looked out the window knew that!) The day Nina was born, we lost a major filesystem. I knew it was on a backup volume, and I knew I was off duty. I left my beeper, which is normally welded to my side, at home. I did not call in to work. I knew the process was documented. The problem was that they weren't reading the documentation. "Call Curtis!" I was standing in my wife's hospital room, talking about our wonderful child, when the phone rang. Those guys tracked me down and called me in the hospital! They asked me to come in, but because I knew the system was documented well, the answer was no! (I think I actually hung up without saying another word.) This is an example of the lengths to which people will go to find you if you don't have proper documentation or if they have not been shown how to use it.

Documentation is also an important method of letting your internal customers know what you are doing. For example, if you skip certain types of files, drives, or filesystems, it is good if you let people know that. I remember at least one very long conversation with a user who really didn't want to hear that I didn't back up /tmp: "I never knew that tmp was short for temporary!"

2.10.1.2. Strategy: Make backups an integral part of the installation process

When a new system comes in the door, someone makes sure that it has power. Someone is responsible for the network connection, assigning an IP address, adding it to the NIS configuration, and installing the appropriate patches. All those things happen because things don't work if they don't happen. Unfortunately, no one notices if you don't add the machine to your backup list. That is, of course, until it crashes, and they need something restored. You have the difficult task of making something as "unimportant" as backups become just as natural as adding the network connection.

A new system coming in the door is usually the best test machine for a complete server recovery/duplication test. Not many people miss a machine they don't have yet.

The only way that this is going to happen is if you become very involved in the whole process. Perhaps you are a junior person, and you never sit in on the planning meetings because you don't understand what's going on. Perhaps you do understand what's going on but just hate to go to meetings. So do I. If you don't want to attend every meeting, just make sure that someone is looking out for your interests in those meetings. Maybe an ex-backup operator who goes to the meetings and is sympathetic to your cause. Have briefings with her, and remind her to make sure that backup needs are being addressed or to let you know about any new systems that are coming down the pike. Occasionally, go to the meetings yourself, and make sure that people know that you and your backups exist; hopefully, they'll remember that the next time they think about installing a new system without telling you. Never count on this happening, though. You've got to be ever-diligent, looking for new systems in need of backup.

New installations are not the only thing that can affect your backups. New versions of the operating system, new patches, and new database versions all can break your backups. Most system administrators bring in a new version of their operating system or database and run it on a new box or development box before they commit it to production. Make sure that your backup programs run on the new platform as well. I can think of a number of times that new versions broke my backups. Here are a few examples:

• When HP-UX 10 came out, it supported file sizes greater than 2 GB, but the dump manpage said that dump does not back up a filesystem with large files.

• Oracle has changed a number of times. Sometimes it maintains backward compatibility; sometimes it doesn't.

• The Windows Encrypting File System couldn't be backed up by some backup systems when it first came out.

• Mac OS versions (post Mac OS X) always seem to be a little ahead of the backup curve. The methods you used to back up or image the last version no longer work in the new version, giving rise to many unofficial versions of tools that actually work.

The longer I think about it, the more stories I come up with. If you've been doing this for a while, I'm sure you have a few of your own. Suffice it to say that OS and application upgrades can and do cause problems for the backup person. Test, test, test!

2.10.2. Get the Money Your Backups Need

This final section has absolutely nothing to do with backups. It has to do with politics, budgeting, money, and cost justifications. I know that sometimes it sounds as if I think that backups get no respect. Maybe you work at Utopia, Inc., where the first thing they think about is backups. The rest of us, on the other hand, have to fight for every volume, drive, and piece of software that we buy in order to accomplish this increasingly difficult task of getting it all on a backup volume.

Getting the money you need to accomplish your task can sometimes be very difficult. Once a million dollar computer is rolled in the door and uncrated, how do you tell the appropriate department that the small, standalone backup drive that came with it just isn't going to cut it? Do you know how many hoops they went through to spend a million dollars on one machine? You want them to spend how much more?

2.10.2.1. Be ready

The first thing is to be prepared. Be ready to justify what you need. Be ready with information such as:

• Statistics on recoveries that you have performed.

• Any numbers that you have on what downtime and lost data would cost the company, including any recovery-time-objective and recovery-point-objective requirements from business units (RTO and RPO are covered in more detail in Chapter 24).

• Numbers that demonstrate how a purchase would help reduce staff costs.

• Numbers that demonstrate how the current backup system is being negatively impacted by growth or new applications.

• Cost comparisons between the one-time cost of a newer storage system and the continuing cost of the manual labor required to swap volumes every night (be prepared to explain how a larger jukebox or virtual tape library reduces the chance for human error and how that helps the company as well).

• A documented policy that every new gigabyte results in a surcharge of a certain amount of money.

A well-designed presentation of what service your backups will provide and the speed at which you can recover data. (Don't commit yourself to unrealistic restore times, but if the new system can significantly improve restore times, show that!)

• A letter all ready to go, with which your boss is comfortable, that explains very matter-of-factly what the company can expect if it doesn't provide the funding you need.

2.10.2.2. Make a formal presentation

The more expensive your solution, the more important it is that you make a formal presentation, especially if you are in a corporate environment. A formal technical presentation has three parts: an executive summary, an overview section that goes into more detail, and a technical specifications section for those who are really interested:

Executive summary

This should be one page and should explain on a very high level what is proposed in the rest of the presentation. Global figures and broad descriptions are good; do not go into too much detail. This is made for the VP, who needs to do the final sign-off but has 20 other presentations just like yours to review at the same time. Basically, state the current problem, and describe your solution.

Overview section

Go into detail in this section. Use plenty of section headings, allowing your readers to read ahead or skim over it if they like. Headings also allow the people who read only the executive summary to look up any specific area they are not clear on. The outline of the general section should match that of the executive summary. You can include references to other publications, such as magazine reviews of a particular product, but do not quote them in detail. If they are relevant, you can attach copies in the technical section. Make sure you demonstrate that you have thought this through and that it is not just a stopgap measure. Put a high-level comparison between the option you chose and the other options available, and explain why you chose the one you did. Describe how it allows for future growth and how much growth it will allow before you must reconsider. Also explain what plans you have for the old methodology and what conversion method you are going to use, such as running both in parallel for a while. Tables are also good. If you can use real numbers, it is much more effective: just make sure you can back them up. If anyone believes that the numbers are made up, it will totally invalidate the report. Try to compare the up-front cost of your solution with the surprise cost of lost data.

Technical specifications

Go wild. If anyone has made it this far, they're either really interested or a true computer techie just like you! If this report is the cost justification for a new backup drive, find a table that compares the relative cost per megabyte of all the various options. Include hard numbers and any white papers that are included with the proposed product. If you think it is relevant, but possibly too long and boring, this is the place to put it.