Working in Production Environments

There are so many different levels of production environments that it is difficult to speak to them all. The principles in this chapter can be applied to any important computing environment. Because this book is about Internet architectures, we won't address anything but the infrastructure used to directly service customers over the Internet.

Because most large architectures are run by multidisciplinary teams, there tend to be more than one set of guidelines for avoiding failure.

In the end it comes down to "don't be stupid." Although this is a simple and an intuitive expectation, what is clearly "stupid" on a small architecture is often subtle on large architectures run by multiple teams. There are three reasons that teams are used to manage projects:

• The work to be done on large systems exceeds what is humanly possible for a single individual to accomplish.

• It is less expensive and easier to find individuals with a deep, focused expertise in one of the technologies that powers the system than to find individuals with deep and broad expertise across every technology in the system.

• The "key man problem"if your key man is hit by a bus (or simply quits), you must have business continuity.

All are obvious reasons, all are important, and all contribute a bit of chaos to the overall system. Through the application of good practices and the use of established tools, the chaos can be kept in check. Without the appropriate approach, working in a fast-moving production architecture is like working on a construction site without OSHA compliance and no hard hatstupid.

Although it may not seem an obvious application at first, in the end it boils down to Murphy's Lawmore specifically how to avoid it. Finagle's Law (a more generalized version of Murphy's Law) says: "Anything that can go wrong, will." However, more formally, Finagle's Law looks something like this:

((U+C+I) x (10-S))/20 x A x 1/(1-sin(F/10))

urgency (U): 1 <= U <= 9,

complexity (C): 1 <= C <= 9,

importance (I): 1 <= I <= 9,

skill (S): 1 <= S <= 9,

frequency (F): 1 <= F <= 9

This equation illustrates Finagle's Law (a.k.a. Sod's Law) more formally. Although this equation, commissioned by British Gas, may seem a bit contrived despite it regressing well for British Gas's dataset, it still provides an excellent insight into the nature of the problems architects face.