There are so many different levels of production environments that it is difficult to speak to them all. The principles in this chapter can be applied to any important computing environment. Because this book is about Internet architectures, we won't address anything but the infrastructure used to directly service customers over the Internet.
Because most large architectures are run by multidisciplinary teams, there tend to be more than one set of guidelines for avoiding failure.
In the end it comes down to "don't be stupid." Although this is a simple and an intuitive expectation, what is clearly "stupid" on a small architecture is often subtle on large architectures run by multiple teams. There are three reasons that teams are used to manage projects:
All are obvious reasons, all are important, and all contribute a bit of chaos to the overall system. Through the application of good practices and the use of established tools, the chaos can be kept in check. Without the appropriate approach, working in a fast-moving production architecture is like working on a construction site without OSHA compliance and no hard hatstupid.
Although it may not seem an obvious application at first, in the end it boils down to Murphy's Lawmore specifically how to avoid it. Finagle's Law (a more generalized version of Murphy's Law) says: "Anything that can go wrong, will." However, more formally, Finagle's Law looks something like this:
((U+C+I) x (10-S))/20 x A x 1/(1-sin(F/10))
urgency (U): 1 <= U <= 9,
complexity (C): 1 <= C <= 9,
importance (I): 1 <= I <= 9,
skill (S): 1 <= S <= 9,
frequency (F): 1 <= F <= 9
This equation illustrates Finagle's Law (a.k.a. Sod's Law) more formally. Although this equation, commissioned by British Gas, may seem a bit contrived despite it regressing well for British Gas's dataset, it still provides an excellent insight into the nature of the problems architects face.