With the increasing software automation of modern society, there are some hard questions coming about how much we should trust the software that runs our mission-critical systems. Do we trust the software more than its human minders? Should the software monitor and prevent human error or should human flexibility be put at a premium over and above software's rigidity?
When the Airbus A320 commercial airliner was introduced in the early 1990s, it had extensive computer-controlled aviation systems, including the first fly-by-wire system on a commercial aircraft. The on-board computers also had preprogrammed restrictions on what the pilots were allowed to do with the plane. This "guardian angel" behavior, known more formally as the Flight Envelope Protection System, is actually considered to be more revolutionary than the fly-by-wire controls. The A320's designers decided that their judgment about the plane's limits was more important than any pilot's judgment.
Imagine, for instance, that the A320 pilot has to perform a violent avoiding maneuver upon seeing another plane nearby. No matter how hard the pilot manipulates the sidestick, the computers will not allow the airframe to sustain a stress of greater than 2.5G. This allows the pilot to perform the evasion maneuver as fast and as violently as possible, knowing that the software will prevent his or her inputs from actually damaging the plane. This is similar to an automated braking system on a car that allows you to brake as hard as you wish, because the software prevents the wheels from locking up and sending your car into a skid.
While the 2.5G stress limit allowed by the A320 appears to be generous and allows some quite violent maneuvers, there is at least one precedent for exceeding this figure. In February 1985, China Airlines flight 006 was cruising at 41,000 feet some 300 miles northwest of San Francisco. A combination of power loss and undue reliance on the autopilot led to a sideways roll followed by a near-vertical dive and an inverted spin. For 3 minutes the plane plunged nearly 6 miles before the captain was able to pull out of the dive at 9,500 feet. In doing so, he measurably warped the wings and caused several million dollars of other structural damage. But he saved the airplane and its passengers at the cost of pulling an estimated 5.5G, well beyond the 747's safety limits. The evidence is that if the plane had been an A320, the software might have prevented the pilots from pulling out of the dive. On the other hand, it is possible that the computer software might have prevented the problem in the first place.