1.1 What Is Special about Real-Time Systems?

If you read the popular computer press, you would come away with the impression that most computers sit on a desktop (or lap) and run Windows. In terms of the numbers of deployed systems, embedded real-time systems are orders of magnitude more common than their more visible desktop cousins. A tour of the average affluent American home might find one or even two standard desktop computers but literally dozens of smart consumer devices, each containing one or more processors. From the washing machine and microwave oven to the telephone, stereo, television, and automobile, embedded computers are everywhere. They help us evenly toast our muffins and identify mothers-in-law calling on the phone. Embedded computers are even more prevalent in industry. Trains, switching systems, aircraft, chemical process control, and nuclear power plants all use computers to safely and conveniently improve our productivity and quality of life.^[1]

^[1] Not to mention that they also keep a significant number of us gainfully employed.

The software for these embedded computers is more difficult to construct than software for the desktop. Real-time systems have all the problems of desktop applications plus many more. Systems that are not real-time do not concern themselves with timelines, robustness, or safety at least not to nearly the same extent as real-time systems. Real-time systems often do not have a conventional computer display or keyboard, but lie at the heart of some apparently noncomputerized device. The user of these devices may never be aware of the CPU embedded within, making decisions about how and when the system should act. The user is not intimately involved with such a device as a computer per se, but rather as an electrical or mechanical appliance that provides services. Such systems must often operate for days or even years without stopping, in the most hostile environments. The services and controls provided must be autonomous and timely. Frequently, these devices have the potential to do great harm if they fail.

Real-time systems encompass all devices with performance constraints. Hard deadlines are performance requirements that absolutely must be met. A missed deadline constitutes an erroneous computation and a system failure. In these systems, late data is bad data. Soft real-time systems are characterized by time constraints which can (a) be missed occasionally, (b) be missed by small time deviations, or (c) occasionally skipped altogether. Normally, these permissible variations are stochastically characterized. Another common definition for soft real-time systems is that they are constrained only by average time constraints (examples include on-line databases and flight reservation systems), although such constraints actually refer to throughput requirements rather than the timeliness of specific actions. In soft real-time systems, late data may still be good data, depending on some measure of the severity of the lateness. The methods presented in this text may be applied to the development of all performance-constrained systems, hard and soft alike. When we use the term real-time alone, we are specifically referring to hard real-time systems. In actuality, most real-time systems are a mixture of hard and soft real-time constraints, together with some requirements that have no timeliness requirements whatsoever. It is common to treat these different aspects separately, although when present, the hard real-time constraints tend to dominate the design.

An embedded system contains a computer as part of a larger system and does not exist primarily to provide standard computing services to a user. A desktop PC is not an embedded system, unless it is within a tomographical imaging scanner or some other device. A computerized microwave oven or VCR is an embedded system because it does no standard computing. In both cases, the embedded computer is part of a larger system that provides some noncomputing feature to the user, such as popping corn or showing Schwarzenegger ripping telephone booths from the floor.^[2]

^[2] Commando, a heart-warming tale if ever there was one.

Most real-time systems interact directly with electrical devices and indirectly with mechanical ones. Frequently, custom software, written specifically for the application, must control or access the device. This is why real-time programmers have the reputation of being "bare metal code pounders." You cannot buy a standard device driver or Windows VxD to talk to custom hardware components. Programming these device drivers requires very low-level manipulation. This kind of programming requires intimate knowledge of the electrical properties and timing characteristics of the actual devices.

Virtually all real-time systems either monitor or control hardware, or both. Sensors provide information to the system about the state of its external environment. Medical monitoring devices, such as electrocardiography (ECG) machines, use sensors to monitor patient and machine status. Air speed, engine thrust, attitude, and altitude sensors provide aircraft information for proper execution of flight control plans. Linear and angular position sensors sense a robot's arm position and adjust it via DC or stepper motors.

Many real-time systems use actuators to control their external environment or to guide some external processes. Flight control computers command engine thrust and wing and tail control surface orientation so that the aircraft follows the intended flight path. Chemical process control systems control when, what kind, and the amounts of different reagents are added to mixing vats. Pacemakers make the heart beat at appropriate intervals with electrical leads attached to the walls inside the (right-side) heart chambers.

Naturally, most systems containing actuators also contain sensors. While there are some open loop control systems,^[3] the majority of control systems use environmental feedback to ensure that control loop is acting properly.

^[3] An open loop system is one in which feedback about the success of the performed action is not used to control the action. A closed loop system is one in which the action is monitored and that sensory data is used to modify future actions.

Standard computing systems react almost entirely to users and little else.^[4] Real-time systems, on the other hand, may interact with the user, but have more concern for interactions with their sensors and actuators.

^[4] It is true that behind the scenes even desktop computers must interface with printers, mice, keyboards, and networks. The point is that they do this only to facilitate the user's whim.

One of the problems that arise with environmental interaction is that the universe has an annoying habit of disregarding our opinions of how and when it ought to behave. The order and arrival times of external events are frequently unpredictable. The system must react to events when they occur rather than when it might be convenient. An ECG monitor must alarm quickly following the cessation of cardiac activity if it is to be of value. The system cannot delay alarm processing until later that evening when the processor load is lower. Many hard real-time systems are reactive in nature, and their responses to external events must be tightly bounded in time. Control loops, as we shall see later, are very sensitive to time delays. Delayed actuations destabilize control loops.

Most real-time systems do one or a small set of high-level tasks. The actual execution of those high-level tasks requires many simultaneous lower-level activities. This is called concurrency. Since single processor systems can only do a single thing at a time, they implement a scheduling policy that controls when tasks execute. In multiple processor systems, true concurrency is achievable since the processors execute asynchronously. Individual processors within such systems schedule many threads pseudoconcurrently (executing a single thread at a time, but switching among them according to some scheduling policy) as well.

Embedded systems are usually constructed with the least expensive (and therefore less powerful) computers able to meet the functional and performance requirements. Real-time systems ship the hardware along with the software as part of a complete system package. Because many products are extremely cost sensitive, marketing and sales concerns push for using smaller processors and less memory. Providing smaller CPUs with less memory lowers the manufacturing cost. This per-shipped-item cost is called recurring cost, because it recurs with each manufactured device. Software has no significant recurring cost all the costs are bound up in development, maintenance, and support activities, making it appear to be free.^[5] This means that most often choices are made that decrease hardware costs while increasing software development costs.

^[5] Unfortunately, many companies opt for decreasing (primarily hardware) recurring costs without considering all of the development cost ramifications, but that's fodder for another book.

Under UNIX, a developer needing a big array might just allocate space for 1 million floats with little thought for the consequences. If the program doesn't use all that space who cares? The workstation has hundreds of megabytes of RAM and gigabytes of virtual memory in the form of hard disk storage. The embedded systems developer cannot make these simplifying assumptions. He or she must do more with less, resulting in convoluted algorithms and extensive performance optimization. Naturally, this makes the real-time software more complex and expensive to develop and maintain.

Real-time developers often use tools hosted on PCs and workstations, but targeted to smaller, less capable computer platforms. This means that they must use cross-compiler tools, which are often more temperamental (i.e., buggy) than the more widely used desktop tools. Additionally, the hardware facilities available on the target platform such as timers, A/D converters, and sensors cannot be easily simulated accurately on a workstation. The discrepancy between the development and the target environments adds time and effort for the developer wanting to execute and test his or her code. The lack of sophisticated debugging tools on most small targets complicates testing as well. Small embedded targets often do not even have a display on which to view error and diagnostic messages.

Frequently, real-time developers must design and write software for hardware that does not yet exist. This creates very real challenges since developers cannot validate their understanding of how the hardware functions, even though, according to my experience, the hardware never functions exactly as specified. Integration and validation testing become more difficult and time consuming.

Embedded real-time systems must often run continuously for long periods of time. It would be awkward to have to reset your flight control computer because of a GPF^[6] while in the air above Newark. The same applies to cardiac pacemakers, which last up to 10 years after implantation. Unmanned space probes must function properly for years on nuclear or solar power supplies. This is different from desktop computers that may be frequently reset at least daily. It may be acceptable to have to reboot your desktop PC when an application crashes the operating system, but it is much less acceptable to have to reboot a life support ventilator or the control avionics of a commercial passenger jet.

^[6] General Protection Fault, a term that was introduced to tens of millions of people with Microsoft's release of Windows 3.1.

Embedded system environments are often adverse and computer-hostile. In surgical operating rooms, electrosurgical units create intense electrical arcs to cauterize incisions. These produce extremely high EMI (electromagnetic interference) and can physically damage unprotected computer electronics. Even if the damage is not permanent, it is possible to corrupt memory storage, degrading performance or inducing a systems failure.

Apart from increased reliability concerns, software is finding its way ever more frequently into safety systems. Medical devices are perhaps the most obvious safety related computing devices, but computers control many kinds of vehicles such as aircraft, spacecraft, trains, and even automobiles. Software controls weapons systems and ensures the safety of nuclear power and chemical plants. There is compelling evidence that the scope of industrial and transportation accidents is increasing [1,2].^[7]

^[7] It is not a question of whether or not developers of safety-critical software are paranoid. The real question is "Are they paranoid enough?"

For all these reasons, developing real-time software is generally much more difficult than developing other software. The development environments traditionally have had fewer tools, and the ones that exist are often less capable than those for desktop environments or for "Big Iron" mainframes. Embedded targets are slower and have less memory, yet must still perform within tight timing and performance constraints. These additional concerns translate into more complexity for the developer, which means more time, more effort, and (unless we're careful indeed) more defects than standard desktop software of the same size. Fortunately, advances have been made in the development tools, approaches, and processes, which is what this book is all about.