6.1 Important HTTP Headers

only for RuBoard - do not distribute or recompile

6.1 Important HTTP Headers

To understand what makes web objects cachable (or uncachable), you need to know how HTTP presents certain information to caches. In this chapter, we will discuss five HTTP headers that are particularly important and useful for web caches: Date , Last-modified , Expires , Cache-control , and Content-length . Although some of these headers may appear in requests , for this chapter, we are interested only in response headers.

6.1.1 Date

When an origin server generates an HTTP response, it places the current time in the Date header. The time is represented in a common format and always as Greenwich Mean Time (GMT), so clients and servers around the globe do not need to worry about time zone conversions.

The Date header allows a cache or other agent to derive some useful information about a response. Obviously, you can determine the age of the response (but not the resource!) from this header. If the given date is equal to the current time, you probably have a response that just came from the origin server. On the other hand, if the date is in the past, you probably have a response that came from a cache.

You can never be absolutely sure about the Date header because you don't know if the origin server's clock is correct. We call this the clock skew problem. There is no guarantee that the origin server's clock is synchronized to your own. Either clock may be incorrectly set. If you find a Date header with a future time, then you know that either your clock or the server's clock is wrong. If the time is in the past, either a clock is out of sync, or it is a cached response. A number of new features in HTTP/1.1 are designed to eliminate ambiguities and reliance on absolute dates. For example, the Age header represents the age (in seconds) of the response. Previously, caches had to determine the age by calculating the difference between the Date header and the current time. Even though HTTP/1.1 has these new headers, you should always use NTP to keep your system clocks synchronized.

The Date header is also important to caches because it provides a basis for comparing the other timestamp headers in a response. For example, to calculate the resource's age when the response was generated, caches subtract the Date value from the Last-modified value. Caches may also need to compare the Expires value to the response date. If they are equal, or if the expires time is earlier, the response is preexpired and must be revalidated upon its next request.

RFC 2616 requires origin servers to generate a Date header for every response. In reality, a Date header is present in about 91% of responses (see Section A.3.2). The RFC also says that proxies should insert their own Date header if none exists.

6.1.2 Last-modified

The Last-modified header specifies the time when the resource was most recently modified. This timestamp is important to web caches because it is used as a cache validator. A cache says to an origin server, "I have this object with this last-modified timestamp; is it still current?" The server compares the validator timestamp with the resource's modification time. If they are the same, the resource has not been modified, and the server returns a 304 (Not Modified) response. Recall from Section 2.2, that RFC 2616 recommends a response should not be cached unless it includes either a cache validator or an explicit expiration time. The last-modified timestamp is not the only type of cache validator, however. With HTTP/1.1, entity tags (see Section 2.5.2) may be used as well.

Last-modified times are also useful in another way. As I mentioned earlier, by subtracting the date from the last-modified time, we know how old the resource was when the server generated the response. We call this the last-modified age , or LM-age. Caches may use the LM-age to estimate the stability of the resource. A young resource is considered unstable and more likely to change; an old resource, however, is more stable and less likely to change. This heuristic is often used for revalidation when a resource does not have a specific expiration time. The LM-age can also be used in a cache replacement algorithm. When choosing objects for deletion, it may be better to select the youngest ones because they have probably changed and need to be transferred again upon the next request anyway.

Unfortunately, we see Last-modified headers in responses less frequently than Date headers. The statistics in Section A.3.2, show that only about 52 percent of responses have a last-modified timestamp. RFC 2616 says that an origin server "should send a Last-Modified value if it is feasible to send one . . . ". For objects stored as plain disk files, it is trivial to retrieve the modification time from the filesystem. However, responses generated from CGI scripts, Active Server Pages, and HTML with server-side includes often do not have a last-modified timestamp.

6.1.3 Expires

The Expires header is great for caches because it tells the cache exactly how long the response may be considered fresh. A response that includes an Expires header may be reused without validation (i.e., a cache hit) until the expiration time is reached. This removes a lot of guesswork from the cache and places responsibility with the origin server. Without an Expires header, a cache may be to blame for returning an out-of-date (stale) response. By providing an expiration time, the content provider takes responsibility for ensuring users receive fresh responses.

The presence of an Expires header can also turn an otherwise uncachable response into a cachable one. For example, responses to POST requests are uncachable by default, but they can be cached if there is an Expires line in the reply headers.

Expires headers appear relatively infrequently in practice. Today, we see expiration values in about 12% of all responses (see Section A.3.2). Fortunately, the trend is increasing over time. When I started analyzing cache logs in 1995, only about 5% of responses had expiration times. But why was this number so low? The most likely reason is that, by default, web servers never send Expires headers. Another is that it can be very difficult to predict when a resource will actually change. When will you change your home page next? When will a product's price go up or down? What if you guess wrong? It is not too difficult to imagine a situation where an information provider wants or needs to retract a document before its expiration time is reached. It could be as harmless as a spelling mistake or as serious as slanderous accusations against an innocent person. Note, however, that this problem is not unique to the Web. Newspaper, television, and radio also often publish or broadcast information that later requires a correction.

Another difficult aspect of expiration times is the interface (or lack thereof) between humans and the HTTP server. Who should set expiration times? The document author or the server administrator? The answer depends on the actual HTTP server being used. Some, like Apache, provide ways for authors to insert or modify certain HTTP headers, as we'll see later. Others require configuration that only the server administrator may perform.

6.1.4 Cache-control

The Cache-control header has a number of directives, all of which are important to caching in one way or another. In this chapter, we are only concerned with the reply directives. We already talked about most of these in Section 2.2.4, but we'll go over them again here:

max-age and s-maxage

max-age is an alternate form of the Expires header. Both specify the amount of time until a response becomes stale. max-age uses relative values, while Expires uses absolute times. The max-age directive is preferred for some of the reasons already described. Another good thing about max-age is that it can change a response from uncachable to cachable. The s-maxage directive is the same, but it only applies to shared caches.

public

The public directive designates a response as cachable. Many responses are cachable by default, so public is not always needed. However, responses that are uncachable by default become cachable when this Cache-control directive is present. If you want a particular response to be cached, it is a good idea to add the public directive just to be safe.

private

When an origin server wants to allow the response to be cached by a browser but not by a proxy, it includes the private cache-control directive. Such a response must not be cached by a shared (proxy) cache, but the user 's browser may cache the response and reuse it if all other standard conditions are met.

no-cache

This directive is somewhat tricky. It is similar to the must-revalidate directive. Responses that include the no-cache directive can be stored, but they must be revalidated before reuse. The no-cache directive can also specify a list of header names , in which case those headers must be removed before storing the response.

no-store

The no-store directive is the best way to make a response uncachable. It is intended to prevent even temporary storage of information that the owner considers highly sensitive. RFC 2616 says that "the cache MUST NOT intentionally store the information in non-volatile storage, and MUST make a best-effort attempt to remove the information from volatile storage as promptly as possible after forwarding it." Some caching products might prefer to buffer incoming data on disk, which is especially useful if the client is slower than the server. This should not happen, however, if the no-store directive is present.

no-transform

By and large, caching proxies are allowed to change, or transform, the objects they serve to clients. For example, a proxy might recode images from one format to another that uses less space and less transmission time. Similarly, a cache might apply compression to text objects before giving them to clients and storing them on disk. According to RFC 2616, these sorts of transformations have been known to cause problems in some cases ”for instance, when clients and servers communicate using SOAP or XML-RPC over HTTP.

To prevent such modifications, the server includes a no-transform directive. With this directive, a proxy cache cannot change the type, length, or encoding of the object content.

must-revalidate

RFC 2616 permits caches to intentionally return stale responses in some circumstances, such as extremely poor network connectivity. The must-revalidate directive is a way for the server to prevent stale responses in those cases. In other words, this directive takes higher precedence than the user-agent and/or cache configuration. Section 14.9.4 of the RFC advises servers to send must-revalidate "if and only if failure to revalidate a request on the entity could result in incorrect operation."

must-revalidate and no-cache are quite similar. The difference is that no-cache supersedes expiration times, but must-revalidate does not. In other words, must-revalidate responses are validated only after becoming stale, whereas no-cache responses are always validated. Both should be used with extreme caution.

proxy-revalidate

proxy-revalidate is just like must-revalidate , except that it applies only to shared (proxy) caches, not to single-user (browser) caches. A response with proxy-revalidate may be reused by a browser cache without revalidation. A proxy cache, on the other hand, is required to revalidate the response before giving it to a client.

6.1.5 Content-length

The Content-length header specifies the size of the HTTP message body in a response. Knowing the body size allows a cache to determine whether it received a complete response. If a cache or client doesn't know what the body size should be, it might store a partial response and believe it received the whole thing. A future request for the object may then result in the partial response being sent as a cache hit.

There are two ways to find the end of an HTTP message body when the response doesn't have a Content-length value. The simplest approach is just to close the connection. However, since HTTP/1.1 wants connections to be persistent, it provides something called chunked transfer encoding .

Under HTTP/1.0, connections are always closed at the end of a transfer; HTTP/1.1 agents can do the same when the headers don't include a content length. Unfortunately, this makes it hard for a client to tell whether it got the whole message successfully or received just part of the message due to an error. The client doesn't know how big the message body is supposed to be. Consider the case when a proxy is reading such a response from an origin server and relaying it to the client. Remember that there are two TCP connections ”one to the client and one to the server. The proxy gets a read error, such as "connection reset by peer," during the middle of the transfer. The proxy knows the response is most likely incomplete. But how does it notify the client? HTTP does not provide any way to indicate an error condition while sending the message body. If we just close the client's connection, the client is likely to believe the response is complete. The proxy should instead perform an abortive close so the client also receives a TCP reset packet. ^[1]

^[1] To make an abortive close with Berkeley sockets, use the SO_LINGER option with a linger time of 0, and then close the socket.

HTTP/1.1 has persistent connections designed to reduce retrieval latencies by eliminating the TCP connection handshake for every request. Though closing a persistent connection to indicate end-of-message is allowed, it is not desirable. To support unknown message sizes and persistent connections together, HTTP/1.1 uses a chunked transfer encoding. The chunked transfer encoding divides the message body into a sequence of chunks , each of which includes the chunk size. This allows the server to tell the client, "Here are N more bytes of data, but it's not the end yet." The server indicates the end-of-message with a chunk of size of 0. If a client doesn't receive the 0- sized chunk, it should assume the response is incomplete.

The Content-length header is useful to caches in a number of other ways as well. It can be used to preallocate cache storage space or calculate the amount of data left to be transferred. When a user aborts an HTTP transfer before it has finished, a cache may want to finish the download anyway. If resources have been used to transfer 95% of an object, the cache may just as well get the remaining 5%, especially if there is a chance it could result in a cache hit. The content length is an important factor in deciding whether to finish an aborted transfer.

only for RuBoard - do not distribute or recompile