10.6 Internet Cache Protocol

ICP is a lightweight object location protocol invented as a part of the Harvest project. ^[1] An ICP client sends a query message to one or more ICP servers, asking if they have a particular URI cached. Each server replies with an ICP_HIT , ICP_MISS , or other type of ICP message. The ICP client uses the information in the ICP replies to make a forwarding decision.

^[1] For more information, see the following papers: "A Hierarchical Internet Object Cache," by Danzig, Chankhunthod, et al, USENIX Annual Technical Conference, 1995 , and "The Harvest information discovery and access system," by C. Mic Bowman, Peter B. Danzig, Darren R. Hardy, Udi Manber, and Michael F. Schwartz, Proceedings of the Second International World Wide Web Conference.

In addition to predicting cache hits, ICP is also useful for providing hints about network conditions between Squid and the neighbor. ICP messages are similar to ICMP pings in this regard. By measuring the query/response round-trip time, Squid can estimate network congestion. In the extreme case, ICP messages may be lost, indicating that the path between the two is down or congested . From this, Squid decides to avoid the neighbor for that particular request.

Increased latency is perhaps the most significant drawback to using ICP. The query/response exchange takes some time. Caching proxies are supposed to decrease response time, not add more latency. If ICP helps us discover cache hits in neighbors, then it may lead to an overall reduction in response time. See Section 10.10 for a description of the query algorithm implemented in Squid.

ICP also suffers from a number of design deficiencies: security, scalability, false hits, and the lack of a request method. The protocol doesn't include any security features. In general, Squid can't verify that an ICP message is authentic ; it relies on address-based access controls to filter out unwanted ICP messages.

ICP has poor scaling properties. The number of ICP messages (and bandwidth) grows in proportion to the number of neighbors. Unless you use some kind of partitioning scheme, this places a practical limit on the number of neighbors you can have. I don't recommend having more than five or six neighbors.

ICP queries contain only URIs, with no additional request headers. This makes it difficult to predict cache hits with perfect accuracy. An HTTP request may include additional headers (such as Cache-Control : max-stale=N ) that turn a cache hit into a cache miss . These false hits are particularly awkward for sibling relationships.

Also missing from the ICP query message is the request method. ICP assumes that all queries are for GET requests . A caching proxy can't use ICP to locate cached objects for non- GET request methods .

You can find additional information about ICP by reading:

• My book Web Caching (O'Reilly)

• RFCs 2186 and 2187

• My article with kc claffy: "ICP and the Squid Web Cache" in the IEEE Journal on Selected Areas in Communication , April 1998

• http://icp.ircache.net/

10.6.1 Being an ICP Server

When you use the icp_port directive, Squid automatically becomes an ICP server. That is, it listens for ICP messages on the port you've specified, or port 3130 by default. Be sure to tell your sibling and/or child caches if you decide to use a nonstandard port.

By default, Squid denies all ICP queries. You must use the icp_access rule list to allow queries from your neighbors. It's usually easiest to do this with src ACLs. For example:

 acl N1 src 192.168.0.1 acl N2 src 172.16.0.2 acl All src 0/0 icp_access allow N1 icp_access allow N2 icp_access deny All 

Note that only ICP_QUERY messages are subject to the icp_access rules. ICP client functions, such as sending queries and receiving replies, don't require any special access controls. I also recommend that you take advantage of your operating system's packet filtering features (e.g., ipfw , iptables , and pf ) if possible. Allow UDP messages on the ICP port from your trusted neighbors and deny them from all other hosts .

When Squid denies an ICP query due to the icp_access rules, it sends back an ICP_DENIED message. However, if Squid detects that more than 95% of the recent queries have been denied , it stops responding for an hour . When this happens, Squid writes a message in cache.log :

 WARNING: Probable misconfigured neighbor at foo.web-cache.com WARNING: 150 of the last 150 ICP replies are DENIED WARNING: No replies will be sent for the next 3600 seconds 

If you see this message, you should contact the administrator responsible for the misconfigured cache.

Squid was designed to answer ICP queries immediately. That is, Squid can tell whether or not it has a fresh, cached response by checking the in-memory index. This is also why Squid is a bit of a memory hog. When an ICP query comes in, Squid calculates the MD5 hash of the URI and looks for it in the index. If not found, Squid sends back an ICP_MISS message. If found, Squid checks the expiration time. If the object isn't fresh, Squid returns ICP_MISS . For fresh objects, Squid returns ICP_HIT .

By default, Squid logs all ICP queries (but not responses) to access.log . If you have a lot of busy neighbors, your log file may become too large to manage. Use the log_icp_queries directive to prevent logging of these queries. Although you'll lose the detailed logging for ICP, you can still get some aggregate stats via the cache manager (see Section 14.2.1.24).

If you have sibling neighbors, you'll probably want to use the miss_access directive to enforce the relationship. It specifies an access rule for cache misses. It is similar to http_access but is checked only for requests that must be forwarded. The default rule is to allow all cache misses. Unless you add some miss_access rules, any sibling cache can become a child cache and forward cache misses through your network connection, thus stealing your bandwidth.

Your miss_access rules can be relatively simple. Don't forget to include your local clients (i.e., web browsers) as well. Here's a simple example:

 acl Browsers src 10.9.0.0/16 acl Child1 src 172.16.3.4 acl Child2 src 192.168.2.0/24 acl All src 0/0 miss_access allow Browsers miss_access allow Child1 miss_access allow Child2 miss_access deny All 

Note that I haven't listed any siblings here. The child caches are allowed to request misses through us, but the siblings are not. Their cache miss requests are denied by the deny All rule.

10.6.1.1 The icp_hit_stale directive

One of the problems with ICP is that it returns ICP_MISS for cached but stale responses. This is true even if the response is stale, but valid (such that a validation request returns "not modified"). Consider a simple hierarchy with a child and two parent caches. An object is cached by one parent but not the other. The cached response is stale, but unchanged, and needs validation. The child's ICP query results in two ICP_MISS replies. Not knowing that the stale response exists in the first parent, the child forwards its request to the second parent. Now the object is stored in both parents, wasting resources.

You might find the icp_hit_stale directive useful in this situation. It tells Squid to return an ICP_HIT for any cached object, even if it is stale. This is perfectly safe for parent relationships but can create problems for siblings.

Recall that in a sibling relationship, the client cache is only allowed to make requests that are cache hits. Enabling the icp_hit_stale directive increases the number of false hits because Squid must validate the stale responses. Squid normally handles false hits by adding the Cache-Control: only-if-cached directive to HTTP requests sent to siblings. If the sibling can't satisfy the HTTP request as a cache hit, it returns an HTTP 504 (Gateway Timeout) message instead. When Squid receives the 504 response, it forwards the request again, but only to a parent or the origin server.

It makes little sense to enable icp_hit_stale for sibling relationships if all the false hits must be reforwarded. This is where the ICP client's allow-miss option to cache_peer becomes useful. When the allow-miss option is set, Squid omits the only-if-cached directive in HTTP requests it sends to siblings.

If you enable icp_hit_stale , you also need to make sure that miss_access doesn't deny cache-miss requests from siblings. Unfortunately, there is no way to make Squid allow only cache-misses for cached, stale objects. Allowing cache misses for siblings also leaves your cache open to potential abuse. The administrator of the sibling cache may change it to a parent relationship without your knowledge or permission.

10.6.1.2 The ICP_MISS_NOFETCH feature

The command-line -Y option to Squid causes it to return ICP_MISS_NOFETCH , instead of ICP_MISS , while rebuilding the in-memory indexes. ICP clients that receive ICP_MISS_NOFETCH responses should not send HTTP requests for those objects. This reduces the load placed on Squid and allows the rebuild process to complete sooner.

10.6.1.3 The test_reachability directive

If you enable the netdb feature (see Section 10.5), you might also be interested in enabling the test_reachability directive. The goal behind it is to accept only requests for origin servers Squid can reach. Enabling test_reachability causes Squid to return ICP_MISS_NOFETCH , instead of ICP_MISS , for origin server sites that don't respond to ICMP pings. This can help reduce the number of failed HTTP requests and increase the chance that the end user receives the data promptly. However, a significant percentage of origin server sites intentionally filter out ICMP traffic. For these, Squid returns ICP_MISS_NOFETCH even though an HTTP connection would succeed.

Enabling test_reachability also causes Squid to make netdb measurements in response to ICP queries. If Squid doesn't have any RTT measurements for the origin server in question, it sends out an ICMP ping (subject to the rate limiting mentioned previously).

10.6.2 Being an ICP Client

First, you must use the cache_peer directive to define your neighbor caches. See the section Section 10.3.

Second, you must also use the icp_port directive, even if your Squid is only an ICP client. This is because Squid uses the same socket for sending and receiving ICP messages. It is perhaps a bad design decision in retrospect. If you are a client only, use icp_access to block queries. For example:

 acl All src 0/0 icp_access deny All 

Squid sends ICP queries to its neighbors for most requests by default. See Section 10.10 for a complete description of the way that Squid decides when, and when not, to query its neighbors.

After sending one or more queries, Squid waits some amount of time for ICP replies to arrive . If Squid receives an ICP_HIT from one of its neighbors, it forwards the request there immediately. Otherwise, Squid waits until all replies arrive or until a timeout occurs. The timeout is calculated dynamically, based on the following algorithm.

Squid knows the average round-trip time between itself and each neighbor, taken from recent ICP transactions. When querying a group of neighbors, Squid calculates the mean of all the neighbor ICP RTTs, and then doubles it. In other words, the query timeout is twice the mean of RTTs for each neighbor queried. Squid ignores neighbors that appear to be down when calculating the timeout.

In some cases, the algorithm doesn't work well, especially if you have neighbors with widely varying RTTs. You can change the upper limit on the timeout using the maximum_icp_query_timeout directive. Alternatively, you can make Squid always use a constant timeout value with the icp_query_timeout directive.

10.6.2.1 cache_peer options for ICP clients

weight= n allows you to weight parent caches artificially when using ICP/HTCP. It comes into play only when all parents report a cache miss. Normally, Squid selects the parent whose reply arrives first. In fact, it remembers which parent has the best RTT for the query. Squid actually divides the RTT by the weight, so that a parent with weight=2 is treated as if it's closer to Squid than it really is.

no-query disables ICP/HTCP for the neighbor. That is, your cache won't send any queries to the neighbor for cache misses. It is often used with the default option.

closest -only refers to one of Squid's netdb features. It instructs Squid to select the parent based only on netdb RTT measurements and not the order in which replies arrive. This option requires netdb at both ends.

10.6.2.2 ICP and netdb

As mentioned in the section Section 10.5, netdb is mostly used with ICP queries. In this section, we'll follow all the steps involved in this process.

1. A Squid cache, acting as an ICP client, prepares to send a query to one or more neighbors. If query_icmp is set, Squid sets the SRC_RTT flag in the ICP query. This informs the ICP server that Squid would like to receive an RTT measurement in the ICP reply.

2. The neighbor receives the query with the SRC_RTT flag set. If the neighbor is configured to make netdb measurements, it searches the database for the origin server hostname. Note that the neighbor doesn't query the DNS for the origin server's IP address. Thus, it finds a netdb entry only if that particular host has already been measured.

3. If the host exists in the netdb database, the neighbor includes the RTT and hop count in the ICP reply. The SRC_RTT flag is set in the reply to indicate the measurement is present.

4. When Squid receives the ICP reply with the SRC_RTT flag set, it extracts the RTT and hop count. These are added to the local netdb so that, in the future, Squid knows the approximate RTT from the neighbor to the origin server.

5. An ICP_HIT reply causes Squid to forward the HTTP request immediately. If, on the other hand, Squid receives only ICP_MISS replies, it selects the parent with the smallest (nonzero) measured RTT to the origin server. The request is logged to access.log with CLOSEST_PARENT_MISS .

6. If none of the parent ICP_MISS replies contain RTT values, Squid selects the parent whose ICP reply arrived first. In this case, the request is logged with FIRST_PARENT_MISS . However, if the closest-only option is set for a parent cache, Squid never selects it as a "first parent." In other words, the parent is selected only if it is the closest parent to the origin server.

10.6.3 Multicast ICP

As you already know, ICP has poor scaling properties. The number of messages is proportional to the number of neighbors. Because Squid sends identical ICP_QUERY messages to each neighbor, you can use multicast to reduce the number of messages transmitted on the network. Rather than send N messages to N neighbors, Squid sends one message to a multicast address. The multicast routing infrastructure makes sure each neighbor receives a copy of the message. See the book Interdomain Multicast Routing: Practical Juniper Networks and Cisco Systems Solutions by Brian M. Edwards, Leonard A. Giuliano, and Brian R. Wright (Addison Wesley) for more information on the inner workings of multicast.

Note that ICP replies are always sent via unicast. This is because ICP replies may be different (e.g., hit versus miss) and because the unicast and multicast routing topologies may differ . Because ICP is also used to indicate network conditions, an ICP reply should follow the same path an HTTP reply takes. The bottom line is that multicast only reduces message counts for queries.

Historically, I've found multicast infrastructure unstable and unreliable. It seems to be a low priority for many ISPs. Even though it works one day, something may break a few days or weeks later. You're probably safe using multicast entirely within your own network, but I don't recommend using it for ICP on the public Internet.

10.6.3.1 Multicast ICP server

A multicast ICP server joins one or more multicast group addresses to receive messages. The mcast_groups directive specifies these group addresses. The values must be multicast IP addresses or hostnames that resolve to multicast addresses. The IPv4 multicast address range is 224.0.0.0-239.255.255.255. For example:

 mcast_groups 224.11.22.45 

An interesting thing about multicast is that hosts, rather than applications, join a group. When a host joins a multicast group, it receives all packets that are transmitted to that group. This means that you need to be a little bit careful when selecting a multicast group to use for ICP. You don't want to select an address that's already being used by another application. When this kind of group overlap occurs, the two groups become joined and receive each other's traffic.

10.6.3.2 Multicast ICP client

Multicast ICP clients transmit queries to one or more multicast group addresses. Thus, the hostname argument of the cache_peer line must be, or resolve to, a multicast address. For example:

 cache_peer 224.0.14.1 multicast 3128 3130 ttl=32 

The HTTP port number (e.g., 3128) is irrelevant in this case because Squid never makes HTTP connections to a multicast neighbor.

Realize that multicast groups don't have any access controls. Any host can join any multicast group address. This means that, unless you're careful, others may be able to receive the multicast ICP queries sent by your Squid. IP multicast has two ways to prevent packets from traveling too far: TTLs and administrative scoping. Because ICP queries may carry sensitive information (i.e., URIs that your users access), I recommend using an administratively scoped address and properly configured routers. See RFC 2365 for more information.

The ttl=n option is for multicast neighbors only. It is the multicast TTL value to use for ICP queries. It controls how far away the ICP queries can travel. The valid range is 0-128. A larger value allows the multicast queries to travel farther, and possibly be intercepted by outsiders. Use a lower number to keep the queries close to the source and within your network.

Multicast ICP clients must also tell Squid about the neighbors that will be responding to queries. Squid doesn't blindly trust any cache that happens to send an ICP reply. You must tell Squid about legitimate , trusted neighbors. The multicast- responder option to cache_peer identifies such neighbors. For example, if you know that 172.16.2.3 is a trusted neighbor on the multicast group, you should add this line to squid.conf :

 cache_peer 172.16.3.2 parent 3128 3130 multicast-responder 

You can, of course, use a hostname instead of an IP address. ICP replies from foreign (unlisted) neighbors are ignored, but logged in cache.log .

Squid normally expects to receive an ICP reply for each query that it sends. This changes, however, with multicast because one query may result in multiple replies. To account for this, Squid periodically sends out "probes" on the multicast group address. These probes tell Squid how many servers are out there listening. Squid counts the number of replies that arrive within a certain amount of time. That amount of time is given by the mcast_icp_query_timeout directive. Then, when Squid sends a real ICP query to the group, it adds this count to the number of ICP replies to expect.

10.6.3.3 Multicast ICP example

Since multicast ICP is tricky, here's another example. Let's say your ISP has three parent caches that listen on a multicast address for ICP queries. The ISP needs only one line in its configuration file:

 mcast_groups 224.0.14.255 

The configuration for you (the child cache) is a little more complicated. First, you must list the multicast neighbor to which Squid should send queries. You must also list the three parent caches with their unicast addresses so that Squid accepts their replies:

 cache_peer 224.0.14.225 multicast 3128 3130 ttl=16 cache_peer parent1.yourisp.net parent 3128 3130 multicast-responder cache_peer parent2.yourisp.net parent 3128 3130 multicast-responder cache_peer parent3.yourisp.net parent 3128 3130 multicast-responder mcast_icp_query_timeout 2 sec 

Keep in mind that Squid never makes HTTP requests to multicast neighbors, and it never sends ICP queries to multicast-responder neighbors.