AH can be used in either transport or tunnel mode, just like ESP. The difference is the data being protected, either an upper-layer protocol or an entire IP datagram.
When used in transport mode, AH protects end-to-end communication. The communications endpoint must be IPSec endpoint. The AH header is inserted into the datagram by placing it immediately following the IP header (and any options) and before the upper-layer protocol header.
When used in tunnel mode, AH encapsulates the protected datagram. An additional IP header is added before the AH header after that. The "internal" IP datagram maintains the original addressing of the communication and the "outer" IP datagram contains the addresses of the IPSec endpoints. Tunnel mode can be used as a replacement to transport mode for end-to-end security. However, as there is no confidentiality and therefore no protection against traffic analysis, it is pointless. AH is used only to guarantee that the received packet was not modified in transit, that it was sent by the party claiming to have sent it, and,
When an outbound packet matches an SPD entry indicating protection with AH, the SADB is queried to see whether a suitable SA exists. If there is no SA, IKE can be used to dynamically create one. If there is an SA, AH is applied to the matched packet in the mode dictated by the SPD entry. If there is an SPD bundle, the order of application depends on the protocols involved. AH always protects ESP, not the other way around.
As mentioned in Chapter 3, when an outbound SA is created—either manually or by IKE—the sequence number counter is
The remaining fields of the AH header are filled in with their appropriate value—the SPI field is assigned the SPI from the SA; the
Unlike ESP, AH extends its protection to immutable or predictable fields of the outer IP header. It is therefore necessary to zero out the mutable fields prior to computation of the Integrity Check Value (ICV). The mutable fields of an IPv4 header that are not included in the authenticating ICV (and are therefore "
For IPv4 options or IPv6 extension headers, if they are immutable or predictable, they are included in the computation of the ICV. Otherwise, it is necessary to zero them out before computing the ICV.
Padding may be needed depending on the requirements of the authenticator or for alignment reasons. Some MACs—for instance, DES-CBC MAC—require the data over which the MAC is applied be a multiple of the algorithm's block
The AH header must be a multiple of 32 bits for IPv4 and 64 bits for IPv6. If the output of the MAC is such that this requirement is not met, the AH header must be
The ICV is calculated by passing the key from the SA and the entire IP packet (including the AH header) to the algorithm identified as the authenticator in the SA. Since the mutable fields have been zeroed out the actual values are not included in the ICV. The ICV value is then
AH processing is now complete and the AH-protected IP packet can be transmitted. Depending on the size of the packet, it might be fragmented prior to placing on the wire or it might be
Reassembly may be required prior to AH input processing if the protected packet was fragmented prior to its receipt. This is a
The first thing to do when processing any IPSec packet is to find the SA that was used to protect it, and AH is no different than ESP in this respect. The SA is, again, identified by the destination address of the IP header, the protocol (in this case 51), and the SPI from the AH header. If no SA is found, the packet is discarded.
Once the SA is found, a sequence number check is made. This step is optional but the cost-to-benefit ratio is so low that there is really no reason not to perform an antireplay check. The
The ICV must now be checked. First, the ICV value in the authentication data field of the AH header is saved and that field is zeroed. All mutable fields in the IP are also zeroed (these are the same fields, described above, that were zeroed prior to ICV calculation). If the authenticator algorithm and payload length are such that implicit padding is required to bring the size of the data authenticated up to the requirements of the algorithm, implicit padding is added. This implicit padding must contain the value zero. The authenticator algorithm is then applied to the entire packet and the resulting digest is compared to the saved ICV value. If they match, the IP packet has been authenticated; if they do not match the packet must be discarded.
Once the ICV has been