Section 18.7. IP

18.7. IP

As mentioned before, all the transport layers have been merged in the IP module, which is fully multithreaded and acts as a pseudo device driver as well a STREAMS module. The key change in IP was the removal of IP client functionality and the multiplexing of the inbound packet stream. The new IP classifier (which is still part of the IP module) classifies the inbound packets to the correct connection instance. The IP module is still responsible for network layer protocol processing and plumbing and managing the network interfaces.

Let's quickly look at how plumbing of network interfaces, multipathing, and multicast works in the new stack. The table below describes the data types referenced in the following section, some of which have been discussed earlier in the chapter, but are included here for completeness. The structure definitions can be found in usr/src/uts/common/inet/ip.h.

Table 18.1. Network Interface Structure Types and Names

Name/Type	Description
ill / ill_t	The structure associated with a physical NIC interface
ipif / ipif_t	The logical interface, associated with an ill
ire / ire_t	Internet route entry. Contains the information to get the packet to its correct destination, including the correct physical interface
ilg / ilg_t	Maintains the state of multicast addresses associated with a socket (and network communication endpoint)
ilm / ilm_t	Maintains the state of multicast addresses associated with a physical interface (NIC)

18.7.1. Plumbing NICs

Plumbing is a long sequence of operations involving message exchanges between IP, Address Resolution Protocol (ARP), and device drivers. Set ioctl() calls are typically involved in plumbing operations. A natural model is to serialize these ioctls. For example, plumbing of hme0 and qfe0 can go on in parallel without any interference, but various set ioctls on hme0 will all be serialized.

Another possibility is to refine even further and serialize operations for the virtual interface (ipif) rather than the physical interface (ill). This will be beneficial only if many ipifs are hosted on an interface and if the operations on different ipifs don't interfere with one another. Another possibility is to completely multithread all ioctls with standard Solaris multithreading techniques, but this is needlessly complex and does not add much value. It is hard to hold locks across the entire plumbing sequence, which involves waits and message exchanges with drivers or other modules.

Not much is gained in performance or functionality by simultaneously allowing multiple set ioctls on an ipif at the same time, since these are purely nonrepetitive control operations. Broadcast ires are created for each ill rather than for each ipif. Hence, trying to start more than one ipif simultaneously on an ill involves extra complexity in the broadcast ire creation logic. On the other hand, serializing plumbing operations for each ill lends itself easily to the existing IP code base. During the course of plumbing, IP exchanges messages with the device driver and ARP. The messages received from the underlying device driver are also handled exclusively in IP. This is convenient since we can't hold standard mutex locks across the putnext() in trying to provide mutual exclusion between the write-side and read-side activities. Instead of the all-exclusive PERMOD syncq, this effect can be easily achieved by a per-ill serialization queue.

18.7.2. IP Network Multipathing

IP network multipathing (IPMP) operations are all driven around the notion of an IPMP group. Failover and failback operations operate between two ills, usually part of the same IPMP group. The ipifs and ilms are moved between the ills. This move involves shutting down the source ill and could involve starting up the destination ill. Shutting down or starting up ills affects broadcast ires. Broadcast ires need to be grouped as an IPMP group to suppress duplicate broadcast packets that are received. Thus, broadcast ire manipulation affects all members of the IPMP group. Setting IFF_FAILED or IFF_STANDBY causes evaluation of all ills in the IPMP group and causes regrouping of broadcast ires. Thus, serializing IPMP operations for each IPMP group lends itself easily to the existing code base. An IPMP group includes both the IPv4 and IPv6 ills.

18.7.3. Multicast

Multicast joins operate on both the ilg and ilm structures. Multiple threads operating on an Interprocess Communication (IPC) socket, trying to do multicast joins, need to synchronize when operating on the ilg. Multiple threads (potentially operating on different socket endpoints) trying to do multicast joins could eventually end up trying to manipulate the ilm simultaneously and need to synchronize on the access to the ilm. Both are amenable to standard Solaris mutlithreading techniques. Considering all the aboveplumbing, IPMP, and multicastthe common denominator is to serialize all the exclusive operations for each IPMP group. If IPMP is not enabled, then serialize on a physical interface. For example, hme0 v4 and hme0 v6 ills taken together share a physical interface NIC. In the above, multicast has a potentially higher degree of multithreading. But it has to coexist with other exclusive operations. For example, we don't want a thread to create or delete an ilm when a failover operation is already trying to move ilms between two ills. So the lowest common denominator is to serialize multicast joins for the physical interface or for each IPMP group.