8.3 Single Control Card Multiple Line Card Architecture

8.3 Single Control Card + Multiple Line Card Architecture

An example of the single control card + multiple line card architecture is shown in Figure 8.4. There is one control card, with four line cards and a switch fabric card. The switch fabric and the line cards are connected through high-speed serial links. The control card and the line card can be connected by PCI or Fast Ethernet. The latter is sometimes known as the internal Ethernet or Ethernet on the backplane. The control card– to–line card interface is used for messages that need to be processed by the control card’s processor(s) and for messages originated by the control card to be sent to the physical ports. The bandwidth requirement on this control card–to–line card interface is not as high as the line card–to–switch fabric interface. However, when systems with a large number of line cards are built, the traffic on this interface can become an issue. System designers often address this by using a high-speed switched connection like Switched Gigabit Ethernet or Infiniband.

8.3.1 Line Card–Line Card Communication

Line card–to–line card data transfer is through the switch fabric. The switch fabric card provides the switching function for the payload sent by the line cards. In the IPS, if a frame is forwarded from Port 1 of Line Card 1 to Port 3 of Line Card 2, it is sent on the serial link from Line Card 1 to the Switch Card and then from the Switch Card to Line Card 2. A header on the frame indicates the port number on Line Card 2 where the frame is to be sent. The line card (switch fabric electronics) usually has two parts—a line card component and a switch fabric component. Second, the frames are sliced into fixed- length cells and sent on the link. Switching is done faster with fixed-size cells, and the delay is predictable.

8.3.2 Line Card–Control Card Communication

A messaging interface is used for interactions between the tasks on the control card CPU and the tasks on the line card CPU, as shown in Figure 8.3. The types of interactions in Table 8.1 take place on this messaging interface.

Table 8.1: Messaging interactions.

Type of Message/Function	Direction of Communication	Examples/Comments
Message generated by Control Card for transmission on external port	Control Card to Line Card	Routing Protocol Updates, Keep Alive
Control/Management message received on external port	Line Card to Control Card	Routing Protocol or Spanning Tree Protocol messages
Control/Status Message Request	Control Card to Line Card	Query for statistics information on a specific port
Control/Status Message or Response	Line Card to Control Card	Message indicating status change of a port, response for a statistics query from Control Card
Payload message after CPU processing	Control Card to Line Card	MAC frame forwarded on all ports if destination address is unknown
Payload message requiring CPU processing	Line Card to Control Card	MAC frame with Unknown destination address
Heartbeat Message	Control Card to Line Card	Used by Control Card to test whether the line card is functional and also for testing the link between the two. Used for redundancy
Heartbeat Response Message	Line Card to Control Card	Sent by line card to indicate the current status

As the table indicates, each message type from the control card to the line card has an equivalent message type in the reverse direction. A heartbeat message is typically sent as a request from the control card to the line cards. The line card has to respond to this heartbeat request with a response in the reverse direction. If the control card has not received responses to a series of heartbeat messages, it can timeout the line card and flag it as down—note that the control card can only report this as an error but not determine if the line card has failed or if the link has failed.

Protocol messages destined for tasks on the control card are forwarded by the line card. In the IPS, these could be control plane packets like routing protocol PDUs and packets destined to the IPS, such as TCP/telnet packets The line card will make the distinction by comparing the destination IP address in the packet with the IP address of the switch ports or the broadcast/valid multicast address. If there is a match, the packet will be passed on to the CPU (i.e., control card)—the typical IP end node operation.

Figure 8.3: Single Control Card + Multiple Line Card Architecture.

8.3.3 Message Types and Protocols

Table 8.1 indicates a simple summary of message types required on the interface. In more complex scenarios, the control card can affect the forwarding behavior of the line card. This can require additional types of messages from the control card. One scenario is the modification of the traffic management parameters on the line card. This can be the changing of the queue sizes and/or algorithms for packet discard. This area is being investigated by the Forwarding and Control Element Separation (ForCES) Working Group of the IETF. ForCES is specifying the protocol and the format and semantics of the messages between the control element and the forwarding element in a communications system with plane separation.

The message protocol used between the control card and the line card is usually proprietary since it is only used “inside the box.” However, some systems do run protocols such as TCP to provide reliable communication between the control card and the line card, but this is inefficient since TCP is a general-purpose protocol.

While TCP (or a proprietary protocol) may be used for transport, there is a need to have another protocol for card-to-card communication, Inter-Card Communications Protocol (ICCP), which will run over the reliable transport protocol, like TCP. This protocol will identify the type of message, specify the command or response, contain parameters related to the messages, and so on.

8.3.4 Partitioning Software Between Control Card and Line Card

Chapter 3 detailed the functional architecture of a Layer 3 switch or router. This was a single-processor design in which tasks interfaced with each other using Inter-Process Communication (IPC), and the driver programmed the hardware controller as well as received and transmitted frames from the Ethernet links (see Figure 3.5). In this section, this basic design is expanded into a distributed architecture using a control card with its control processor and a line card with multiple Ethernet ports.

Figure 8.4: Software Partitioning on multiboard router

In Figure 8.3 (139 ), the routing tasks such as OSPF, BGP and SNMP are now located on the control card. These tasks build routing tables and provide information to a Route Table Manager (RTM). The RTM has the following functions:

• It is responsible for route leaking or route redistribution, based on a configured policy—e.g., which BGP routes are to be known to OSPF.

• It uses the “raw” routing table/Routing Information Base (RIB) information from the various routing protocols and constructs a forwarding table/Forwarding Information Base (FIB).

• It distributes the FIB to the individual line cards.

The RTM `is responsible for providing the FIB to the individual line cards. It is also responsible for FIB modifications. Moving from the single-processor architecture of xrefparanum to the control card + line card architecture of Figure 8.3, we note that the IP Switching Task in xrefparanum is now on the line card. This task uses the RTM-provided FIB information to program the hardware on the line card for forwarding, including building the hardware forwarding tables. The details vary with the type of hardware- forwarding or -switching silicon devices.

8.3.5 Partitioning Abstraction

Abstraction implies that the higher layer software does not know task distribution across multiple boards and treats the software as if it were running on a single CPU system. Higher layer software tasks continue to use IPC to communicate with another task, except that the task may be running on another board. The communication mapping to another board is done in the IPC implementation and does not affect the two communicating tasks.

As shown in Figure 8.4, the routing protocol tasks perform as if they are on a single board system with 8 ports. This function is provided in the abstraction layer, which will be the only component changing when moving from a single-board to a multi-board system. In Figure 8.4, we call this a Distribution Abstraction Layer (DAL). Note that the DAL on the control card has a corresponding DAL on each of the line cards.

The DAL needs to perform the following functions:

• Maintain mapping of interfaces to board number + port number

• Maintain mapping of task and resource names so that distribution is hidden from the higher layer tasks. Application tasks only use the name for the resource—mapping is done at the DAL.

• Provide IPC services across the cards using system services as appropriate

Mapping Interfaces

Mapping interfaces to the board and port number are required by all tasks using the interfaces, both logical and physical. An example of a logical interface ID to the physical port numbers is provided in Table 8.2.

Table 8.2: Logical and physical interface mappings.

Logical Interface ID	Shelf Number	Board Number	Port Number
1	1	1	1
2	1	2	2
3	2	2	1
4	2	2	3

The protocol and system tasks use only the logical interface ID. The DAL performs the mapping so that messaging can be done appropriately. For example, if the RIP task sends out an update over Logical Interface 2, the DAL will map it to Shelf Number 1, Board 2 and Port 2. This mapped information will be used to send the routing update over the control card–to–line card link to Board 2 of Shelf 1.

Mapping Names

A related and more general version of interface mapping is achieved by the Name Database. This is a single repository for the location of all the resources in the system. Consider an IPS implementation in which the control card runs a TCP/IP end node implementation while a line card may run the PPP protocol as a task. To communicate with the PPP task, the IP task on the control card makes a call such as the following:

SendMsg ("PPP", msgPtr)

The SendMsg routine uses the DAL to determine the location of the PPP task. It does this by using the mapping table maintained by the Name Database, which resembles Table 8.3. The Name string is used to look up the details of the task or resource and determine its location within the system. In the example, the PPP task is on Line Card 1, so the DAL routes the message through the ICCP to Line Card 1. The application calling SendMsg will not know about this translation and routing.

Table 8.3: Interface mapping by the Name Database.

Resource Name	Resource Type	Location (Board/CPU)	Access Parameters/Comments
"RIP"	Task	Control Card	Only one control card in the system running routing protocols
"PPP"	Task	Line Card 1	Only Line Card 1 has serial ports running PPP
"SNMP"	Task	Control Card	Control Card also runs management functions
"IP Switching Task"	Task	All Line Cards	Use Interface ID for distribution to correct line card

When messages are to be sent from the control card out on specific ports on the line card, the DAL uses the logical interface mapping to locate the board and port number on the board to receive the message. The DAL and ICCP may use the services of the real- time operating system for multi-board communication.