7.3. The CAM Layer To reduce the complexity of the individual disk drivers, much of the complexity of handling a modern controller has been abstracted out to a separate CAM layer that sits below the GEOM layer and above the device-driver level. The CAM layer handles the device-independent tasks of resource allocation and command routing. These tasks include the tracking of requests and notifications between the controller and its clients. They also include the routing of requests across the many I/O busses to get the request to the correct controller. The CAM layer leaves to the device driver the device-specific operations such as the setup and teardown of the DMA (Direct Memory Access) maps needed to do the I/O. Some device drivers can still get complex. For example, the Fibre Channel device driver has a lot of code to handle operations, such as asynchronous topology changes as drives are removed and attached. A driver responds to a CAM request by converting the virtual address to store the data to the appropriate physical address. It then marshals the device-independent parameters like IO request, physical address to store the data, and transfer length into a firmware-specific format and executes the command. When the I/O completes, the driver returns the results back to the CAM layer. In addition to disks, the CAM layer manages any other storage device that might be connected to the system such as tape drives and flash memory dongles. For other character devices such as keyboard and mice, CAM will not be involved. A USB keyboard will be attached directly to the keyboard driver. SCSI Subsystem The CAM SCSI subsystem provides a uniform and modular system for the implementation of drivers to control various SCSI devices and to use different SCSI host adapters through host-adapter drivers. The CAM system is made up of three layers: The CAM peripheral layer that provides open, close, strategy, attach, and detach operations for the supported SCSI devices. CAM supported devices includes direct access (da) disk drives, cdrom (cd) CD-ROM drives, sequential access (sa) tape drives, and changer (ch) jukeboxes. The CAM transport layer that builds, executes, and interprets results of SCSI commands. CAM starts by building a generic I/O command using a CCB (CAM Control Block). The CCB contains a CDB (Command Descriptor Block) composed of a 6-16 byte SCSI command. For example, the command "READ_10, block_offset, count" gets back a status of success or various error codes. If there is an error, the drive may also include sense data to give more information about the cause of the error. The CAM SIM (Software Interface Module) or HBA (Host Bus Adapter) interface layer provides bus routing to devices. Its job is to allocate a path to the requested device, send a CCB action request to the device, and then collect notification of the I/O completion from the device.
The operation of the CAM layer is most easily understood by tracing an I/O request through it. The Path of an I/O Request Through the CAM Subsystem The path of a request through the CAM I/O subsystem is shown in Figure 7.7 (on page 278). In the FreeBSD framework, the filesystem sees a single contiguous disk. I/O requests are based on block numbers within this idealized disk. In Figure 7.7, the filesystem determines a set of blocks on which it wants to do I/O, and it passes this request down to the GEOM layer by calling the strategy() routine. Figure 7.7. The path of an I/O request through the CAM subsystem. 
The GEOM layer takes the request and determines the disk to which the request should be sent. In this example the request is on a da SCSI disk. Sometimes a request may span several disks. When this happens, the GEOM layer breaks up the original request into a set of separate I/O requests for each of the disks on which the original request resides. Each of these new requests are passed down to the CAM layer by calling the appropriate strategy( ) routine for the associated disk (the dastrategy( ) routine in Figure 7.7). The CAM dastrategy( ) routine gets the request and calls bioq_disksort( ), which puts the request on the disk queue of the specified SCSI disk. The dastrategy( ) routine finishes by calling the xpt_schedule( ) function. The xpt_schedule( ) function allocates and constructs a CCB (CAM Control Block) to describe the operation that needs to be done. If the disk supports tagged queueing, an unused tag is allocated, if it is available. If tagged queueing is not supported or a tag is not available, the request is left on the queue of requests pending for the disk. If the disk is ready to accept a new command, the xpt_schedule( ) routine calls the drive start routine set up for it (dastart( ) in this example). The dastart( ) routine takes the first request off the disk's queue and begins to service it using the CCB that was constructed by dastrategy( ). Because the command is destined for a SCSI disk, dastart( ) needs to build a SCSI READ_10 command based on the information in the CCB. The resulting SCSI command that includes a READ_10 header, a pointer to the virtual address that references the data to be transferred, and a transfer length is placed in the CCB and given the type XPT_SCSI_IO. The dastart( ) routine then calls the xpt_action( ) routine to determine the bus and controller (adaptor) to which the command should be sent. The xpt_action( ) routine returns a pointer to a cam_path structure that describes the controller to be used and has a pointer to the controller's action routine. In this example we are using the Adaptec SCSI controller whose action routine is ahc_action(). The xpt_action() routine queues the CCB with its cam_path and schedules it to be processed. The request is processed by calling the controller specific action routine, ahc_action(). The ahc_action() routine gets the CCB and converts its generic SCSI command into a hardware-specific SCB (SCSI Control Block) to handle the command. The SCB is filled out from information in the CCB. It is also filled out with any hardware-specific information and a DMA request descriptor is set up. The SCB is then passed to the driver firmware to be executed. Having completed its task, the CAM layer returns back to the caller of dastrategy(). The controller fulfills the request and DMAs the data to or from the location given in the SCB. When done, a completion interrupt arrives from the controller. The interrupt causes the ahc_done() routine to be run. The ahc_done() routine updates the CCB associated with the completed SCB from information in the SCB (command completion status or sense information if there was an error). It then frees the previously allocated DMA resources and the completed SCB and passes the completed CCB back to the CAM layer by calling xpt_done(). The xpt_done() routine inserts the associated CCB into the completion notification queue and posts a software interrupt request for camisr(), the CAM interrupt service routine. When camisr() runs, it removes the CCB from the completion notification queue and calls the specified completion function which maps to dadone() in this example. The dadone() routine will call the biodone() routine, which notifies the GEOM layer that one of its I/O requests has finished. The GEOM layer aggregates all the separate disk I/O requests together. When the last I/O operation finishes, it updates the original I/O request passed to it by the filesystem to reflect the result (either successful completion or details on any errors that occurred). The filesystem is then notified of the final result by calling the biodone() routine. | 
