5.3. PCI-Based Buses

PCI is a standard type of I/O bus commonly found in many computers, including HP ProLiant servers. The PCI protocol has gone through several iterations and more are planned.

Figure 5-7 shows a diagram representing a standard PCI bus system.

Figure 5-7. The PCI bus.

The PCI bus specification was introduced in 1992 as an industry-standard independent bus. The first PCI specification defined a 32-bit bus running at 33MHz. In 1994, the 2.1 specification defined a 66MHz and 64-bit PCI.

The following table shows the evolution of the PCI protocol through the years.

Width	Speed	Throughput	Specification
32-bit card	33MHz	133MB/s	1.0
32-bit card	66MHz	267MB/s	2.1
64-bit card	33MHz	267MB/s	2.01
64-bit card	66MHz	533MB/s	2.2
64-bit card	100MHz	800MB/s	PCI-X
64-bit card	133MHz	1066MB/s	PCI-X

5.3.1 Features of the PCI Bus

The main features of the PCI bus include the following:

• Full independence

• Plug-and-play architecture

• IRQ sharing

• Multiplexing

• Device loads

• PCI bus burst transfers

• 32- and 64-bit, 33 and 66MHz cards

5.3.1.1 FULLY INDEPENDENT BUS

PCI is fully independent of the processor local bus. As a result, PCI can be found in a variety of platforms, including Intel-based systems, Alpha-based systems, and Apple computers.

The host-to-PCI bridge is a fundamental part of the PCI design. It connects the PCI bus to the frontside bus of the processor. The bridge supports data buffering. This enables activity to take place on the PCI bus at the same time as processor-to-memory bus activity. The bridge makes the activity independent from processor signals and ISA and EISA I/O bus signals.

5.3.1.2 PLUG-AND-PLAY ARCHITECTURE

Plug and play is autoconfigured by the system BIOS or the System Configuration utility in legacy ProLiant servers. System resources are assigned automatically. PCI expansion cards do not need a configuration file because the card tells the computer which resources it requires. As a result, there is a greatly reduced chance of resource allocation conflicts.

5.3.1.3 IRQ SHARING

PCI allows shared IRQ lines. PCI uses INT lines rather than IRQ lines, which are then routed to IRQ lines or directly routed through the advanced programmable interrupt controller (APIC) bus of the processor (full table mode). IRQ sharing is mandatory only for cards complying with specifications for the PCI 2.1 or later.

5.3.1.4 MULTIPLEXING

The PCI standard reduces cost by using a multiplexed address/data bus that reduces the pin count and the size of the PCI slots.

5.3.1.5 DEVICE LOADS

The PCI protocol supports up to 256 devices per bus and 256 buses per system. The actual number of peripheral devices the bus can handle is based on loads, which relate to inductance, capacitance, and other electrical characteristics. Electrical loading reduces the number of slots per bus to 10. The PCI chipset requires three slots, leaving seven for peripherals. Controllers built on the system board use one load, and controllers that plug into an expansion slot use 1.5 loads. Using PCI/PCI-bridges allows for multiple PCI buses and thus for dozens of slots.

5.3.1.6 PCI BUS BURST TRANSFERS

The PCI bus executes all data transfers as burst transfers. Bursts can be either to memory or to the I/O space.

A 32-bit PCI bus running at 33MHz has a peak transfer rate of 133MB/s (33.3 million transfers x 4 bytes). Each PCI device has a latency timer that defines the maximum period of time that a device is allowed to use the bus when another PCI device wants to use the bus.

5.3.1.7 32- AND 64-BIT, 33 AND 66MHZ

PCI can be built either as a 32-bit bus or a 64-bit bus, running at 33 or 66MHz.

5.3.1.7.1 32-Bit PCI at 33MHz

PCI 33MHz/32-bit systems can move up to 90MB/s of data on the standard PCI bus. The rest is spent on overhead and latency resulting from poor bus utilization.

5.3.1.7.2 64-Bit PCI at 66MHz

A 64-bit PCI bus running at 66MHz achieves a peak transfer rate of 533MB/s. A 64-bit PCI bus has better bus utilization and double the bandwidth of a 32-bit PCI bus. The 64-bit address space is enormous (4 billion x 4GB). 32-bit PCI cards can support 64-bit addresses by using two consecutive 32-bit PCI cycles known as dual address cycle.

5.3.1.7.3 32- and 64-Bit, 33 and 66MHz Card Rules

The rules that govern the usage of PCI cards are as follows:

1. 66MHz PCI cards can be used on a 33MHz PCI bus.

2. A 33MHz card in a 66MHz PCI bus automatically operates at 33MHz.

3. 32-bit PCI cards can be installed in a 64-bit PCI slot.

4. 64-bit cards can be installed in a 32-bit slot and will work in 32-bit mode.

5.3.2 PCI Bus Performance Planning

A PCI bus can be so heavily used that it becomes a performance bottleneck. HP recommends that you plan for optimal performance when configuring the PCI devices.

Before beginning the configuration, consider the following:

• The PCI buses and slots available in the server

• The PCI protocols supported by the devices to be configured

• The expected amount of data sent to and from each of the devices (throughput)

Throughput does not include PCI bus interrupts or PCI protocol overhead but will yield reasonable bus planning results.

The following rules provide optimal configuration.

5.3.2.1 RULE 1. MATCH 66MHZ SLOTS WITH 66MHZ DEVICES

If you have 66MHz slots but no 66MHz devices, treat the 66MHz slots like 64-bit slots.

If you have 66MHz devices but no 66MHz slots, treat the devices like 64-bit devices.

If you have 66MHz slots and devices, place the 66MHz devices with the highest throughput in the 66MHz slots. Any remaining devices will be treated like 64-bit devices.

5.3.2.2 RULE 2. MATCH 32-BIT SLOTS WITH 32-BIT DEVICES

If you have 32-bit slots and devices, place the 32-bit devices with the highest throughput in the 32-bit slots. Any remaining devices will be placed in 64-bit slots. Be aware that some 32-bit slots are on 64-bit buses and should be treated appropriately.

If you have 32-bit devices but no 32-bit slots, the 32-bit devices must be placed in 64-bit slots.

5.3.2.3 RULES FOR REMAINING DEVICES

If you have available 32-bit slots, place the minimum number of 64-bit devices in the 32-bit slots, using the devices with the lowest throughput.

If you still have more devices than available 64-bit slots, the 66MHz slots will have to run at 33MHz (64-bit).

Distribute the remaining devices among the available 64-bit slots. You must double the throughput of the 32-bit devices because they use only half of the data transfer bits. Use these guidelines:

1. Place the unassigned device with the highest throughput on the bus with the fewest slots.

2. Of the remaining unassigned devices, place the device with the highest throughput on the next bus with the fewest slots. Again, double the throughput of 32-bit devices.

3. Continue assigning the device with the highest throughput on the next available bus, using all the buses in round-robin fashion. Do not be concerned if you run out of slots on a bus before all devices have been assigned.

5.3.3 PCI Bus Frequency Arbitration

PCI bus frequency is arbitrated during server initialization and cannot change without a server reboot. The arbitrated bus frequency is either the slowest PCI device or the maximum PCI bus speed, whichever is slower. All 32-bit and 64-bit devices can operate on the same 64-bit PCI bus without limitations; 32-bit devices use 32 bits of the data bus, but 64-bit devices use all 64-bits of the data bus. A hot-plugged PCI device must support the arbitrated PCI protocol a nonsupported device will not initialize.

5.3.4 PCI Bus Number Assignments

Each PCI bus is assigned a bus number and each slot on the bus is also assigned a number. Requests to a device are addressed to the bus and slot number.

Some PCI cards contain a PCI controller on the card itself. When a card like this was installed in a PCI slot in older ProLiant servers, it caused bus numbers to change, preventing requests from reaching the right PCI device.

Newer ProLiant servers have preassigned PCI bus numbers for every slot. This preassignment ensures that the PCI bus numbers do not change when a card with a PCI bridge is installed.

5.3.5 PCI Hot Plug Support

PCI Hot Plug enables you to replace and install new PCI cards without powering down the system. PCI Hot Plug was developed by Compaq and has been accepted as an open industry standard.

The benefits of PCI Hot Plug include the following:

• Uninterrupted service You can add or replace a network or other I/O controller board with the system up and operating. This can be accomplished through hot insertion or hot removal.

• Broad, current compatibility PCI Hot Plug technology addresses compatibility concerns by using standard PCI adapters. A hot-plug system requires a hot-plug platform, a hot-plug operating system, and hot-plug adapter drivers. A system can include any combination of hot-plug and conventional versions of each of these components, including a mix of both hot-plug and conventional adapter drivers. However, a particular adapter can be hot plugged only if all three components for that adapter support hot-plug operation.

• Hot Plug Reservation Three PCI Hot Plug Reservation configurations are available, but they require that PCI Hot Plug Reservation be enabled:

  • Hot replacement Replacing an existing PCI adapter

  • Hot removal Removing an existing PCI adapter

  • Hot upgrade Adding a new PCI adapter

The hot-plug electronics designed by HP consist of two separate elements: the hot-plug controller and the slot-specific power control. The hot-plug controller manages the following components.

5.3.5.1 PCI BUS

The controller communicates with isolation devices on the PCI bus to electrically isolate a single PCI slot from the rest of the system. Slot isolation permits insertion or removal of an adapter without interruption to the server or other active adapters.

5.3.5.2 POWER

The controller receives a command from the operating system to power up or power down a single PCI slot. To perform this function, the controller uses the slot-specific power control. The slot-specific power-control electronics allow the proper power sequencing on the PCI bus and guarantee safe control of the power to the individual PCI adapters.

5.3.5.3 SLOT LED INDICATORS

The hot-plug controller also governs the slot LEDs. In the HP implementation of hot-plug hardware, each slot has a green and an amber LED to indicate slot status. The green LED indicates power to the slot and flashes while performing a power state change; the amber LED indicates that the slot requires attention.

5.3.5.4 PCI HOT PLUG BUTTON

This button is pressed to signal the software to initiate a power state change. Although the button is more convenient, the same functionality is provided through the software interface. Each slot has its own button to indicate which slot is to be addressed by the supporting software.

5.3.6 Adding a Hot-Plug Device to an Empty Slot

To add an adapter to an empty slot, follow these steps when the system is running:

1. Prepare the slot for installation of the adapter by opening the appropriate slot release lever and removing the expansion slot cover.

2. Install the adapter into the appropriate expansion slot.

3. Close the slot release lever.

4. Use the PCI Hot Plug button or the software user interface to notify the operating system that power can be applied to the slot. The green LED flashes when the operating system performs the power state change.

The operating system enables power to the slot and either automatically locates and loads the appropriate device driver or prompts the administrator to locate and load the driver.