Example 8: Atomic Read-Modify-Write

Example 8:Atomic Read-Modify-Write

Problem: Device 2 (UnitID2) in Figure 7-10 targets main memory with an Atomic RMW transaction. Memory start address is 0082402020h.

Figure 7-10. Atomic Read-Modify-Write Targeting Main Memory

Example 8: Atomic RMW Request Packet Setup

(Refer to [1] in Figure 7-10 on page 182)

Command[5:0] Field (Byte 0, Bit 5:0)

This is the Atomic RMW command code. There are no option bits within this field and the following limits are imposed: isochronous Atomic RMW is not allowed, target location may be cached, the response to an Atomic RMW may not pass posted writes . For this example, field = 111101b.

SeqID[3:0] Field (Byte 0, Bit 7:6) and (Byte 1, Bit 6:5)

This field tags groups of requests that were issued as part of an ordered sequence; the field is cleared if a request is not part of a sequence. In this example, this request is not part of an ordered sequence. For this example, field = 0000b.

UnitID[4:0] Field (Byte 1, Bits 4:0)

For both upstream and downstream requests, this field is programmed with the UnitID of the requester. UnitID is assigned during initialization by configuration software. In this example, the request originates at Device 2 (UnitID2). For this example, field = 00010b.

PassPW Bit Field (Byte 1, Bit 7)

This bit is reserved for Atomic RMW requests, and should be cleared. For this example, field = 0b.

SrcTag[4:0] Field (Byte 2, Bits 4:0)

This field is used to tag the 32 outstanding non-posted transactions allowed to each UnitID. The code is assigned by requester from a pool of available tags. Assume that the tag for this read request is 10d (Ah). For this example, field = 01010b.

Compat Bit Field (Byte 2, Bit 5)

Used by bridges to tag downstream requests targeting the compatibility chain (where subtractive decoder is found). Atomic RMW requests target main memory, so this bit should be clear. For this example, field = 0b.

Mask/Count[3:0] Field (Byte 2, Bits 7:6) and (Byte 3, Bits 1:0)

The value programmed in this field depends on which of the two Atomic RMW variants is being performed. For the Fetch and Add operation, the request is accompanied by one qword (2 dwords) of data. In this case, the Mask/Count would be programmed with one ( dword count -1). For the Compare and Swap operation, the request is accompanied by two qwords (4 dwords) of data. In this case, the Mask/Count would be programmed with three (again, dword count -1). Note: the value in this field is used by the Host Bridge to determine which variant is being requested . In this Fetch and Add example, the transfer is one qword (two dwords). For this example, field = 0001b.

Start Address Field (Bytes 4-7, Bit 7:0) and (Byte 3, Bit 7:2)

This field carries the 40-bit target start address. Note that the lower bits [1:0] are not in the field, and are assumed to be = 00b. This means that the start address is dword-aligned. For Atomic RMW, the target address is also expected to be in the main memory portion of the system memory map. For this example, the 40-bit address = 0082402020h.

Example 8: Atomic RMW Request Data Packet

(Refer to [2] in Figure 7-10 on page 182)

The Atomic RMW Fetch and Add request is accompanied by a one qword Input value carried in the data packet. This is the value which will be added to the value currently in the target memory location. For this example, the Input value is: 104B300B2C3D0042h.

Example 8: Atomic RMW Request, Sequence Of Events

(Refer to Figure 7-10 on page 182)

1. UnitID2 checks its transmitter non-Posted Request (CMD ) and non-Posted Request (Data ) flow control counters to make sure it has at least one "credit" in each before issuing the request.

2. The transmitter asserts CTL and starts sending out the request packet information. On this eight bit interface, the transmitter sends one byte of request packet info during each successive bit time, least significant byte first. Following the request, it will de-assert CTL and drive the qword (2 dwords) data packet containing the Input value across the interface.

3. When the tunnel receives this read request and data, it checks its own transmitter non-posted flow control counters for the upstream link and prepares to reissue the packets towards the host bridge.

4. When the host bridge receives the request and data, it will lock the memory location which prevents access by other agents .

5. The Host Bridge reads the current data from target memory address.

6. The Host Bridge adds the value just read to the Input value which accompanied the request and writes the sum back to the target memory location.

7. The Host Bridge unlocks the memory location and prepares to send the read response and (readback) data to the original requester.

Example 8: Atomic RMW Response Packet Setup

(Refer to [3] in Figure 7-10 on page 182)

When the Host Bridge has completed the read/modify/write of the target memory location, it still must return the "original" value read from memory to the requester. This is always a qword (2 dword) data object. The Host Bridge prepares to initiate a Read response to accompany the "readback" data, using the original UnitID and Source Tag information provided in the request.

Command[5:0] Field (Byte 0, Bit 5:0)

This is the command code for the Read Response. There are no option bits in this field. For this example, field = 110000b.

Isoc Bit Field (Byte 0, Bit 7)

This bit indicates whether this response should travel in the isochronous response virtual channel as it moves back to the original requester. Atomic RMW transactions are not allowed to use the isochronous virtual channels so this bit should be cleared. For this example, field = 0b.

UnitID[4:0] Field (Byte 1, Bits 4:0)

For responses, there are two ways this field is handled. If the response is traveling upstream, it is always programmed with the UnitID of the bridge (UnitID0). If the response is traveling downstream (as this one is), the UnitID field contains the Unit ID of the original Requester. For this example, field = 00010b.

Bridge Bit Field (Byte 1, Bit 6)

Set by host bridges in responses they issue downstream (such as this example). Devices use this information to decide if they may attempt to decode and claim a response; only downstream responses (Bridge = 1) may be claimed by non-bridges. For this example, field = 1b.

PassPW Bit Field (Byte 1, Bit 7)

This bit should be cleared because Atomic RMW responses are not allowed to pass posted writes. For this example, field = 0b.

SrcTag[4:0] Field (Byte 2, Bits 4:0)

The field is set to the same value as seen in the request. In our Atomic RMW example, the Source Tag was 0Ah. For this example, field = 01010b.

Error Bit Field (Byte 2, Bit 5)

In a read transaction, this bit in the response will be set to indicate an error was encountered in obtaining requested data. When the response finds its way back to the original requester, this bit means that the accompanying data is not valid; action taken by the requester is design-specific, but may range from an interrupt to a sync flood. Refer to "Response Errors" on page 248 for a discussion of how response errors are handled in HyperTransport. For this example, field = 0b.

Count[3:0] Field (Byte 2, Bits 7:6) and (Byte 3, Bits 1:0)

For both variants of Atomic RMW, this field should be set = one (2 dwords = 1 qword). For this example, field = 0001b.

NXA Bit Field (Byte 3, Bit 5)

This response bit is only valid if the Error bit (Byte 2, Bit 5) is set. IF NXA and Error are both set, the original request did not find the proper target at all, and the read response is being returned by a device at the end of the chain. If NXA is clear and Error set, then a read error occurred at the target device. For this example, field = 0b.

Example 8: Atomic RMW Response Data Packet

(Refer to [4] in Figure 7-10 on page 182)

The Atomic RMW request that caused this data packet to be sent always results in a return of 1 qword (2 dwords) of data. The response data packet contains the "original" data value read from memory at the beginning of the Fetch and Add operation. For this example, the qword of "original" data at the target start address was: 102400000014000Fh.

Example 8: Atomic RMW Response, Sequence Of Events

(Refer to Figure 7-10 on page 182)

1. After the Host Bridge performs the memory fetch, adds the input value to it and writes the result back to memory, it prepares to send the read response and "original" read data back to the requester. The UnitID (2) and SrcTag (0Ah) fields are set up in the response to match the request, the Bridge bit is set = 1 and if there were no errors performing the operation, the Error and NXA bits will also be clear. The Compat and Isoc bits are also cleared.

2. The Host Bridge transmitter sends one byte of the four byte response packet during each successive bit time on its 8-bit interface. The CTL signal is asserted during the transmission of the response, and the least significant byte is sent first.

3. When the response packet has been sent, the Host Bridge deasserts CTL and sends the single qword (two dword) data packet.

4. When the tunnel receives the response and data from the Host Bridge, it checks the Bridge bit to see if it is set (it is) and UnitID to determine if the response belongs to it (it doesn't). Unable to claim the response and data, the tunnel forwards the packets as-is downstream to the next device.

5. When the response reaches Device 2 (UnitID2) it is decoded and claimed. The SrcTag[4:0] field is used to identify the particular outstanding request this response is associated with, in this case the transaction is #10d (0Ah).

Some Notes About Atomic RMW Operations

(Refer to Figure 7-10 on page 182)

• The purpose of this request type is to allow an atomic read, modify, and write of a memory location without the possibility another agent will access the location while any part of the hybrid operation is in progress. The memory controller is required to lock at least the one qword of address range which contains the target start address until it completes.

• The Fetch and Add variant of Atomic RMW takes the Input Value (1 qword of data) and adds it to the value currently at the target location and writes it back to memory. The "original" value read from memory (before the addition) is returned to the requester with the read response. This means that both the request and the response packets are accompanied by data packets.

• The Compare and Swap variant of Atomic RMW request is accompanied by a two qword data packet: the Input Value and a Compare value. When the memory controller detects a Compare and Swap (based on the value in the Mask/Count[3:0] field), it reads the value currently at the target memory location and checks it against the Compare value; if the Compare value is greater than the memory location, the Input value is over-written to memory (a swap has occurred). If the Compare value is less than the "original" value in memory, the Input value is discarded (no swap). In either case, the "original" qword value in memory is returned to the requester with the read response.

• The value in the Atomic RMW request Mask/Count[3:0] field is expected to be either one (Fetch and Add) or three (Compare and Swap). Behavior resulting from programming a value other than these two into the request Mask/Count field is undefined. Note that the Response returned for either variant of Atomic RMW request always carries a Mask/Count[3:0] field of one (1 qword).

• Atomic RMW requests move upstream towards the Host bridge. The Compat in the response should be clear.

• No target, other than a host, is required to support atomic operations. Non- hosts which decode Atomic RMW requests may either return a single qword of data with the Error bit set in the response or they may allow the read, modify, and write in a non-atomic manner.

• A host is not required to support Atomic RMW requests which target address outside of main memory. If they see one, they may abort it or reflect it as a peer-to-peer operation. Peer-to-peer Atomic RMW operations are to be supported in future revisions of the HyperTransPort Specification.