18.3 Reliable Datagram (RDG)

Reliable Datagrams provide messaging between applications in a cluster. We described this function in the chapter introduction as "horn honking and turn signals." Some cluster applications require a low latency, high bandwidth messaging system. When you honk your horn, it is usually to indicate a significant event to other drivers. Traditionally, datagrams are an unreliable mechanism for communication (your horn may not be heard). Datagrams are usually connectionless, meaning that a datagram is sent to a target without any guarantees of delivery. The benefit of traditional datagram messaging is that it is low overhead and usually faster than a connection-based mechanism.

The Reliable Datagram service provides a mechanism whereby a cluster-aware application can reliably send messages to other applications in the cluster (arrangements are made so that your horn will definitely be heard), yet still maintain the high speed and low latency associated with traditional datagrams.

Two basic usage models are provided for application developers interested in RDG: endpoints and buffers.

Endpoints
- The Receiver calls RDG to create an endpoint.
- Port number is chosen by the system or specified.
- The Receiver prequeues receive addresses.
- The Sender sends to the receiver's endpoint (node number, port).
- Node numbers are mapped to IP addresses.

Buffers
- The Receiver address space buffer is mapped for receiving data.
- The buffer ID is chosen by the system or specified.
- The Receiver prequeues regions to hold messages.
- The Sender writes by specifying target node number and buffer ID.

Statistics pertaining to RDG activities can be viewed by using the lightly documented (no reference page) rdgstats command. The rdgstats command displays a subset of the statistics from the "sysconfig –q rdg" command output. By default, it displays statistics since the system booted. You can get a snapshot of statistics over a span of time using the "-d" option. For instance, "rdgstats –d 20" gathers statistics for 20 seconds before displaying them. You may also want to zero out the RDG statistics before running the rdgstats command by using "sysconfig –r rdg stats_zero=1".

 # sysconfig -r rdg stats_zero=1 stats_zero: reconfigured

 # find / -name '*rdg*' /usr/sbin/rdginit /usr/sbin/rdgstats /usr/shlib/librdg.so                 ←Note the command /usr/ccs/lib/librdg.a                ←RDG shared library /usr/opt/TruCluster/sys/rdg.mod      ←RDG object library /usr/share/man/man5/sys_attrs_rdg.5  ←RDG kernel module /usr/sys/include/sys/rdg.h           ←RDG man page for attributes /usr/sys/XPNINE/rdg.mod              ←RDG header file /usr/sys/XPNINE/rdg.h

 # sysconfig -q rdg rdg: version               = 39 ident                 = RDG V39.46 max_objs              = 1024 msg_q_size            = 32 msg_size              = 16384 stats_collect         = 1 stats_zero            = 0 assertions            = 0 ...                                   ←Lots of attributes (mostly statistics) rdg_tcp_interface     = tu0

The following output shows maximums and minimums pertaining to RDG.

 # sysconfig -Q rdg rdg: version -         type=INT       op=Q     min_val=1      max_val=10000 ident -           type=STRING    op=Q     min_len=1      max_len=256 max_objs -        type=INT       op=CQ    min_val=0      max_val=50000 msg_q_size -      type=INT       op=CQ    min_val=0      max_val=10000 msg_size -        type=INT       op=CQ    min_val=0      max_val=1048576 stats_collect -   type=INT       op=CRQ   min_val=0      max_val=1 stats_zero -      type=INT       op=CRQ   min_val=0      max_val=1 assertions -      type=INT       op=CRQ   min_val=0      max_val=1 ...

The following output would be of more use in a cluster running applications built using the RDG API (such as some database products).

 # rdgstats Cluster membership status of node molari-ics0 is MEMBER RDG statistics - time duration: 0 molari-ics0:         snd_msg_attempts = 0         snd_buf_attempts = 0         snd_blocks = 0         snd_msg_successes = 0         snd_buf_successes = 0         snd_thread_wakeups = 0         snd_kernel_queued_local = 0         snd_kernel_queued_remote = 0         rcv_msg_attempts = 0         rcv_blocks = 0         rcv_msg_successes = 0         rcv_thread_wakeups = 0         bytes_sent = 0         bytes_rcvd = 0 Aggregate sends: 0 Aggregate receives: 0

The following performance comments are approximations only.

If the "Aggregate sends" divided by the count of Memory Channel rails approaches 300-400 MB/s, the cluster may perform better by adding another Memory Channel rail. Additionally, for each system, sum the "bytes_sent" and "bytes_rcvd" attributes and divide the sum by the number of Memory Channel rails. If the total for each system is greater than 60 MB/s, adding an additional rail may provide benefit. Contact your HP representative for more details on RDG performance and statistics.

In order to utilize RDG in your cluster-aware application, you will call RDG routines from the RDG library. This is commonly done by database software. The following extracts some function names from the RDG shared library. Once again, this API is not published, so further information should come from your HP representative.

 # odump -Dt /usr/shlib/librdg.so | grep Rdg [ 14] RdgInit [ 15] RdgShutdown [ 16] RdgNodeLookup [ 17] RdgClusterIncarn [ 18] RdgEpLookup [ 19] RdgBufLookup [ 20] RdgBufCreate [ 21] RdgBufDestroy [ 22] RdgEpCreate [ 23] RdgEpDestroy [ 24] RdgIoPost [ 25] RdgIoPoll [ 26] RdgIoCancel