Section 4.6. System V Message Queues

4.6. System V Message Queues

Message queues provide a means for processes to send and receive messages of various sizes in an asynchronous fashion on a Solaris system. As with the other IPC facilities, the initial call when message queues are used is an ipcget call, in this case, msgget(2). The msgget(2) system call takes a key value and some flags as arguments and returns an identifier for the message queue. Once the message queue has been established, it's simply a matter of sending and receiving messages. Applications use msgsnd(2) and msgrcv(2) for those purposes. The sender simply constructs the message, assigns a message type, and calls msgsnd(2). The system places the message on the appropriate message queue until a msgrcv(2) is successfully executed. Sent messages are placed at the back of the queue, and messages are received from the front of the queue; thus, the queue is implemented as a FIFO (First In, First Out).

The message queue facility implements a message type field, which is defined by the user (programmer). So programmers have some flexibility, since the kernel has no embedded or predefined knowledge of different message types. Programmers typically use the type field for priority messaging or directing a message to a particular recipient.

Last, applications use the msgctl(2) system call to get or set permissions on the message queue and to remove the message queue from the system when the application is finished with it. For example, msgct(2) offers a clean way to implement an application shutdown procedure because the system will not remove an empty and unused message queue unless it is explicitly instructed to do so or the system is rebooted.

4.6.1. Kernel Resources for Message Queues

Like the IPC facilities previously discussed, the message queue facility comes in the form of a dynamically loadable kernel module, /kernel/sys/msgsys, and depends on the IPC support module, /kernel/misc/ipc, to be loaded in memory.

The number of resources that the kernel allocates for message queues is tuneable. Values for various message queue tuneable parameters can be increased from their default values so that more resources are made available for systems running applications that make heavy use of message queues. Table 4.7 summarizes the tuneable parameters and lists their default and maximum values.

Table 4.7. Message Queue Resource Limits

Name	Default	Max	Description
project.max-msg-ids	50	1<<24	Maximum number of message queue identifiers
process.max-msg-qbytes	4096	Unlimited	Maximum number of bytes on a message queue
process.max-msg-messages	40	2B	Maximum number of message headers

• project.max-msg-ids. Maximum number of message queue IDs allowed a project. When msgget() is used to allocate a message queue, one ID is allocated. If the ID allocation doesn't succeed, msgget() fails and errno is set to ENOSPC (previously returned when "The system-imposed limit on the maximum number of allowed message queue identifiers systemwide would be exceeded"). Upon successful msgctl(, IPC_RMID), the ID is deallocated.

• process.max-msg-qbytes. Maximum number of bytes of messages on a message queue. When msgget() successfully allocates a message queue, the minimum enforced value of this limit is used to initialize msg_qbytes (which was previously "set to the system limit").

  If an application attempts to put a new message on a message queue that will result in the total bytes being greater then max-msg-qbytes, then either the msgsnd(2) call returns an error or the process blocks, waiting for the message to be pulled off the queue, depending on whether the IPC_WAIT flag is true.

• process.max-msg-messages. Maximum number of messages on a message queue. When msgget() successfully allocates a message queue, the minimum enforced value of this limit is used to initialize a per-queue limit on the number of messages.

When the /kernel/sys/msgsys module is first loaded, an initialization routine executes. The routine does much the same sort of work that is done for shared memory and semaphore initialization.

The kernel data structure that describes each message queue is the msqid_ds structure.

struct msg; struct msqid_ds {         struct ipc_perm msg_perm;        /* operation permission struct */         struct msg      *msg_first;      /* ptr to first message on q */         struct msg      *msg_last;       /* ptr to last message on q */         msglen_t        msg_cbytes;      /* current # bytes on q */         msgqnum_t       msg_qnum;        /* # of messages on q */         msglen_t        msg_qbytes;      /* max # of bytes on q */         pid_t           msg_lspid;       /* pid of last msgsnd */         pid_t           msg_lrpid;       /* pid of last msgrcv */ #if defined(_LP64)         time_t          msg_stime;       /* last msgsnd time */         time_t          msg_rtime;       /* last msgrcv time */         time_t          msg_ctime;       /* last change time */ #else         time_t          msg_stime;       /* last msgsnd time */         int32_t         msg_pad1;        /* reserved for time_t expansion */         time_t          msg_rtime;       /* last msgrcv time */         int32_t         msg_pad2;        /* time_t expansion */         time_t          msg_ctime;       /* last change time */         int32_t         msg_pad3;        /* time_t expansion */ #endif         short           msg_cv;         short           msg_qnum_cv;         long            msg_pad4[3];    /* reserve area */ };                                                                           See sys/msg.h 

4.6.2. Kernel Implementation of Message Queues

Let's walk through the kernel flow involved in the creation of a message queue and the sending and receiving of messages, since these represent the vast majority of message queue activities.

1. The creation of a message on behalf of an application calling the msgget(2) system call starts with a call to the kernel ipc_get() routine. An ipc_perm structure is available for every message queue identifier.

2. Once a structure has been allocated, the system initializes the structure members as specified by the UID and GID of the calling process and the permission mode bits passed by the calling code, then sets the IPC_ALLOC bit to signify the ipc_perm structure has been allocated. (The ipc_get() code is covered in Section 4.1.)

   If ipc_get() returns successfully, the application code has a valid message queue identifier, can send and receive messages, and can run message control (msgctl(2)) operations.

A message send (msgsnd(2)) call requires the application to construct a message, setting a message type field and creating the body of the message (for example, a text message).

1. The code copies the message from the user address space to a designated area in the kernel.

2. Next, we do general housekeepingsuch as incrementing the processor statistics to announce a message queue system call is being executed. The cpu_sysinfo structure maintains an msg counter that reflects the total number of message queue system calls executed.

3. The code verifies the calling process's access permissions.

4. The rest of the message send flow is best represented in pseudocode.

copy message data from user space to kernel space (map area) lookup a message id via ipc_lookup() if (message queue no longer exists)         return EIDRM error if (current bytes on queue + bytes in new msg > msgmax)         if (IPC_NOWAIT is set)                 return EAGAIN         else                 increment the message's msg_snd_cnt         /* on wakeup, code will validate msqid and set EDIRM if it has been removed */         /* sleep waiting for space */         call cv_wait_sig()         /*          * Once the wakeup is issued, the necessary resources are available for          * putting the message on the queue          */         increment msg_qnum         msg_cbytes += new message size         mds_lspid = PID of caller         msg_stime = current time        insert onto message queue (msg_list)        if (messages are available)                /* wakeup waiters */                cv_broadcast() return success to calling program 

The msgrcv support code is a little less painful, since now we're looking for a message on the queue (as opposed to putting one on the queue). Kernel resources do not need to be allocated for a msgrcv. The general flow of the kernel code path for receiving messages involves checking permissions for operation in a loop through all the messages on the queue.

1. If the requested message type matches a message on the queue, the code copies the message type to the user-supplied location and copies the message data to the user-supplied location.

2. Next, the code updates the msqid_ds structure fields, subtracts the message size from msg_cbytes, sets the PID in msg_lrpid, sets time in msg_rtime, frees the message resources, frees the message header (msg structure), and frees the resource map entry.

   If the code looped through all messages and found no matching type, it returns a No Message error.

   When the sequence is completed, the application code has the message type and data in a buffer area supplied in the msgrcv(2) system call. The only remaining callable routine for applications to use is the msgctl(2) system call. The control functions are straightforward; they typically involve either retrieving or setting values in a message queue's ipc_perm structure.

3. When msgctl(2) is invoked with the IPC_RMID flag, meaning the caller wants to remove the message queue from the system, the kernel walks the linked list of messages on the queue, freeing up the kernel resources associated with each message.

4. The kernel sends a wakeup signal to processes (threads) sleeping on the message queue. The processes ultimately end up with an EIDRM error (ID removed).

5. The system simply marks the msqid_ds structure as being available, and returns.