Multiprocessing Data Structures

Each processor has a data structure that is used to keep track of data specific to that processor. This structure is of type struct mpinfo , which is typdef ed to mpinfo_t . The global variable mpproc_info is declared as a pointer to a struct mpinfo . At boot time, the system allocates enough memory to hold one struct mpinfo for each processor. It then points mpproc_info to the beginning of this allocated memory. In this way, mpproc_info can be used as an array of struct mpinfo by using the processor index. For example, mpproc_info[0] is the mpinfo structure for the first processor in the system. The mpinfo structure is also sometimes called the per-processor data pointer (PPDP).

The mpinfo structure is a huge structure—over 13 KB in an 11.11 system. A large portion of this is the ki_proc_info_data . This structure is used for kernel profiling and tracing, and has counters to keep track of syscall and procedure calls. Table 12-1 lists some of the more useful and interesting members of the mpinfo structure. Note that the code uses the terms "processor" and "SPU" interchangeably. An SPU is a System Processor Unit.

Table 12-1. The `mpinfo` Structure

Type and Name	Description
`prochpa`	Pointer to the processor's hard physical address (HPA)
`iva`	Interrupt vector address
`procindex`	Processor number
`spinlock_depth`	Number of spinlocks held
`entry_int_level`	Interrupt level the processor was at when the first spinlock was taken
`spu_state`	State of the SPU: one of `SPU_ENABLED` , `SPU_DISABLED` , `SPU_PENDING` , `SPU_INTR_ENABLED` , `SPU_DISABLED_HW_FAILURE`
`threadp`	Pointer to the current thread structure
`uareasid`	Space ID of the `uarea` structure
`prevthreadp`	Pointer to the previous thread structure
`mpcntrs`	Per-processor counters
`mp_rq`	Run queues

The mpcntrs member has some interesting information in it and is worth a further look (Listing 12.1).

Listing 12.1. Declaration for the `mpcntrs` structure

typedef struct mpcntrs {

         u_long fsreads;         /* # of actual reads to FS blocks */

         u_long fswrites;        /* # of actual writes to FS blocks */

         u_long nfsreads;         /* # NFS reads issued */

         u_long nfswrites;        /* # NFS writes issued */

         u_long bnfsread;         /* # bytes read via NFS */

         u_long bnfswrite;        /* # bytes written via NFS */

         u_long phread;           /* # physical reads issued */

         u_long phwrite;          /* # physical writes issued */

         u_long runocc;           /* # of times rq occupied since boot */

         u_long runque;           /* cumulative len of rq since boot */

         u_long sysexec;          /* # of exec()'s since boot */

         u_long sysread;          /* # of read/readv()'s since boot */

         u_long syswrite;         /* # of write/writev()'s since boot */

         u_long sysnami;          /* # of filename lookups since boot */

         u_long sysiget;          /* # of inode fetches since boot */

         u_long sysselect;        /* # of select calls since boot */

         u_long dirblk;           /* # of directory disk blocks read */

         u_long semacnt;          /* # of SysV sema ops since boot */

         u_long msgcnt;           /* # of SysV msg ops since boot */

         u_long muxincnt;         /* # of mux input xfers since boot */

         u_long muxoutcnt;        /* # of mux output xfers since boot */

         u_long ttyrawcnt;        /* # of raw chars read since boot */

         u_long ttycanoncnt;      /* # of canonical chars processed */

         u_long ttyoutcnt;        /* # of chars output since boot */

         long activeprocs;        /* # of proc  table entries allocated */

         long activethreads;      /* # of thread table entries allocated */

         long activeinodes;       /* # of inode table entries allocated */

         long activefiles;        /* # of file  table entries allocated */

} mpcntrs_t;

These statistics can be useful in comparing various types of loads across the processors. Note that these numbers are per-processor, so each processor has its own set of counters. The numbers are also cumulative since boot time. The exception is the four counters that start with "active": activeprocs , activethreads , activeinodes , and activefiles . These counters represent the current number of items used by this processor. For example, activeprocs is incremented when a process is created and decremented when the process is destroyed . Examining these values for the different processors can tell us if a particular type of activity is getting tied up on a particular processor.

Exploring the System:
To load all the mpinfo structures and display some data from each:
q4> load struct mpinfo from mpproc_info max

nmpinfo



q4> print -x procindex%d prochpa spinlock_depth
To display the mpcntrs data structures from a particular processor:
q4> load struct mpinfo from mpproc_info max

nmpinfo



q4> keep procindex == 1



q4> print -tx  grep mpcntrs

{% if main.adsdop %}{% include 'adsenceinline.tpl' %}{% endif %}

Another part of the mpinfo structure that is worth looking into is the mp_rq member. This is the run queue structure for this processor. Each processor has its own set of run queues. As we discussed in Chapter 5, "Process and Thread Management from the Process's Viewpoint," threads are placed on a run queue when they become ready to run. The details of how threads move on and off the run queues are covered in that chapter. Here we look at the actual structure of the queues themselves (Listing 12.2).

Listing 12.2. Declaration for the `struct mp_rq`

struct mp_rq {

     int bestq;

     int neavg_on_rq;                     /* 256X avg of neff_on_rq */

     int nready_free;                     /* active, not locked to any spu */

     int nready_free_alpha;               /* active, not locked, needs alpha */

     int nready_locked;                   /* active, locked to this spu */

     int nready_locked_alpha;             /* active, locked, needs alpha*/

     int asema_ignored;                   /* alpha related. see pm_policy.c */

        u_int ticks_last_migration;       /* ticks_since_boot at last migration */

     ulong_t cycles_last_migration;       /* IT at last migration */

     struct _horse *nexthorse;            /* next horse in the carousel (FSS) */

     lock_t *run_queue_lock;              /* lock for this processor's run Qs */

     struct mp_threadhd qs[NQS];

     int fsid_active;

     int spares[9];

};

The queues themselves are in the array of struct mp_threadhd members. The value NQS is the number of queues per processor. NQS is currently defined at 160, so there are 160 struct mp_threadhds . The struct mp_threadhd is simply a pair of pointers to threads. It is used as the head of a doubly linked list of threads:

typedef struct  mp_threadhd {

        struct kthread *th_link;   /* linked list of running threads */

       struct kthread *th_rlink;

} mp_threadhd_t;

The th_link member points to the first thread in the queue, and the th_rlink member points to the last thread in the queue. Within the threads, kthread->th_link is the forward pointer and kthread->th_rlink is the backward pointer.