Remember to use the same OS

Remember to use the same OS!

Following a hunch (in other words, a fellow OS support engineer said, "And what OS are you running? Try using a SPARCstation 10 system instead!"), we moved the analysis work over to a lab system called "sun4m-23," a SPARCstation 10 running Solaris 2.3 FCS. Here is what we found.

 sun4m-23...  modinfo  grep semsys  60 fc2dc000  1b0c  53   1  semsys (System V semaphore facility)  sun4m-23...  su  Password:  #  adb -k /dev/ksyms /dev/mem  physmem 1e15  ipcaccess/10i  ipcaccess:  ipcaccess:      save    %sp, -0x60, %sp                   ld      [%i2 + 0x4], %o0                   orcc    %g0, %o0, %g0                   bne,a   ipcaccess + 0x1c                   ld      [%i0], %o2                   ba      ipcaccess + 0x7c                   clr     %i1                   cmp     %o0, %o2                   be,a    ipcaccess + 0x6c                   ld      [%i0 + 0x10], %i0  $q  # 

Ah, that's much better! But why was there a difference between the two systems in the first place?

What we failed to remember was that the SS20 running Solaris 2.3 being used to analyze the SS10 crash was actually running a special version of Solaris 2.3 known as Hardware Release 5/94. 5/94 was specifically generated and released to support the newer members of the Sun family of hardware, which included the SPARCstation 20. While the kernel architecture was the same, thus allowing for analysis of SS10 crashes, some of the software on 5/94 had been enhanced, such as the semsys module, making comparison at the assembly instruction level unfeasible.

To be on the safe side, we redid the analysis on a SS10 running Solaris 2.3 FCS. The results were the same as those given on the SS20, with the exception of the code in semsys .

So, it appears that the customer's semsys ipcaccess() routine was corrupt. How was that possible? Let's explore the possibilities.