Documenting the Systems

This section considers system documentation, the reasons for having it, and the kind of information that it should contain to be of any value. This task is frequently overlooked or viewed as a trivial, time-consuming background project that remains at the bottom of the priority list to be done when there's time.

System documentation, in this context, is the information that describes the entire system, its hardware and software configuration, its business function, and its relationship to other systems on the network. This documentation needs to contain information about how the disks are partitioned, any modifications that might have been made to the operating environment, or patches that were installed. These are just a few examples, of course ”many more are discussed in the following subsections.

The system manager is responsible for the provision of the IT service, which includes the systems under his control. Therefore, he is also responsible for ensuring that he has a complete inventory of those responsibilities and their relevance to the rest of the business.

The Importance of Good Documentation

System documentation is an extremely important part of system management. It has ramifications for other business functions within the company if it is not present or if it is out-of-date, for the following reasons:

System documentation is the one place where all the details of the systems can be found, without needing to assemble pieces of information from a variety of sources. When the job has been done thoroughly, this is also the only place where the system manager can be sure to find a complete and comprehensive description of the systems under his control.
The disaster recovery plan could refer to a good set of system documentation. This removes the need for duplication of effort and hence the increased risk of error. If the documentation is accurate and kept up-to-date, it is the ideal place to obtain the information that would be needed if disaster were to strike the company's computer systems.
The system documentation has a part to play in fault resolution and recovery. Suppose that a system has crashed and completely corrupted a system disk volume containing a number of partitions. The information is held in the original Solaris installation log file /var/sadm/system/install_log, but getting to it is more difficult if the disk containing /var has just crashed or if the system won't boot. Moreover, this log file will not contain details of any disk volumes that were subsequently added to the system or any repartitioning that might have been carried out. Similarly, with a failed system and a call to the Sun response center, the system engineer would normally ask for details of kernel modifications and patches installed on the system, to aid resolution of the problem. Having all this information on hand can greatly speed up fault resolution by eliminating unnecessary delays. In a mission-critical environment, this becomes even more important.
Documentation demonstrates professionalism . For example, a project manager designing a replacement for one of the existing systems under the system manager's control might require information about the system being replaced . With good documentation, this information can be provided in seconds, probably electronically if stored on the LAN as well as in hard copy.
When planning upgrades, the documentation provides comprehensive information that could highlight potential problems or conflicts.

System Information

The purpose of system documentation is to provide information about a specific system or group of systems. It is a functional document aimed at a technical audience. If disaster strikes the computer systems, a nontechnical disaster coordinator could, in the absence of the system manager, pass the system document to a Sun supplier, who would then have sufficient information to be able to organize a replacement system of the same (or equivalent) configuration. Similarly, if a major system failed, the current kernel parameters or the partition information for a failed disk volume could be easily located to assist in the system recovery.

Figure 8.1 shows an example technical data sheet. This provides a good system summary and some generic information about the system.

Figure 8.1. The technical data sheet yields important information about the system and its generic configuration.

$graphics\08fig01.gif$

The next three subsections identify the type of information that needs to be included in the system documentation to ensure that it contains a comprehensive description of the system, its configuration, and the commands that can be used to obtain the required data.

System Hardware and Device Configuration

A list of the hardware installed in the system is critical to the documentation, especially if faced with a disaster in which the computer could be completely destroyed . The same is true of device configuration information.

The following information normally is found in the system hardware and device configuration section of the documentation:

Output from the prtdiag and sysdef commands ”These commands display the current system definition, including all hardware, pseudo and system devices, loadable modules, and the values of some tunable kernel parameters.
Physical disk partition information ”This is the only place where the disk partitioning information will be held, apart from the system itself. See Listing 8.1 for a sample output from the format and prtvtoc commands.
Storage subsystems ”Include any information about RAID configurations or Solstice Disksuite metadevice usage. Even though this is technically a software issue, it is concerned with configuring the underlying hardware and should be included here.
Special devices ”This includes any information about other devices that require special consideration, such as the use of a third-party autoloading tape stacker.

Listing 8.1 shows the relevant part of the format command to list the disks attached to the system, as well as the output from running prtvtoc on two of the disk volumes. The advantage of using prtvtoc over format for this particular information is that it shows the mount point on which the partition is currently mounted (which format does not) and gives details about the geometry of the disk.

Listing 8.1 Annotated Output from the Commands `format` and `prtvtoc` Showing Physical Disk Partition Information and the Current Mount Points.

 #format  AVAILABLE DISK  SELECTIONS:   0. c0t0d0 <SUN9.0G cyl 4924 alt 2 hd 27 sec 133>   /pci@1f,4000/scsi@3/sd@0,0   1. c0t8d0 <SUN9.0G cyl 4924 alt 2 hd 27 sec 133>   /pci@1f,4000/scsi@3/sd@8,0   2. c0t9d0 <SUN9.0G cyl 4924 alt 2 hd 27 sec 133>   /pci@1f,4000/scsi@3/sd@9,0   3. c0t10d0 <SUN9.0G cyl 4924 alt 2 hd 27 sec 133>   /pci@1f,4000/scsi@3/sd@a,0   #prtvtoc  /dev/dsk/c0t0d0s2   * /dev/dsk/c0t0d0s2 partition map   *   * Dimensions:  *    512 bytes/sector  *    133 sectors/track  *     27 tracks/cylinder  *   3591 sectors/cylinder  *   4926 cylinders  *   4924 accessible cylinders  *  * Flags:  *   1: unmountable  *  10: read-only  *  *                          First     Sector    Last  * Partition  Tag  Flags    Sector     Count    Sector  Mount Directory         0      2    00          0    254961    254960   /         1      3    01     254961    610470    865430         2      5    00          0  17682084  17682083         3      4    00    2420334   4197879   6618212   /var         4      4    00    6618213   4197879  10816091   /opt         5      4    00   10816092   4668300  15484391   /application         6      4    00     865431   1246077   2111507   /usr         7      8    00    2111508    308826   2420333   /export/home  #  #prtvtoc /dev/dsk/c0t9d0s2   * /dev/dsk/c0t9d0s2 partition map*  * Dimensions:  *     512 bytes/sector*  133 sectors/track  *      27 tracks/cylinder  *    3591 sectors/cylinder  *    4926 cylinders  *    4924 accessible cylinders  *  * Flags:  *  1: unmountable  * 10: read-only  *  *  First       Sector     Last   * Partition  Tag  Flags    Sector       Count     Sector  Mount Directory  2     5    01         0  17682084  17682083        7     0    00         0  17682084  17682083   /data/tables  #

Software Configuration

This section contains information on how the system software has been configured. It is not enough to show the initial install detail here because it has most likely changed when system tuning was carried out or software was added, requiring the modification of specific parameters.

The following information normally is found in the system software configuration section of the documentation:

Solaris software option ”State here which option was used to install the operating environment, either core , end user , developer, or entire distribution. If required, use the output from the pkginfo command to include specific package information.
Name service ”Details about name service should be supplied here, including the domain name, if applicable , and the status ”for example, NIS master or slave server.
Use of the system ”Include a few words to describe the purpose of the system. Examples include file server, database server, application server, boot server, backup server, and so on.
Kernel parameters ”Include the file /etc/system to show the kernel parameters that have been modified. (See Listing 8.2 for an example of a customized configuration file.) This provides the necessary information and makes the document easy to update for further modifications.
Configuration files ”Include details of files that might be changed as a result of software requirements or security implementations . Such files include /etc/inittab, which shows the initialization levels; /etc/nsswitch.conf, which shows the order in which key files are accessed; /etc/syslog.conf, for system logging parameters; and /etc/inetd.conf, for details of disabled services for security reasons. The startup files are located in /etc/rc* and the /etc/rc*.d directories, which will be modified if applications are added or services removed.
Remote dependencies ”Provide details of services provided to other systems or file systems that are mounted from other systems. The contents of files such as /etc/dfs/dfstab could be included to show all file systems that are shared, as well as the files /etc/auto_master and /etc/auto_home to show details of automounted file systems.

Listing 8.2 The File /etc/system Displaying the Kernel Configuration Options in Use

 *ident   "@(#)system    1.1897/06/27 SMI" /* SVR4 1.5 */  ** SYSTEM SPECIFICATION FILE*  *  #Oracle Parameters  set shmsys:shminfo_shmmax=130023424  set shmsys:shminfo_shmmin=1  set shmsys:shminfo_shmmni=300  set shmsys:shminfo_shmseg=10  set semsys:seminfo_semmns=1200  set semsys:seminfo_semmni=300  set semsys:seminfo_semmsl=1000  #Set the maxusers parameter to 128 to increase the size of the task table.  set  maxusers=128  #Increase  the  number  of  descriptors  set  rlim_fd_cur=1024  set  rlim_fd_max=1024

Patch Status

An essential part of the current system configuration is the capability to quickly and easily identify the patches that have been applied to the operating environment and to application software such as Sparc compilers.

The output from the command showrev -p displays information about all patches that are currently applied to the system. The command patchadd -p also produces information about patches that are installed. The output from these commands is useful because it contains the following information:

Patch ”The identifier and version of the patch. For example, 106542-08 represents patch 106542 and version 08.
Obsoletes ”The patches that have been superceded by installing the patch.
Requires ”Any prerequisites that this patch depends on. These patches must already be installed, or the patch installation will abort.
Incompatibles ”Any patches that conflict with this patch.
Packages ”The software packages that are affected by the installation of the patch.

An example of the type of information produced is shown in Listing 8.3.

Listing 8.3 The Output from the Command `showrev -p` on an Intel Platform Running Solaris 7

 #showrev -p  Patch: 106542-08 Obsoletes: 106833-03, 106914-04, 106977-01, 107440-01, 107032-01,  107118-05, 107447-01 Requires: 107545-02 Incompatibles:  Packages: SUNWkvm,  SUNWcsu, SUNWcsr, SUNWcsl, SUNWcar, SUNWesu, SUNWarc, SUNWatfsr, SUNWdpl, SUNWhea,  SUNWipc, SUNWtoo, SUNWpcmci, SUNWpcmcu, SUNWscpu, SUNWtnfc, SUNWvolr  Patch: 107545-03 Obsoletes:  Requires:  Incompatibles:  Packages: SUNWcsu, SUNWcsr  Patch: 106794-03 Obsoletes:  Requires:  Incompatibles:  Packages: SUNWcsu, SUNWhea  Patch: 107452-02 Obsoletes:  Requires: 107118-03 Incompatibles:  Packages: SUNWcsu  Patch: 107455-03 Obsoletes:  Requires:  Incompatibles:  Packages: SUNWcsu  Patch: 107793-01 Obsoletes:  Requires:  Incompatibles:  Packages: SUNWcsu  Patch: 108302-01 Obsoletes:  Requires:  Incompatibles:  Packages: SUNWcsu  Patch: 108483-01 Obsoletes:  Requires:  Incompatibles:  Packages: SUNWcsu  Patch: 107457-01 Obsoletes:  Requires:  Incompatibles:  Packages: SUNWcsr  Patch: 106945-02 Obsoletes:  Requires:  Incompatibles:  Packages: SUNWcsr  Patch: 107894-04 Obsoletes: 108123-01, 108238-01 Requires:  Incompatibles:  Packages: SUNWtltk  Patch: 108375-01 Obsoletes: 107882-10 Requires:  Incompatibles:  Packages:  SUNWdtbas, SUNWdtdte, SUNWdtinc, SUNWdtmad  Patch: 106935-03 Obsoletes:  Requires:  Incompatibles:  Packages: SUNWdtbas  Patch: 108220-01 Obsoletes:  Requires:  Incompatibles:  Packages: SUNWdtbas  Patch: 107588-01 Obsoletes:  Requires:  Incompatibles:  Packages: SUNWaccu  Patch: 106979-09 Obsoletes:  Requires: 107457-01 Incompatibles:  Packages:  SUNWadmap, SUNWadmc  Patch: 108663-01 Obsoletes:  Requires:  Incompatibles:  Packages: SUNWadmfw  Patch: 107888-08 Obsoletes: 107002-01 Requires:  Incompatibles:  Packages:  SUNWdtdte, SUNWdtdst, SUNWdtma  Patch: 107181-12 Obsoletes:  Requires:  Incompatibles:  Packages: SUNWdtdte  Patch: 107023-05 Obsoletes:  Requires: 108375-01 Incompatibles:  Packages:  SUNWdtdmn, SUNWdtdst, SUNWdtma  Patch: 108222-01 Obsoletes:  Requires:  Incompatibles:  Packages: SUNWdtdmn  Patch: 106737-03 Obsoletes:  Requires:  Incompatibles:  Packages: SUNWoldst  Patch: 106953-01 Obsoletes:  Requires:  Incompatibles:  Packages: SUNWbnuu  Patch: 107039-01 Obsoletes:  Requires:  Incompatibles:  Packages: SUNWdoc  Patch: 107886-06 Obsoletes: 107220-02 Requires: 106935-03 Incompatibles:  Packages: SUNWdticn, SUNWdtdst, SUNWdthev, SUNWdtma  Patch: 107201-11 Obsoletes:  Requires: 108375-01, 107888-08 Incompatibles:  Packages:  SUNWdtdst,  SUNWdtma  Patch: 108344-02 Obsoletes:  Requires: 108375-01 Incompatibles:  Packages:  SUNWdtezt  Patch: 106328-06 Obsoletes:  Requires:  Incompatibles:  Packages: SUNWlibC  Patch: 107339-01 Obsoletes:  Requires:  Incompatibles:  Packages: SUNWkcsrt,  SUNWkcspg  Patch: 106961-01 Obsoletes:  Requires:  Incompatibles:  Packages: SUNWman  Patch: 107116-03 Obsoletes:  Requires:  Incompatibles:  Packages: SUNWpcu, SUNWpsu  Patch: 107637-03 Obsoletes:  Requires:  Incompatibles:  Packages: SUNWxi18n,  SUNWxim  Patch: 107685-01 Obsoletes:  Requires:  Incompatibles:  Packages: SUNWsndmu  Patch: 107973-01 Obsoletes:  Requires:  Incompatibles:  Packages: SUNWsutl  Patch: 107260-01 Obsoletes:  Requires:  Incompatibles:  Packages: SUNWvolu  #

This information can be invaluable if the system crashes or needs to be reinstalled from scratch. In the former instance, the relevant page of documentation can be sent by fax or email to the Sun response center very quickly, greatly assisting the fault-resolution process.

Application Information

In addition to the system configuration information, there is a requirement to document the application software installed on a specific system or group of systems. This information is of value because it is frequently the only record that describes what is installed on the whole system. Of course, each application should have its own documentation, but this applies to the whole system, which might run several applications. The information contained within the documentation is useful in a number of situations, such as considering disaster recovery initiatives, capacity planning for future upgrades, or determining the number of licenses for renewal purposes.

Usually, this type of documentation is split into two categories: applications that are developed and maintained within the company (known as in-house software) and applications that are purchased from a third-party supplier ”that is, commercial software or outsourced developments. The categories are discussed separately in the following two subsections.

In-House Applications

For application software that is developed and maintained within the company, the following information should be recorded about each application:

The name of the application and a basic description of its components and function within the business ”for example: "This is an Oracle forms/ reports -based application that queries and updates the main customer database."
Any software dependencies that exist for the application ”for example, the assumption that the Oracle Developer runtime product is already installed so that forms and reports can be run.
The users of the application ”for example, sales, accounts, marketing. An indication of how many users actually make use of the application should also be included.
The current version of the software that is installed.
The installed location of the software on the system and the amount of disk space required. Include the location of any installation media and documentation.
Contact details of the development section and, if different, the application support authority.

Third-Party Applications

Third-party application software is usually subject to a formal maintenance and support agreement with the supplier of the software. The following information should be present for each application listed:

The name of the application package and details of the vendor and supplier (if they are different).
A brief description of what the package does, as well as what department(s) make use of it.
The version number of the software installed on the system, along with also details of any patches that the supplier has delivered (and are installed).
Details of any existing support and maintenance agreement for the software, including the level of support, the renewal date, the number of licenses purchased, and how to contact the support authority, together with any account numbers/serial numbers that might be necessary to obtain support.
The installed location of the software on the system and the amount of disk space required. Also include the location of the installation media and supporting documentation.

Relational Database Information

Relational database management systems (RDBMS) warrant their own section in the system documentation because of their importance to the business. They frequently hold business-critical information such as customer data, sales or marketing information, and personnel data. For this reason alone, each database that resides on the system should be clearly identified and documented.

A further reason for documenting separately the existence of a database on the system is that it might be maintained by the "database administration (DBA) section," a section that is often separate from the system administrators and one that also might administer a number of different databases on different systems. Ideally, the two sections should be co-located, but this is not always the case. If the DBA section maintains its own documentation for the database, the central system documentation will be incomplete, and multiple sources will be needed to get the complete picture ”that's a far from ideal situation. A better solution should ensure that the DBA section maintains this part of the document so that the entire system documentation remains in a single reference document.

The information contained within the document needs to include the following details:

The RDBMS product and version installed on the system, together with any patches that have been subsequently applied to the product
Details of support and maintenance agreements, including contact numbers for the DBA section as well as the supplier (such as Oracle), and any account numbers/serial numbers that might be required to obtain support in the absence of a DBA
A brief description of the database instance and its function within the company
The location of the installation media and supporting documentation
Information about the size of the database and location on the system of its components, such as tablespaces, redo logs, archive logs, and so on

Beware of Raw Disk Usage

Most, if not all, RDBMS products allow the storage of the database either in a file system, as with any other file, or with raw disk (bypassing the UNIX I/O buffering). The advantage of using raw disk is that there is a performance gain through direct communication with the disk. The downside is that it requires a greater administration overhead, and it is not visible to the unfamiliar person, unlike the presence of a file system, for example.

Raw disk partitions that are used as part of a relational database are not mounted like file systems. They are only defined in the database configuration file, usually by a symbolic link to the raw disk device in question. Volume management software, such as that supplied by Veritas, is often used to manage the disk allocation on systems configured for use with relational databases. A good idea is to store a list of managed volumes and mirrors on a regular basis using the command vxprint , for example, which is part of the Veritas volume management software.

If the raw disk option is being used, it must be clearly identified, as the following example demonstrates.

Suppose that a system administrator is hired who has not previously used a relational database environment. If he is requested to create a new file system, he might look at the format command to see the disks available to him and then he might use the prtvtoc command to see how a disk is partitioned and to see any mount points associated with the partitions. After checking the file /etc/vfstab to check for any file systems not mounted, the system administrator could assume that a partition that is not referenced is actually free for use, when in reality it is being used by the database ”the problem is that it isn't documented clearly. If this system administrator ran the newfs command on the partition to create the file system, the previous contents would be lost and the database would be trashed.

The impact of this scenario, although avoidable, is potentially severe:

The database will have to be recovered, and there is the potential for loss of business.
The cause of the problem will not be immediately clear, creating a delay while it is investigated.
The system administrator won't come clean because he is not aware of having done anything wrong!

Show Database Instances and Their Purpose

The main reason for documenting existing database instances is that there might be a requirement to re-create the instance if a serious failure or disaster occurs. If the database uses UNIX file systems for the storage of the database, then the recovery can be carried out using the normal file system restore from the latest backup. However, if raw disk is being used, then recovery is more complicated and the database instance must be created and initialized before a restore can be carried out. A further reason for documenting this information is that it is useful for familiarizing a new department staff member with the configuration. The documentation should contain the following information for each instance running on the system:

The database name. This is the unique name by which the instance is identified. For example, in Oracle, it is the ORACLE_SID variable; in Informix, it is the DBSERVERNAME parameter in the tbconfig file.
A brief description of the database, what it is used for, who uses it, and how many people use it (approximately).
The type of storage used. This could be a file system, raw disk, or possibly a combination of both.
Database creation and configuration information. Most of the required information can be derived from the configuration file and from a query of some of the system tables, showing the tablespaces, their size, and the location of the data files, for example.

Keeping It Up-to-Date

Probably the most important aspect of documenting the systems in use is to keep everything up-to-date. Documentation is of any real value only if it contains information about the current state of the systems. This is the most frequently omitted part of documentation because it is thought to be complete once the document has been created rather than being an ongoing task requiring regular attention. This task is also often seen as being "boring" when there are so many more interesting things to do, so it finds itself at the bottom of the priority pile ”as a result, the documentation quickly becomes outdated and loses any value it had. One way of making the documentation easier to update is to include the output of Solaris commands, such as the one used in the earlier section "Patch Status." If a new patch is installed, the documentation can be quickly and simply updated to reflect the change by running the command and "pasting" the output ”this also makes the updating more likely to be done altogether.

Presentation or Content

Including command output can affect the presentation quality of the document. Although it is important to present the system documentation professionally, the content and ease of update should be the overriding priority.

The system manager must ensure that the system documentation is regularly updated, either each week or, at the most, each month. The best policy, of course, is to update the document as part of carrying out a change, while it is still fresh in the mind of the system administrator. It may not always be practical to do this, but the longer the interval is between updates, the greater the risk of omission or errors becomes.

Top