6.3 OPERLOG
|
| < Day Day Up > |
|
6.3 OPERLOG
The MVS operations log (OPERLOG) function of the MVS console services component provides a sysplex-wide merged message log by using the MVS system logger services. There is only one OPERLOG log stream within the entire sysplex but the recording of the messages into the log stream can be activated on a system basis. You might have systems that need to merge in the OPERLOG log stream and other systems that can continue to record independently on their syslog.
6.3.1 Functional description
The OPERLOG log stream is a funnel-type log stream where each system connected to this log stream puts its own messages.
The messages are logged in the form of message data blocks (MDB). An MDB is a structured control block that contains a complete representation of a message, including both text and control information. These messages are written into the log stream in chronological order as they are received and processed by system logger services; this does not assure that the message are written in the log stream in the order they are generated since they might get to the logger in a different order than the one they are generated. This might be visible when you retrieve records and you can see that records from different systems are not ordered correctly by timestamp. This process can also be influence by the speed of the processor where the application using logger services is actually running.
Criticality/persistence of data
OPERLOG keeps the log records in the log stream merged from all the system in the sysplex that are connected to the log stream. Since a system can record to both OPERLOG and SYSLOG and if a system cannot record to the OPERLOG log stream it automatically switches to SYSLOG even though SYSLOG was not activated, data on OPERLOG log stream are not considered to be critical. Losing the OPERLOG capability forces the installation to fall back to individual syslog.
OPERLOG is anyway not considered the final archival place for operations log data. Usually, installations keep the active portion of the operations data in OPERLOG, that is, for a week, and then retrieve the data from the log stream to be achieved on more traditional media and readable format.
OPERLOG is usually mapped to a dedicated Coupling Facility structure. You can control OPERLOG duration of the data:
-
Through the log stream definitions AUTODELETE and RETPD if you want to let system logger to perform the cleaning of the unwanted records
-
Through your installation scheduling environment by using the IEAMDBLG Program. Refer to "IEAMDBLG Program" on page 218 for more details.
Log stream sizing
By being a funnel type log stream, OPERLOG relies on the offload process to accumulate data in time. So, no matter the size of the structure mapping the log stream, you should always expect offload at a certain point, how many and how frequent depending on the overall configuration of the OPERLOG environment.
For an initial sizing of your OPERLOG log stream, you can simply go in your current environment and see how many writes per second are happening currently on each of your system (that is, using SDSF), aggregate them and use the web-based tool to determine the size of the coupling facility structure. The tool can be located at the IBM web site:
http://www.ibm.com/servers/eserver/zseries/pso
Once that you enable the function, you can the use RMF Coupling Facility Report and SMF88 data to understand if the structure size and offload data sets size require any adjustments.
6.3.2 Definition
Subsystem definition
You can record system messages and commands to the system log (SYSLOG), the operations log (OPERLOG), or an MCS printer. The recording of the messages and commands is controlled by the hardcopy message set. The hardcopy message set represents messages that can be recorded in hardcopy on these media. It is defined in the HARDCPY definition in Parmlib CONSOLxx member. OPERLOG is defined as part of the hardcopy setup.
HARDCOPY DEVNUM {(devnum) }, {(SYSLOG) }, {(OPERLOG) }, {(devnum,OPERLOG)}, {(SYSLOG,OPERLOG)} To enable an image to use the operlog, you can defined the set up in the CONSOLxx parmlib member or use the VARY command to activate and/or deactivate it after IPL has completed.
The name of the OPERLOG log stream is fixed, SYSPLEX.OPERLOG. So, when a system is enabled to use the OPERLOG log stream, it will connect to SYSPLEX.OPERLOG log stream.
Structure definition in the CFRM Policy for the OPERLOG log stream
Before using the OPERLOG log stream, you need to define that the correspondent Coupling Facility structure in the CFRM policy and make sure that the CFRM policy has been activated in the sysplex through the SETXCF command. Here is an initial sample to define the Coupling Facility structures for the OPERLOG log stream.
Example 6-12: Structure definition for OPERLOG sample
 |
//OPERLOG JOB CLASS=A,MSGCLASS=A //POLICY EXEC PGM=IXCMIAPU //SYSPRINT DD SYSOUT=A //SYSIN DD * DATA TYPE(CFRM) STRUCTURE NAME(OPERLOG) SIZE(40448) INITSIZE(40448) PREFLIST(FACIL01,FACIL02)
|
Logger definition
Most of the installation use OPERLOG as a coupling facility log stream, although you can use a DASD-only log stream if you re running a monoplex or single system sysplex or if you want to enable OPERLOG only on one system in your sysplex. The following is an example of how to define the log stream to the LOGR policy. Remember, before executing the LOGR policy, check that the corresponding CFRM structure has been defined in the CFRM policy and activated in the sysplex. Refer to "Security definitions" on page 216 for the further informations on CFRM parameters for the OPERLOG structure.
Example 6-13: OPERLOG log stream definitions for the LOGR policy sample
|
//OPERLOG JOB CLASS=A,MSGCLASS=A //POLICY EXEC PGM=IXCMIAPU //SYSPRINT DD SYSOUT=A //SYSIN DD * DATA TYPE(LOGR) DEFINE STRUCTURE NAME(OPERLOG) LOGSNUM(1) MAXBUFSIZE(4092) AVGBUFSIZE(512) DEFINE LOGSTREAM NAME(SYSPLEX.OPERLOG) STRUCTNAME(OPERLOG) LS_DATACLAS(LOGR4K) HLQ(IXGLOGR) LS_SIZE(1024) LOWOFFLOAD(0) HIGHOFFLOAD(80) STG_DUPLEX(NO) RETPD(Note1) AUTODELETE(Note1)
|
LOGSNUM: On the structure definition, LOGSNUM specifies the number of log streams that you want to allocate to the same coupling facility structure. The value you specify determines how many pieces the structure is divided into and how much room there is in each piece. For OPERLOG, the suggested mapping is one log stream per structure.
MAXBUFSIZE: MAXBUFSIZE must be at least 4096 for OPERLOG records.
AVGBUFSIZE: On the structure definition, AVGBUFSIZE specifies the average size, in bytes, of individual log blocks (MDBs) that can be written to log streams allocated to the coupling facility. You can use the 512 value suggested that fits most of the installation or you can use the following method to evaluate the correct value for your installation.
To determine the MDB size in the operation log stream, extract data from the OPERLOG log stream using the LOGR subsystem data set interface. You can use the LOGR subsystem for existing eligible applications that need to access log stream data in data set format. The JCL for the MDB extract job are:
Example 6-14: JCL to evaluate OPERLOG AVGBUFSIZE
|
//RUN1 JOB MSGLEVEL=1,MSGCLASS=X,CLASS=A //S1 EXEC PGM=IEBGENER //SYSPRINT DD SYSOUT=X //SYSUT2 DD DSN=out.data.set,DISP=SHR //SYSUT1 DD DSN=SYSPLEX.OPERLOG, // SUBSYS=(LOGR,IXGSEXIT,'FROM=OLDEST'), // RECFM=VB,BLKSIZE=12292,LRECL=4096,DSORG=PS //SYSIN DD DUMMY ___________________________________________________________________________ From the IEBGENER output, the average MDB size was 338.21 bytes. The MDB size distribution was: ___________________________________________________________________________________________ 200> <=300> <=400> <=500> <=600> <=700> <=800> <=900> <=1000> <=1100> <1200> <=1300> <1400> 78998 65102 32308 1001 478 142 359 114 156 95 82 59 997 Maximum MDB size 60333 Minimum MDB size 234 Number of samples 179891
|
The IEBGENER utility program SYSUT1 DD-statement in the previous JCL allocates the OPERLOG log stream through the subsystem LOGR. You can use the above value to define your log stream in your LOGR policy.
HIGHOFFLOAD: The HIGHOFFLOAD parameter is used to determine when the space dedicated to the log stream in the Coupling Facility is filling up and an offload need to be initiated to re-gain available space. Since the OPERLOG log stream is a funnel type log stream, HIGHOFFLOAD should be set at 80% at least initially and then use the SMF88 report to evaluate if this value need same tuning.
LOWOFFLOAD: The LOWOFFLOAD value defines the amount of data which may be retained in the log stream interim storage following an offload process. In the OPERLOG environment, the LOWOFFLOAD value should be 0.
LS_DATACLAS: With this keyword, you can specify the SMS Dataclass that is used for the allocation of offload data sets. The Automatic Class Selection (ACS) routines can override this value so you should ensure the desired dataclass is used. In addition to the size of the data set, the dataclass defines the CI Size for the data set along with the share options. Offload data sets may use a CI size up to 24K; in general, the larger CI size provides better performance, however, if the log stream consists mostly of log blocks of a much smaller size, the 24K CI size may be wasteful
LS_SIZE: The LS_SIZE specifies the size, in 4K blocks, of the log stream DASD data sets for the log stream.
| Caution | Be sure you either specify an LS_SIZE value on the log stream definition or you use an LS_DATACLAS where you have space allocation defined. If LS_SIZE is not specified in the log stream definition, and an extent size is not specified in the data class pointed to by LS_DATACLAS, the value is taken from the ALLOCxx Parmlib member. The default value in ALLOCxx is 2 tracks. Refer to the z/OS MVS Initialization and Tuning Reference, SA22-7592. Both methods are valid, it depends on your installation and on the variety of the log streams. If you do not have a big variety of offload data set, you can use SMS dataclas to manage the allocation space and simply omit the LS_SIZE parameter. If your installation has multiple different type of offload data set that will generate too many SMS Dataclas, than you might want to consider to use LS_SIZE on each log stream. |
HLQ: The HLQ specifies the high level qualifier for both the offload data set name (and the staging data set name if used). If you do not specify a high level qualifier, a high level qualifier IXGLOGR is defaulted. The data set names format for the OPERLOG log stream data sets is hlq.SYSPLEX.OPERLOG.Annnnnnn where nnnnnnn is a running sequence number.
STG_DUPLEX and DUPLEXMODE: DUPLEX and DUPLEXMODE determine under what conditions, if any, System Logger uses a staging data set for a given log stream on a given system.
STG_DUPLEX(NO) indicates that System Logger never uses staging data sets, regardless of the nature of the connection to the structure on which the log stream resides.
STG_DUPLEX(YES) with DUPLEXMODE(UNCOND) states that System Logger should always use staging data sets for the log stream, even if the connection to the associated structure is failure independent. Even though this guarantees that there will never be a loss of data condition on the log stream, it comes at a price: every write to the log stream is gated by the speed of the I/O to the staging data set.
STG_DUPLEX(YES) with DUPLEXMODE(COND) means that System Logger only uses a staging data set if the CF on which the structure resides becomes volatile, or the structure resides in the same failure domain as the System Logger system. If none of these conditions exists, System Logger duplexes the log blocks in the System Logger dataspace. With this setting, you only pay the penalty of writing to a staging data set in those conditions that might reside in a loss of data condition due to a single failure.
Since OPERLOG can tolerate a loss of data condition, your installation can decide how sensitive is this data and set up the duplexing consequently.
AUTODELETE and RETPD: For the OPERLOG log stream, you can handle the expiration of the oldest records in different ways, mainly depending upon the need of archiving or not of the operations data.
| Notes |
|
Security definitions
The OPERLOG log stream can be protected in the Security Access Facility (SAF) LOGSTRM class. OPERLOG log stream should have READ access as default so everybody can browse the operlog data and UPDATE access level only to those users that need the capability to delete records from the log stream, that is, the userid associated with the job running the IEAMDBLG program.
Example 6-15: Sample log stream security definitions
|
RDEFINE LOGSTRM SYSPLEX.OPERLOG UACC(READ) PERMIT SYSPLEX.OPERLOG CLASS(LOGSTRM) ID(userid1) ACCESS(UPDATE) SETROPTS CLASSACT(LOGSTRM)
|
The log stream name defined to RACF is the name specified on the NAME parameter of the DEFINE LOGSTREAM statement in the LOGR policy
6.3.3 Operational considerations
The following are items to consider when you are using OPERLOG:
Enabling/Disabling OPERLOG
OPERLOG can be turned on and off on each system within the sysplex with the following command, providing the MVS system logger is activated:
Example 6-16: VARY command sample
|
V OPERLOG,HARDCPY IEE889I 20.59.31 CONSOLE DISPLAY 153 MSG: CURR=10 LIM=1500 RPLY:CURR=6 LIM=999 SYS=SC50 PFK=00 CONSOLE/ALT ID --------------- SPECIFICATIONS -------------- SYSLOG COND=H AUTH=CMDS NBUF=2 UD=Y ROUTCDE=ALL OPERLOG COND=H AUTH=CMDS NBUF=N/A UD=Y ROUTCDE=ALL V OPERLOG,HARDCPY,OFF IEE338I OPERLOG INACTIVE AS HARDCPY
|
The intended scope of an OPERLOG is a sysplex; however, a given system in the sysplex may or may not be using OPERLOG at any particular time. The operator commands that control the status of OPERLOG and the initialization parameter that activates it at IPL have a single system scope. Furthermore, a failure in OPERLOG processing on one system does not have any direct effect on the other systems. The result is that an OPERLOG may contain records from an entire sysplex or from only a subset of the systems, depending on the installation's requirements and on the environmental factors.
SDSF considerations
The following considerations apply when using SDSF and OPERLOG:
-
You can understand if you are browsing OPERLOG or SYSLOG by looking on the upper left corner of the log display. SDSF displays which media is currently browsing and you can switch from OPERLOG to SYSLOG by using the LOG OPERLOG or LOG SYSLOG command at any time.
-
If you are browsing the OPERLOG while structure rebuild takes place either due to a failure or to maintenance procedure, the log stream data will be unavailable for the time it takes to rebuild the log stream in the alternate coupling facility. For the duration of this interval, you are notified that data is not available with an error message in your upper right corner as shown in Figure 6-1.
Figure 6-1: Log stream rebuild processing -
In order to deallocate the OPERLOG log stream, all the active connections should be removed. Connections to the OPERLOG log stream are mainly SDSF users and until the last TSO user exits from SDSF, the log stream cannot be deallocated. You can discover which user is currently connected to the log stream by using the DISPLAY LOGGER command as shown in Figure 6-2.
Figure 6-2: Display Logger, Connection
IEAMDBLG Program
SYS1.SAMPLIB contains a sample program (IEAMDBLG) to read log blocks from the OPERLOG log stream and convert them to SYSLOG format. The program is an example of how to use the services of the MVS system logger to retrieve and delete records from the OPERLOG stream. It reads the records created in a given time span, converts them from Message Data Block (MDB) format to Hard-copy Log format (HCL or JES2 SYSLOG), and writes the SYSLOG-format records to a file. It also has an option to delete from the stream all the records created prior to a given date. When you use the delete option, a suggestion is to first copy the records on an alternate media and then conditionally delete them on a separate JCL step: this will ensure that you have a copy of the data before deleting. If you do not execute them on two separate conditional steps, deletion occurs simultaneously with copy without any guarantee that the copy process was successful.
6.3.4 Recovery
The OPERLOG is operationally independent of SYSLOG; that is, an installation can choose to run with either or both. A failure of the OPERLOG when the OPERLOG is active and SYSLOG (or the hardcopy message log) is not active causes SYSLOG to be activated automatically in an effort to reduce the impact to the operations and problem determination tasks. Other systems that were not affected by the failure will continue to write to the OPERLOG.
6.3.5 Performance
After your initial log stream allocation, performance data is produced in a number of forms that can be useful to determine if any adjustments are necessary to your installation:
-
SMF 88 data produced by the MVS System Logger
-
SMF 70–78 data produced by various MVS components
There are several tools which aide in the analysis of log stream performance problems.
-
IXGRPT1 formats and reports the SMF 88 data produced by the MVS System Logger
-
ERBRMFPP reports on the data captured in the SMF 70 -78 records
Operlog is a funnel type log stream: this means that we will expect offloads to be created as result of the OPERLOG process. There are some fields in the SMF88 that you might want to check against this log stream: DASD SHIFT, STRUCTURE FULL, ENTRY FULL and SC1/2/3. Refer to "SMF Type 88 records and IXGRPT1 program" on page 281 for a description of these fields and what to do in case of one of this condition happens in your installation.
|
| < Day Day Up > |
|
