8.2 Estimating log stream sizes

 < Day Day Up > 



8.2 Estimating log stream sizes

Log streams consist of two portions—the part kept in interim storage (in the CF or a staging data set), and the part kept in offload data sets. For the most efficient performance and use of space, you need to size the two of these individually.

8.2.1 Sizing offload data sets

We start with the easier one—the offload data sets. The sizing of these is relatively simple. The total of all the offload data sets for a log stream should be enough to hold the maximum amount of data you plan to keep in the log stream. For simplicity, assume that you will have a maximum of 168 offload data sets per log stream.

For an active exploiter, you would generally not expect to see more than one or two offload data sets, and each would typically not contain many log records. Therefore, the LS_SIZE for these offload data sets would typically be in the range of 10000 to 20000 (40 MB to 80 MB).

For the funnel-type exploiters, the first thing we need to know is how long you will keep the data in the log stream. This could be as small as a number of hours, or as long as many days. The duration will vary from one exploiter to another, and from one installation to another.

The other information you need to know is how much data is moved to the offload data sets in a busy hour or a busy day. This information can be obtained from the IXGRPT1 report, as described in "BYT Deleted interim ST w/DASD". Refer to "IXGRPT1 Field Summary and IXGSMF88 Cross Reference" on page 283 for a description of the field. Remember that each record in the IXGRPT1 report only reflects the data for one SMF interval, so you will need to total all the intervals for the duration you are interested in.

Once you know roughly how much data you wish to keep in the log stream, you need to decide on the size of each offload data set. You have two conflicting requirements here:

  • You don't want to make the offload data sets so large that System Logger has difficulty finding that much free space on a volume. Also, as discussed in "Deleting log data" on page 62, old offload data sets may only get deleted when System Logger allocates a new offload data set, so if the offload data sets are many times larger than necessary, you could end up just wasting DASD space because new offload data sets rarely get allocated and old ones could be kept on DASD for far longer than necessary.

  • On the other hand, having to allocate a new offload data set slows down the offload process. And every time you go through allocation, you face the risk that you won't be able to obtain the required space, or of contention problems within the allocation process, both of which will stop the offload from proceeding. So, you don't want to be allocating offload data sets too frequently either.

On balance then, try to select a size that will hold at least 10 offload's worth of log data, while staying within an amount of space that is easily obtainable in your installation. Also, try to select a size that does not hold more than one day's worth of data.

8.2.2 Sizing interim storage

Whereas the sizing of offload data sets is driven largely by the need to ensure that you have enough space for all the log data, the size of the interim storage is driven by the need to deliver the required performance and also by the need to ensure that interim storage does not fill up on you.

Sizing CF-Structure log streams

The appropriate sizing of the interim storage part of CF log streams is vital in ensuring that System Logger provides the performance and availability you require. However, the method used to calculate the size differs depending on whether the log stream is a funnel-type or active log stream.

Sizing interim storage for funnel-type CF log streams

For funnel-type exploiters, the CF part of the log stream must be large enough that you don't offload too frequently, while at the same time ensuring that you can hold all the log data that will be generated at peak times within the desired offload rate.

Ideally, offload processing should not be invoked more frequently than once every 30–60 seconds. On the other hand, if you run for a whole SMF interval at peak times and have no offloads for a log stream, the interim storage is probably larger than necessary.

To estimate the amount of space required in the CF part of the log stream to hold this much data, you need to use an IXGRPT1 report. Example 8-1 contains sample JCL to run an IXGRPT1 report for just the log stream you are interested in. You should run this against one day's worth of SMF 88 data for a representative busy day.

Example 8-1: Sample job to report on a single log stream

start example
 //IXGRPT1L JOB 'Sample 88 pgm','Stephen Anania', //  MSGLEVEL=(1,1),CLASS=A,NOTIFY=&SYSUID.,MSGCLASS=H //* ******************************************************************* //* This jobs formats the SMF88s for a single log stream.              * //*                                                                   * //* Unfortunately, the SORT SYSIN does not support symbols, so you    * //* must use ISPF EDIT CHANGE command to set the desired logstream.   * //* Update 'logstream_name' to the name of the logstream you want.    * //*                                                                   * //* ******************************************************************* //   SET    SMFIN1=#@OPR17.SMFDATA.G0003V00 //   SET    LIBPRFX=CEE //   SET    RPTOBJ=SYS1.SAMPLIB(IXGRPT1L) //EXTRACT  EXEC PGM=IFASMFDP //SYSUDUMP DD  DUMMY //SYSPRINT DD  SYSOUT=* //SMFGOO   DD  DISP=SHR,DSN=&SMFIN1 //SMFALL   DD  DSN=&&SMF88S,UNIT=SYSDA,DISP=(NEW,PASS), //            SPACE=(CYL,(5,5)) //SYSIN    DD *   INDD(SMFGOO,OPTIONS(DUMP))   OUTDD(SMFALL,TYPE(88)) /* //* SORT EACH SMF INPUT DSN BY TIMESTAMP AND LOGSTREAM NAME //SORT1    EXEC PGM=SORT //SYSOUT   DD  SYSOUT=* //SORTIN   DD  DSN=&SMF88S,DISP=(OLD,KEEP) //SORTOUT  DD  DSN=&&TEMP1,DISP=(NEW,PASS),UNIT=SYSDA, //             SPACE=(CYL,(1,1)) //SYSIN    DD  * OPTION VLSHRT    SORT  FIELDS=(133,8,BI,A,                  105,26,CH,A)    INCLUDE COND=(23,2,BI,EQ,X'0001',AND,                 105,26,CH,EQ,C'logstream_name') /* //ASMC        EXEC PGM=ASMA90,REGION=0M,PARM='OBJECT,NODECK'   * //SYSLIB      DD DSN=SYS1.MACLIB,DISP=SHR          * LIBRARY DSN //SYSLIN      DD DSN=&&OBJ(IXGR1A),SPACE=(TRK,(10,30,10),,,ROUND), //            UNIT=3390,DISP=(MOD,PASS), //            DCB=(BLKSIZE=3120,LRECL=80,RECFM=FB) * ASM OBJ //SYSPRINT    DD SYSOUT=* //SYSUT1      DD UNIT=SYSDA,SPACE=(CYL,(5,5)) //SYSIN       DD DSN=SYS1.SAMPLIB(IXGR1A),DISP=SHR * ASM SOURCE //* //* ******************************************************************* //* LINK/EDIT, AND EXECUTE OBJECT SAMPLIB MEMBER USING LE LIBRARIES. //* ******************************************************************* //LKED     EXEC PGM=IEWL,REGION=1024K //SYSLIB   DD  DSN=&LIBPRFX..SCEELKED,DISP=SHR    * CATALOGUED PROC //SYSPRINT DD  SYSOUT=* //SYSLIN   DD  DSN=&RPTOBJ.,DISP=SHR              * //         DD  DDNAME=SYSIN //SYSLMOD  DD  DSN=&&LOADMOD(GO),DISP=(MOD,PASS),UNIT=SYSDA, //             SPACE=(1024,(50,20,1))             * LOAD MODULE //SYSUT1   DD  DSN=&&SYSUT1,UNIT=SYSDA,SPACE=(1024,(200,20)), //             DCB=BLKSIZE=1024 //SYSIN    DD  DSN=*.ASMC.SYSLIN,DISP=(MOD,PASS)  * ASM OBJ (IN) //GO       EXEC PGM=*.LKED.SYSLMOD,COND=(4,LT,LKED), //             REGION=2048K //STEPLIB  DD  DSN=&LIBPRFX..SCEERUN,DISP=SHR     * CATALOGUED PROC //SYSPRINT DD SYSOUT=* //CEEDUMP  DD SYSOUT=* //SYSUDUMP DD SYSOUT=* //SMFDATA  DD DSN=*.SORT1.SORTOUT,DISP=(MOD,PASS) //RPORT88  DD SYSOUT=*                            * IXGRPT1 OUTPUT 
end example

In the resulting report, select the interval that has the highest "BYT Written to interim storage" value. Divide this value by the number of minutes in the interval to get the per-minute write rate. Increase this value by 50% to allow for unexpected spikes in activity and control data. The resulting value indicates how much space there should be in the interim storage for this log stream between the HIGHOFFLOAD and LOWOFFLOAD thresholds [1]. Because the LOWOFFLOAD threshold for funnel-type log streams should always be 0%, we can ignore this value in our calculations.

For example, assume the SMF88SWB for the busiest interval is 31493284 and the SMF interval is 15 minutes. Dividing the 31493284 by 15 minutes gives a write rate of roughly 2 MB a minute. Increasing by 50% brings this up to 3 MB a minute. If the HIGHOFFLOAD threshold for this log stream is 70%, the CF structure should be sized so that the interim storage for this log stream is roughly 4.25 MB (3 MB / 70%). This would result in an offload approximately every minute during peak times. We will come back to the values to use for HIGHOFFLOAD and LOWOFFLOAD in 8.4.1, "High and low offload thresholds for funnel-type log streams" on page 274.

When calculating the structure size, you need to remember that the space in the structure is divided equally between all the connected log streams in that structure. You also need to allow something for the control space in the structure. So, if the structure normally contains 10 connected log streams, you would size the structure to be 4.25 MB x 10 plus 4 MB (based on CFLevel 12) for control space, giving a structure size of 46.5 MB. This assumes that all the log streams have similar write rates—if some of the log streams sharing the same structure have higher write rates, you should use the write rate of those log streams to determine the structure size.

Sizing interim storage for active CF log streams

For active exploiters, you must use a different methodology to calculate the amount of space in the CF part of the log stream. This is because you ideally want all log data to be deleted before it is offloaded to DASD. To achieve this, you need to work with both the amount of space available in the interim storage of this log stream, and also the HIGHOFFLOAD and LOWOFFLOAD thresholds.

Once again, we must use information from the IXGRPT1 report. There are two indicators that you must watch for:

  • The frequency of offload processing. At peak times, there should not be more than one offload every 30 seconds, and at a minimum there should be at least one offload per SMF interval.

  • The amount of data that is moved to an offload data set. This information is kept in field SMF88LDB in the SMF Type 88 record, and is shown in the field entitled "BYT Written to DASD" in the IXGRPT1 report. For an active log stream, this should always be zero.

To size interim storage for an active log stream, you need to arrive at a size, and LOWOFFLOAD and HIGHOFFLOAD values, that will result in about one offload per minute at peak times and 0 bytes being moved to offload data sets.

To do this, we need some more information from the IXGRPT1 report. We need to know the rate a which data is being written to the log stream—we can get that from the "BYT Written to interim storage" field. Remember to divide this value by the number of minutes in the SMF interval.

We also need to know how often data is being logically deleted from the log stream (using IXGDELET). If there are no IXGDELETs issued between two consecutive offloads, System Logger has no choice but to move the oldest data from interim storage to an offload data set. For some exploiters, like CICS, you can affect how often an IXGDELET is done by adjusting the Activity Keypoint Frequency. However for exploiters where you cannot control this, you need to tune the frequency of offloads by adjusting the interim storage size—the larger the interim storage, the less frequently will an offload be initiated. You want to make sure there is at least one IXGDELET per offload. You can find the number of IXGDELETs in the interval by summing the "# Deletes w/o DASD write" and "# Deletes w/ write fields". Divide this value by the number of minutes in the SMF interval to determine the number of IXGDELETs per minute.

So, now we know how many bytes are being written per minute, and how many IXGDELETs are being issued per minute. If you have more than one IXGDELET per minute, you should aim to have one offload per minute. If you only have one IXGDELET every x minutes, you should aim to have one offload every x minutes.

To calculate the interim storage size, multiply the write rate per minute by the number of minutes between offloads. Increase this value by 50% to allow for control information and workload spikes. The result is the amount of space in the interim storage between the LOWOFFLOAD and HIGHOFFLOAD thresholds. To get the total interim storage space for this log stream, divide this amount by the difference between the HIGHOFFLOAD and LOWOFFLOAD percents. So, if the amount of data is 3 MB, and the HIGHOFFLOAD is set to 80 and the LOWOFFLOAD is set to 60, you should have 3 MB / (80%-60%), or 15 MB of interim storage for this log stream. We will come back to the values to use for HIGHOFFLOAD and LOWOFFLOAD in 8.4.2, "High and low offload thresholds for active log streams" on page 274.

When calculating the structure size, you need to remember that the space in the structure is divided equally between all the connected log streams in that structure. You also need to allow something for the control space in the structure. So, if the structure normally contains 5 connected log streams, you would size the structure to be 15 MB x 5 plus 4 MB (based on CFLevel 12) for control space, giving a structure size of 79 MB. This assumes that all the log streams have similar write rates—if some of the log streams sharing the same structure have higher write rates, you should use the write rate of those log streams to determine the structure size.

Sizing DASD-only log streams

The appropriate sizing of the interim storage part of DASD-only log streams is important in ensuring that System Logger provides the performance and availability you require. However, as for CF log streams, the method used to calculate the size differs depending on whether the log stream is a funnel-type or active log stream.

Sizing interim storage for funnel-type DASD-only log streams

For funnel-type exploiters, the staging data set part of the log stream must be large enough that you don't offload too frequently, while at the same time ensuring that you can hold all the log data that will be generated at peak times within the desired offload rate.

Ideally, offload processing should not be invoked more frequently than once every 30–60 seconds. On the other hand, if you run for a whole SMF interval at peak times and have no offloads for a log stream, the interim storage (staging data set) is probably larger than necessary.

To estimate the amount of space required in the staging data set part of the log stream to hold this much data, you need to use an IXGRPT1 report. Example 8-2 contains sample JCL to run an IXGRPT1 report for just the logstream you are interested in. Again, you should run this against one day's worth of SMF 88 data for a representative busy day.

Example 8-2: Sample job to report on a single log stream

start example
 //IXGRPT1L  JOB  'Sample 88 pgm','Stephen Anania', //  MSGLEVEL=(1,1),CLASS=A,NOTIFY=&SYSUID.,MSGCLASS=H //* ******************************************************************* //* This jobs formats the SMF88s for a single logstream.              * //*                                                                   * //* Unfortunately, the SORT SYSIN does not support symbols, so you    * //* must use ISPF EDIT CHANGE command to set the desired logstream.   * //* Update 'logstream_name' to the name of the logstream you want.    * //*                                                                   * //* ******************************************************************* //   SET    SMFIN1=#@OPR17.SMFDATA.G0003V00 //   SET    LIBPRFX=CEE //   SET    RPTOBJ=SYS1.SAMPLIB(IXGRPT1L) //EXTRACT  EXEC PGM=IFASMFDP //SYSUDUMP DD  DUMMY //SYSPRINT DD  SYSOUT=* //SMFGOO   DD  DISP=SHR,DSN=&SMFIN1 //SMFALL   DD  DSN=&&SMF88S,UNIT=SYSDA,DISP=(NEW,PASS), //            SPACE=(CYL,(5,5)) //SYSIN    DD   *   INDD(SMFGOO,OPTIONS(DUMP))   OUTDD(SMFALL,TYPE(88)) /* //* SORT EACH SMF INPUT DSN BY TIMESTAMP AND LOGSTREAM NAME //SORT1    EXEC PGM=SORT //SYSOUT   DD  SYSOUT=* //SORTIN   DD  DSN=&SMF88S,DISP=(OLD,KEEP) //SORTOUT  DD  DSN=&&TEMP1,DISP=(NEW,PASS),UNIT=SYSDA, //             SPACE=(CYL,(1,1)) //SYSIN    DD  * OPTION VLSHRT    SORT  FIELDS=(133,8,BI,A,                  105,26,CH,A)    INCLUDE COND=(23,2,BI,EQ,X'0001',AND,                 105,26,CH,EQ,C'logstream_name') /* //ASMC        EXEC PGM=ASMA90,REGION=0M,PARM='OBJECT,NODECK'   * //SYSLIB      DD DSN=SYS1.MACLIB,DISP=SHR          * LIBRARY DSN //SYSLIN      DD DSN=&&OBJ(IXGR1A),SPACE=(TRK,(10,30,10),,,ROUND), //            UNIT=3390,DISP=(MOD,PASS), //            DCB=(BLKSIZE=3120,LRECL=80,RECFM=FB) * ASM OBJ //SYSPRINT    DD SYSOUT=* //SYSUT1      DD UNIT=SYSDA,SPACE=(CYL,(5,5)) //SYSIN       DD DSN=SYS1.SAMPLIB(IXGR1A),DISP=SHR * ASM SOURCE //* //* ******************************************************************* //* LINK/EDIT, AND EXECUTE OBJECT SAMPLIB MEMBER USING LE LIBRARIES. //* ******************************************************************* //LKED     EXEC PGM=IEWL,REGION=1024K //SYSLIB   DD  DSN=&LIBPRFX..SCEELKED,DISP=SHR    * CATALOGUED PROC //SYSPRINT DD  SYSOUT=* //SYSLIN   DD  DSN=&RPTOBJ.,DISP=SHR              * //         DD  DDNAME=SYSIN //SYSLMOD  DD  DSN=&&LOADMOD(GO),DISP=(MOD,PASS),UNIT=SYSDA, //             SPACE=(1024,(50,20,1))              * LOAD MODULE //SYSUT1   DD  DSN=&&SYSUT1,UNIT=SYSDA,SPACE=(1024,(200,20)), //             DCB=BLKSIZE=1024 //SYSIN    DD  DSN=*.ASMC.SYSLIN,DISP=(MOD,PASS) * ASM OBJ (IN) //GO       EXEC PGM=*.LKED.SYSLMOD,COND=(4,LT,LKED), //             REGION=2048K //STEPLIB  DD  DSN=&LIBPRFX..SCEERUN,DISP=SHR    * CATALOGUED PROC //SYSPRINT DD  SYSOUT=* //CEEDUMP  DD  SYSOUT=* //SYSUDUMP DD  SYSOUT=* //SMFDATA  DD DSN=*.SORT1.SORTOUT,DISP=(MOD,PASS) //RPORT88  DD SYSOUT=*                           * IXGRPT1 OUTPUT 
end example

In the resulting report, select the interval that has the highest "BYT Written to interim storage" value—this identifies the busiest interval. Unlike CF log streams, the "element size" (actually the staging data set CI Size) is sufficiently large that we do need to take it into account. So, now we check the "Average Buffer Size" value for that interval. Divide the average buffer size value by 4000 and round up to the next whole number—this is the number of CIs required in the staging data set for each IXGWRITE. Next divide the "# Writes invoked" value by the number of minutes in the SMF interval—this gives us the number of IXGWRITEs per minute. Now multiply that number by the number of CIs required for each IXGWRITE and that gives us the number of CIs being written to this log stream per minute. Increase this value by 50% to allow for unexpected spikes in activity and control data. The resulting value indicates how much space there should be in the interim storage for this log stream below the HIGHOFFLOAD threshold, assuming we have one offload per minute. The LOWOFFLOAD threshold for funnel-type log streams should always be 0%, so we can ignore that value in these calculations

For example, assume the average buffer size for the busiest interval is 4100, the number of writes invoked is 45000, and the SMF interval is 15 minutes. Dividing the 45000 by 15 minutes gives a write rate of 3000 writes a minute. Next we divide the average buffer size (4100) by 4000 and round up—this gives us a value of 2 CIs per IXGWRITE. Increasing by 50% brings this up to 3 CIs per IXGWRITE. Multiply this by the number of IXGWRITEs per minute and we get a value of 9000 CIs per minute. So, the staging data set space below the HIGHOFFLOAD threshold should be 9000 CIs. If the HIGHOFFLOAD threshold for this log stream is 70%, the data set should be sized to be roughly 13000 CIs(9000 / 70%). This would result in an offload every minute during peak times. We will come back to the values to use for HIGHOFFLOAD and LOWOFFLOAD in 8.4.1, "High and low offload thresholds for funnel-type log streams" on page 274.

Sizing interim storage for active DASD-only log streams

For active exploiters, you must use a different methodology to calculate the amount of space in the staging data set part of the log stream. This is because you ideally want all log data to be deleted before it is offloaded to the offload data sets. To achieve this, you need to work with both the amount of space available in the interim storage of this log stream, and also the HIGHOFFLOAD and LOWOFFLOAD thresholds.

Once again, we must use information from the IXGRPT1 report. There are two indicators that you must watch for:

  • The frequency of offload processing. At peak times, there should not be more than one offload every 30 seconds, and at a minimum there should be at least one offload per SMF interval.

  • The amount of data that is moved to an offload data set. This information is kept in field SMF88LDB in the SMF Type 88 record, and is shown in the field entitled "BYT Written to DASD" in the IXGRPT1 report. For an active log stream, this should always be zero.

To size interim storage for an active log stream, you need to arrive at a size, and LOWOFFLOAD and HIGHOFFLOAD values, that will result in about one offload per minute at peak times and 0 bytes being moved to offload data sets.

To do this, we need some more information from the IXGRPT1 report. We need to know the rate at which IXGWRITEs are being issued, and the number of CIs used by each IXGWRITE. Using the IXGRPT1 report, select the interval that has the highest "BYT Written to interim storage" value—this identifies the busiest interval. Next we check the "Average Buffer Size" value for that interval. Divide the average buffer size value by 4000 and round up to the next whole number—this is the number of CIs required in the staging data set for each IXGWRITE. Next divide the "# Writes invoked" value by the number of minutes in the SMF interval—this gives us the number of IXGWRITEs per minute. Now multiply that number by the number of CIs required for each IXGWRITE and that gives us the number of CIs being written to this log stream per minute. Increase this value by 50% to allow for unexpected spikes in activity and control data. The resulting value indicates how much space there should be in the interim storage for this log stream between the HIGHOFFLOAD and LOWOFFLOAD thresholds, assuming we have one offload per minute.

We also need to know how often data is being logically deleted from the log stream (using IXGDELET). If there are no IXGDELETs issued between two consecutive offloads, System Logger has no choice but to move the oldest data from interim storage to an offload data set. For some exploiters, like CICS, you can affect how often an IXGDELET is done by adjusting the Activity Keypoint Frequency. However for exploiters where you cannot control this, you need to tune the frequency of offloads by adjusting the interim storage size—the larger the interim storage, the less frequently will an offload be initiated. You want to make sure there is at least one IXGDELET per offload. You can find the number of IXGDELETs in the interval by summing the "# Deletes w/o DASD write" and "# Deletes w/ write fields". Divide this value by the number of minutes in the SMF interval to determine the number of IXGDELETs per minute.

So, now we know how many CIs are being written per minute, and how many IXGDELETs are being issued per minute. If you have more than one IXGDELET per minute, you should aim to have one offload per minute. If you only have one IXGDELET every x minutes, you should aim to have one offload every x minutes.

To calculate the interim storage size, multiply the number of CIs being written per minute by the number of minutes between offloads. The result is the amount of space in the interim storage between the HIGHOFFLOAD and LOWOFFLOAD thresholds. To get the total interim storage space for this log stream, divide this amount by the difference between the HIGHOFFLOAD and LOWOFFLOAD percents. So, if the amount of data is 3MB, and the HIGHOFFLOAD is set to 80 and LOWOFFLOAD is set to 60, you should have 3MB / (80%-60%), or 15MB of space in the staging data set for this log stream. We will come back to the values to use for HIGHOFFLOAD and LOWOFFLOAD in 8.4.2, "High and low offload thresholds for active log streams" on page 274.

[1]To be completely accurate, we should take the CF element size into account when calculating the interim storage requirement. However the element size is sufficiently small that it will not have a significant impact.



 < Day Day Up > 



Systems Programmer's Guide to--Z. OS System Logger
ASP.NET for Web Designers
ISBN: 738489433
EAN: 2147483647
Year: 2002
Pages: 99
Authors: Peter Ladka

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net