2.6 Offload processing

 < Day Day Up > 



2.6 Offload processing

As applications write data to a log stream, the interim storage defined for log data begins to fill, eventually reaching or exceeding its high threshold. You specify (in the System Logger policy) high and low thresholds for each log stream to define when System Logger is to begin and end the process of moving, or offloading, log data to offload data sets. It is important for you to understand the way System Logger offloading works so that:

  • You can set appropriate high and low thresholds to control offloading.

  • You understand when, why, and how often offloading will occur.

  • You understand when log data is physically deleted from the log stream.

  • You can plan CF structure, offload data set, and staging data set sizes to control how often offloading occurs.

Offload processing works differently for CF and DASD-only log streams:

  • For a CF-Structure based log stream, System Logger offloads data from the CF to offload data sets. Offloading is initiated when either of the following occurs:

    • The number of elements in use for the log stream reaches the high threshold.

    • The staging data set for the log stream reaches the high threshold.

    • The last user disconnects from the log stream.

    For CF-Structure based log streams, space in interim storage is freed up for reuse incrementally while the offload runs. That is, you do not have to wait until the offload completes before the freed-up space can be used for new IXGWRITE requests.

  • For a DASD-only log stream, System Logger offloads data from local storage buffers to offload data sets. Offloading is initiated when the staging data set reaches the high threshold. Therefore, for a DASD-only log stream, offload processing moves log data from local storage buffers to offload data sets, but the offloading is actually triggered by staging data set usage.

    For DASD-only log streams, the space occupied by the log data being offloaded is not freed up until the offload has completed.

There are a number of situations that cause the offload process to be invoked. The situations and the actions System Logger takes are listed in Table 2-5 on page 58.

Table 2-5: When offload process is invoked

Situation

DASD-only log stream

CF-Structure log stream

Number of elements used OR number of CIs in staging data set used reaches HIGHOFFLOAD

  1. Delete all log blocks that have been logically deleted by IXGDELET

  2. If LOWOFFLOAD has not been reached, move oldest log blocks to offload data set until LOWOFFLOAD is reached

  1. Delete all eligible log blocks that have been logically deleted by IXGDELET

  2. If LOWOFFLOAD has not been reached, move oldest log blocks to offload data set until LOWOFFLOAD is reached

Number of entries used in CF structure reaches 90%

Not applicable

  1. Delete all log blocks that have been logically deleted by IXGDELET in all log streams in this structure

  2. For every log stream in the structure, if LOWOFFLOAD has not been reached, move oldest log blocks to offload data set until LOWOFFLOAD is reached

Structure full

Not applicable

  1. Delete all log blocks that have been logically deleted by IXGDELET in all log streams in this structure

  2. If LOWOFFLOAD has not been reached, move oldest log blocks to offload data set until LOWOFFLOAD is reached

Last connector from a system to a log stream disconnects

All log blocks moved to offload data sets

All log blocks for that log stream moved to offload data sets, up to the newest log block written by the system that just disconnected

Structure rebuild completes and recovery is required

Not applicable

All log blocks for all log streams in this structure moved to offload data sets

Recovery restart

First connection

First connection

For either type of log stream, when offload processing is initiated because the high offload threshold is reached, System Logger begins the process as follows:

  1. Delete any eligible log blocks in interim storage that have been marked logically deleted (using IXGDELET). For a CF-Structure log stream, free the associated interim storage in blocks as the offload proceeds.

  2. If the low threshold has not been reached, calculate how much data must be offloaded to reach the low offload threshold.

  3. Allocate the offload data set (if it isn't already allocated).

  4. Offload the data to the offload data sets, moving the oldest log blocks first. Note that all valid log blocks are moved until the threshold is reached, regardless of which system created them. So, if the offload was initiated because one system disconnected from the log stream, all log blocks, from all systems, will be moved to the offload data set. Once again, for a CF-Structure log stream, interim storage is freed up as the offload proceeds.

  5. For DASD-only log streams, free all interim storage associated with the offloaded log blocks.

Figure 2-11 on page 60 shows an overview of a two-way sysplex where applications are using CF-Structure based log streams. After some time, the log data is offloaded from the CF structure to offload data sets.

click to expand
Figure 2-11: Offload overview (example is a CF-Structure based log stream)

Figure 2-12 on page 60 is a graphical example of the CF structure state before and after the offload, assuming that the high offload threshold is set to 80% and the low offload threshold is set to 20%. Note that new writes can arrive during the offload process, so it is likely that when offload is complete, the storage percentage used will be greater than the defined LOWOFFLOAD value.

click to expand
Figure 2-12: Offload from HIGHOFFLOAD to LOWOFFLOAD

As we discussed in previous chapters, the HIGHOFFLOAD and LOWOFFLOAD values allow you to control the offload frequency and the amount of data offloaded. For your System Logger application log streams, you should check the related application chapter in this book for suggested values. However, in general for funnel-type log streams, the default values of 80% HIGHOFFLOAD and 0% LOWOFFLOAD are a good setting. They have the advantage of moving larger chunks of log data to DASD at one time, reducing the number of times that offload processing is invoked for the log stream. Active-type log streams on the other hand might find a HIGHOFFLOAD threshold of 80% and a higher low offload threshold (60%, for example) more desirable as more data will be kept in the CF structure or in local buffers so that it is quickly accessible and you don't waste resources moving data that is about to be deleted to DASD.

You can use SMF reports to tune your offload threshold settings; for more information on this, see "SMF Type 88 records and IXGRPT1 program" on page 281. In addition, APAR OW13572 added the ability to create a Component Trace record every time an offload process is started—this information can help determine the frequency of offloads for each log stream.

Important: 

We recommend that you do not define your HIGHOFFLOAD value to greater than the default of 80%. If you define a higher HIGHOFFLOAD value, you will be more vulnerable to filling your CF structure or staging data set space for the log stream during sudden bursts of system activity that push the log data above the high threshold. Once CF structure or staging data set space for a log stream is 100% filled, System Logger rejects all write requests for that log stream until the log data can be offloaded to offload data sets.

2.6.1 CF-Structure size and offloading

For a CF-Structure based log stream, you can influence the frequency of offloads and the processing overhead offloading entails by appropriate sizing of your CF structures. The larger the CF structure, the less often offload processing will occur, and the larger the amount of data that will be offloaded. This minimizes offloading overhead and keeps more log data available in the CF structure for high performance reads of log data.

For example, if you have a log stream with a high volume of data being written to it, and you have allocated a small CF structure for this log stream, the structure will fill quickly and hit its high threshold, initiating frequent offloading of log data to DASD. This increases system usage, but does not interfere with processing of write requests to the log stream, because write requests can be processed at the same time as offload processing.

However, the CF structure for a log stream should not be larger than the application really requires and should reflect the needs of the rest of the installation for CF space. See "Setting up for and defining CF structures" on page 24 for information about optimal sizing of CF structures for response, throughput, and availability.

If the CF space allocated for a log stream reaches 100% utilization, all write requests against that log stream are rejected until offloading can complete.

2.6.2 Staging data set size and offloading

For CF-Structure based log streams: For a CF-Structure based log stream, the high offload threshold defined by an installation for the structure space allocated to a log stream applies also to the staging data sets for that log stream. For example, if you define a high threshold of 80% for a log stream, System Logger will begin offload processing when either the CF structure allocated for a log stream or a staging data set for the log stream reaches or exceeds 80% usage.

This means that when a staging data set reaches the high threshold, System Logger immediately offloads data from the CF structure to offload data sets, even if the structure usage for the log stream is below the high threshold.

Therefore, if your staging data sets are small in comparison to the CF structure size for a log stream, the staging data sets will keep filling up and System Logger will offload CF log data to DASD frequently. This means that your installation experiences frequent offloading overhead that could affect performance. It also means that you have allocated space in the CF that you will never be able to use (because the data keeps being offloaded before the structure gets anywhere near full).

If your staging data sets are too small, you also run the risk of them filling up completely. If this occurs, System Logger immediately begins offloading the CF log data to offload data sets to harden it. System Logger applications will be unable to log data until System Logger can free up staging data set space.

In general, the sizing guideline for staging data sets is to make them large enough to hold all the log data in the CF structure, allowing for the fact that if there are n log streams in the structure, each log stream will only be allocated 1/n of the space in the structure. See 8.2.2, "Sizing interim storage" on page 265 for complete information on sizing staging data sets.

For DASD-only log streams: For a DASD-only log stream, offloading of log data to offload data sets is always triggered by staging data set usage. System Logger offloads data from local buffers to offload data sets when it hits the high threshold defined for the staging data sets. For example, if you define a high threshold of 80% for a DASD-only log stream, System Logger will begin offload processing when the staging data set space defined for a log stream reaches or exceeds 80% usage.

This means that if you size your staging data sets too small, System Logger will offload log data from local buffers to offload data sets frequently, incurring additional overhead. You might also run the risk of filling the staging data sets up completely. If this occurs, System Logger will immediately begin offloading log data to offload data sets.

Attention: 

APAR OW51854 added the ability for System Logger to monitor offloads and provide a mechanism to interrupt hung offloads. This function is explained further in "Offload monitoring" on page 251.

Additional consideration on the offload process can also be found in "The entry-to-element ratio" on page 26.



 < Day Day Up > 



Systems Programmer's Guide to--Z. OS System Logger
ASP.NET for Web Designers
ISBN: 738489433
EAN: 2147483647
Year: 2002
Pages: 99
Authors: Peter Ladka

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net