2.7 Deleting log data

 < Day Day Up > 



2.7 Deleting log data

One of the most important aspects of running System Logger is understanding how and when log data is deleted. Since log data and offload data set deletion is tied to offload processing, make sure you have first read "Offload processing" on page 57.

2.7.1 When is log data marked for deletion?

Log data is marked for deletion (but not yet physically deleted) based on the use of the IXGDELET service. This information is then merged with the values used for RETPD and AUTODELETE (see 2.5, "Log streams" on page 22 for parameter usage) to determine when it will be physically deleted:

  • Using RETPD(0) and AUTODELETE(NO): In this case, the only one that can delete data from the log stream is the connector; that is, System Logger will never delete the data itself. Applications should mark log blocks for deletion using the IXGDELET service as soon as they are no longer needed.

  • Using RETPD(>0) and AUTODELETE(NO): In this case, data must be marked for deletion using the IXGDELTE service and the retention period must have expired before log data will be deleted. For example, if you specify RETPD(7) in the System Logger policy for a log stream, log blocks will not be deleted for at least seven days, even if an IXGDELET is issued immediately after the log block is written to the log stream. Equally, if seven days pass and an IXGDELET still has not been issued, the data will not be deleted, and will remain in the log stream until an IXGDELET is eventually issued.

    Note that for data that has been offloaded, System Logger processes the retention period on an offload data set basis. Only after the retention period for all the data in the offload data set has expired, is the data set eligible for deletion. Until that time, the data is still accessible, even though its retention period might have expired.

  • Using RETPD(>0) and AUTODELETE(YES): AUTODELETE(YES) means that log data can be physically deleted either when the data is marked for deletion by IXGDELET, or when the retention period specified for the log stream expires.

    Use care when specifying AUTODELETE(YES). Automatic deletion is designed to speed physical deletion of log data, which can mean deletion of data that an application still needs. Generally speaking, AUTODELETE(YES) should never be used with an application that issues IXGDELETs to delete log records when it no longer requires them.

    Note that the default retention period is 0, so the log blocks for log streams specifying AUTODELETE(YES) and RETPD(0), or not specifying any RETPD, will be offloaded to offload data sets and then be eligible for deletion immediately.

Some example retention period and autodelete policies can be found in Example 2-10.

Example 2-10: AUTODELETE and RETPD sample policies

start example
 Example 1. Delete Data After No More Than 3 Days, No Matter What: For audit log streams, such as the OPERLOG log stream, the installation must often manage how much data is kept in the log stream. Using automatic deletion and a retention period, you can manage the amount of time log data resides in the log stream, cutting down also on the storage consumed by log data in this log stream. For example, let's say that log AUDIT1 requires that data be kept for no more than 3 days and then physically deleted. For AUDIT1, you would specify RETPD(3) and AUTODELETE(YES) to make sure that data is kept for no more than 3 days. Example 2. Keep Data for At Least 30 Days, Even If It Was Marked for Deletion: Installations may require that their transaction manager log streams retain data for a certain length of time to meet audit requirements. This data must be kept for the retention period, even when it has been marked for deletion via the IXGDELET service. Using automatic deletion and a retention period, you can archive the data for the required length of time without having to move the data out of the log stream. For example, log TRANS1 needs to have data in the log stream for at least 30 days, even if it has been marked for deletion. You can specify RETPD(30) AUTODELETE(NO) to keep the data for the required amount of time. Even if data is marked for deletion via IXGDELET within the 30 day retention period, it is kept in the log stream and can be accessed by a System Logger application using the IXGBRWSE service with VIEW=ALL. Example 3. Keep Data in the Log Stream Until Deletion: Some log streams may not need to keep log data around after it has been marked for deletion via IXGDELET and do not require that data be retained for a particular length of time. For this kind of a log stream, the default settings of RETPD(0) and AUTODELETE(NO) should be used. System Logger will consider data eligible for physical deletion any time after it has been marked for deletion via IXGDELET. 
end example

Note 

This retention period is different from a data set retention period on a JCL DD statement; a System Logger retention period applies to the age of the log data, not the data set.

2.7.2 When is my log data or offload data set physically deleted?

When log data is marked for deletion by IXGDELET, System Logger does not physically delete the log data or offload data set until an offload process is initiated (usually as a result of the high threshold being reached). This means that there will often be a delay before eligible offload data sets are physically deleted, since you have to wait for the next offload to start. If offloads are relatively infrequent, there may be a considerable delay before offload data sets that are eligible for deletion are actually deleted.

The point at which offload data sets are physically deleted depends on whether you have specified a RETPD >0 for this log stream. If the log stream is defined with RETPD=0, expired offload data sets [2] will be deleted the next time an offload process is initiated, even if no data is actually moved to an offload data set.

However, if the log stream is defined with RETPD>0, expired offload data sets will not be deleted until the next time System Logger allocates a new offload data set. If the offload data sets are so large that it takes many offloads before the data set is filled (at which point a new offload data set will be allocated), you could potentially have expired data sets hanging around a long time waiting to be physically deleted. And as long as the data sets have not been deleted, they will be holding entries in the directory extent.

Allocation errors can also delay offload data set deletion. If System Logger cannot delete an offload data set because it is currently allocated, it will attempt to delete the offload data set on a subsequent offload when delete processing is again performed. Successive allocation errors may cause a data set to be "orphaned"; an offload data set that System Logger attempted to delete but was unable to. When an offload data set is orphaned, System Logger issues message IXG252I. System Logger frees up the offload data set's entry in the directory extent, meaning that System Logger no longer has any knowledge of that data set—such data sets must be deleted manually. To get a list of orphaned data sets, run an IXCMIAPU report. Because System Logger has removed the entries for the data sets from its inventory, the report job scans the catalog for data sets with names that match the offload data set for that log stream—any such data sets that are found are reported as orphaned data sets. For more information, see "Orphaned Data sets" on page 247.

An example allocation problem where System Logger would be unable to delete the offload data set is demonstrated by Figure 2-13 on page 65. Assume system A initially allocated and offloaded to the first offload data set (LSN.A0000001); System Logger on system A keeps that data set allocated until it has to offload for that particular log stream again (at which point it would allocate the current offload data set). Note that the reason for System Logger requiring SHAREOPTIONS (3 3) is because it uses a SHR ENQ on the offload data sets: so, just because one System Logger has the offload data set allocated does not mean that another System Logger can't write to that data set.

click to expand
Figure 2-13: Offload data set LSN.A0000001 cannot be deleted

In the meantime, system B maybe have caused several data set switches to occur by offloading so that the current offload data set is now LSN.A0000004. Suppose now that all of the log blocks in data set LSN.A0000001 have been marked for deletion, and system B begins to offload. System Logger will attempt to delete LSN.A0000001, but will fail because system A still has it allocated. System Logger will still go forward and delete other unallocated data sets (LSN.A0000002, and so on) as they become eligible for deletion during offloads.

Important: 

Except for cleaning up orphaned offload data sets, you should never delete offload data sets manually. When System Logger is finished with a data set, it will delete it. Deleting an offload data set by any other means could result in lost data. It will also cause problems for System Logger because it will not be able to perform any clean-up and the deleted data sets will still count toward the data set directory limit.

[2]An "expired" offload data set is one in which all the log blocks have reached their RETPD, and either an IXGDELET has been issued or AUTODELETE(YES) was specified for the log stream.



 < Day Day Up > 



Systems Programmer's Guide to--Z. OS System Logger
ASP.NET for Web Designers
ISBN: 738489433
EAN: 2147483647
Year: 2002
Pages: 99
Authors: Peter Ladka

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net