3.4 Recovery

 < Day Day Up > 



3.4 Recovery

In this section, we discuss how Transactional VSAM reacts to failures that might happen in the logger environment.

3.4.1 Missing data in the log streams

Transactional VSAM reacts to missing data in a different log stream based on the type of log stream that is affected.

First of all, Transactional VSAM only recognizes that there are missing data on the IGWLOG and IGWSHUNT log stream because these are the only log streams where TVS retrieves data from. If any data is lost on the log-of-logs or in any of the forward-recovery log, TVS will not detect it. It is CICSVR or similar that sees the missing data condition asynchronously from the time that the problem occurred.

The impact of having missing data on either IGWLOG or IGWSHUNT to a Transactional VSAM subsystem, is that it will not be able to retrieve data to perform any backout requests. When Transactional VSAM recognizes that there missing data, it comes down and requests a cold start. For information about how to cold start Transactional VSAM, refer to DFSMStvs Planning and Operating Guide, SC26-7348-00. The following is a syslog sample of the scenario:

Example 3-15: Missing data in IGWLOG log stream

start example
 IGW839I 07282003 14.03.32 AN ERROR OCCURRED DURING SYSTEM LOGGER OPERATION IXGBRWSE READBLOCK FOR SYSTEM LOG STREAM IGWTV002.IGWLOG.SYSLOG. SYSTEM LOGGER RETURN CODE 00000008 REASON CODE 00000804 IGW822I 07282003 14.03.32 THE SYSTEM LOGGER FAILED TO LOCATE A BLOCKID REQUESTED BY TRANSACTIONAL VSAM. MISSING BLOCKID 0000000133C3B4C8 CHAIN HISTORY POINT 0000000133BF5F00 IGW833I A FAILURE OCCURRED WHILE READING FROM SYSTEM LOG IGWTV002.IGWLOG.SYSLOG . THE REQUESTED DATA COULD NOT BE FOUND. TRANSACTIONAL VSAM WILL BE QUIESCED SO INFLIGHT TASKS CAN COMPLETE. NEXT TRANSACTIONAL VSAM START MUST BE COLD IGW835I A FAILURE TO READ DATA FROM SYSTEM LOG IGWTV002.IGWLOG.SYSLOG DURING BACKOUT HAS CAUSED JOB **AKP** STEP **AKP** UNIT OF RECOVERY B9C98A7B7E7491440000000701020000 TO BE SHUNTED IGW473I DFSMS VSAM RLS REQUEST TO QUIESCE TRANSACTIONAL VSAM LOGSTREAM IGWTV002.IGWLOG.SYSLOG IS ACCEPTED QUIESCE REASON: TRANSACTIONAL VSAM LOGGER DETECTED AN I/O ERROR *** TRANSACTIONAL VSAM COLD START IS REQUIRED *** IGW834I TRANSACTIONAL VSAM RECEIVED A LOST DATA WARNING FOR SYSTEM LOG IGWTV002.IGWLOG.SYSLOG. TRANSACTIONAL VSAM WILL BE QUIESCED SO INFLIGHT TASKS CAN COMPLETE. NEXT TRANSACTIONAL VSAM START MUST BE COLD IGW10114I TVS003C G003 B9C98A7B7E7491440000000701020000 4 FAILURE OCCURRED DURING BACKOUT PROCESSING WHILE TRYING TO START A BROWSE OF THE UNDO LOG. LOGGER RETURN CODE (00000008) LOGGER REASON CODE (7003000A) ATR306I RESOURCE MANAGER IGWTV002020031960915105983550000 CAUSED A HEURISTIC-MIXED CONDITION FOR URID = B9C98A7B7E7491440000000701020000. 
end example

The impact of having missing data in the log-of-log will not be so dramatic. It only affects performance during the CICSVR operations. DFSMStvs continues to function without this log stream.

For a forward-recovery log stream, a missing data condition prevents the possibility to reconstruct the file by applying these change to the image copy. DFSMStvs closes the data set. To recover the condition, you need to close all the data sets associated with that log stream, delete the log stream, take a data sets backup, define the log stream and e-open the data sets.

3.4.2 Log stream full condition

There are two types of full conditions: one is temporary and it is when the interim storage gets filled up and the second one is permanent when the complete log stream (interim plus offloads) is full.

In the first case, you will see message IGW838I to inform you of the delay but this situation should also resolve as soon as data gets offloaded to DASD. If you see this message very often, you should re-evaluate your logger configuration and ensure it is tuned properly.

The following is a sample of the syslog message for this scenario:

Example 3-16: Interim storage full condition

start example
 
end example

 IGW838I 07282003 15.32.14 A TEMPORARY ERROR CONDITION OCCURRED DURING SYSTEM LOGGER OPERATION IXGWRITE FOR LOG STREAM IGWTV002.IGWLOG.SYSLOG.SYSTEM LOGGER RETURN CODE 00000008 REASON CODE 00000860 

The permanent condition, where the log stream is completely full, can happen in the following situations:

  • When there is no more space on the DASD pool to allocate space for the next offload data set

  • When the DSEXTENT has been saturated

If this situation happens against either the IGWLOG or IGWSHUNT log stream, Transactional VSAM cannot continue and it will come down. The following are the syslog messages generated in this scenario. While Transactional VSAM is down, system logger will keep an internal connection to the log stream to preserve the copy of the data that cannot be offloaded. You can resolve the problem either by adding more space to the offload data set DASD pool or by switching to new LOGR data set formatted with a bigger DSEXTENT value. Once you have resolved the problem, you need to restart Transactional VSAM. At restart time, Transactional VSAM reconnects to the log stream and the offload process resumes.

Example 3-17: Log stream full condition

start example
 IXG251I IKJ56245I DATA SET IXGLOGR.IGWTV002.IGWLOG.SYSLOG.A0000582 NOT ALLOCATED, NOT ENOUGH SPACE ON VOLUMES+ IXG251I IKJ56245I USE DELETE COMMAND TO DELETE UNUSED DATA SETS IXG301I SYSTEM LOGGER FAILED TO OFFLOAD DATA FOR LOG STREAM IGWTV002.IGWLOG.SYSLOG IN STRUCTURE LOG_IGWLOG_001.  RETURN CODE: 00000008  REASON CODE: 00000805  DIAG1: 00000004  DIAG2: 4714041D DIAG3: 0107001B  DIAG4: 00000000 IGW839I 07282003 15.56.37 AN ERROR OCCURRED DURING SYSTEM LOGGER OPERATION IXGWRITE FOR SYSTEM LOG STREAM IGWTV002.IGWLOG.SYSLOG. SYSTEM LOGGER RETURN CODE 00000008 REASON CODE 0000085D IGW827I 07282003 15.56.37 A FAILURE HAS OCCURRED WHILE WRITING TO THE SYSTEM LOG IGWTV002.IGWLOG.SYSLOG . ACCESS TO THE SYSTEM LOG HAS BEEN LOST. TRANSACTIONAL VSAM WILL BE TERMINATED. IGW471I DFSMS VSAM RLS  REQUEST TO DISABLE TRANSACTIONAL VSAM  INSTANCE  IGWTV002 IS ACCEPTED. DISABLE REASON: TRANACTIONAL VSAM DETECTED SYSTEM LOGGER ENVIRONMENTAL ERROR IGW10111I TVSD17C GD17 B9C9A3C77E7480000000D1FB01020000 FAILURE OCCURRED DURING PREPARE PROCESSING WHILE TRYING TO WRITE A COMMIT RECORD TO THE UNDO LOG. LOGGER RETURN CODE (00000008) LOGGER REASON CODE (7002001F) THIS UR WILL BE SHUNTED ATR306I RESOURCE MANAGER IGWTV002020031960915105983550000 CAUSED A HEURISTIC-MIXED CONDITION FOR URID = B9C9A3C77E7480000000D1FB01020000. 
end example

If this situation happens against a forward-recovery log stream, Transactional VSAM closes the data set(s) associated with the log stream since it cannot process the log data.

If this happens against the log-of-logs log stream, Transactional VSAM disables the use of the log stream but it continues to function.

3.4.3 CF failures

In case of a coupling facility failure or a connectivity to a coupling facility failure, system logger detects the loss of connectivity to the single instance of the structure. Then the system that recognized the failure condition initiates a rebuild for the structure, according to the PREFLIST specified in the structure definition for the CFRM policy.

While the recovery is taking place, Transactional VSAM will not be able to access the log streams being rebuilt and receives return code 861 temporary unavailable from system logger. This situation should resolve as soon as the rebuild completes and the log stream is available in the new structure. This temporary error condition will be handled by Transactional VSAM and not propagated to the applications. Following is a sample of the log messages expected during this scenario:

Example 3-18: Log stream rebuild log messages

start example
 IGW838I 07282003 16.48.22 A TEMPORARY ERROR CONDITION OCCURRED DURING SYSTEM LOGGER OPERATION IXGWRITE FOR LOG STREAM IGWTV002.IGWLOG.SYSLOG . SYSTEM LOGGER RETURN CODE 00000008 REASON CODE 00000861 
end example

3.4.4 System Logger failure

As a result of the system logger address space going down, Transactional VSAM reacts in the following ways:

  • Since it is not be able to access the IGWLOG and IGWSHUNT, the instance of Transactional VSAM running on the system where system logger went down will disconnect from the log streams and then terminate

  • It will disconnect from the other log streams, forward recovery and log-of-logs, but these log streams are still available on the other systems in the sysplex.

When the system logger address space, Transactional VSAM automatically restarts and reconnects to the log streams. Following is a syslog of the scenario:

Example 3-19: System Logger failure messages

start example
 IGW473I DFSMS VSAM RLS REQUEST TO MAKE TRANSACTIONAL VSAM LOGSTREAM #@$C.CICSVR.LGOFLOGS UNAVAILABLE FOR USE ON THIS SYSTEM IS ACCEPTED. TRANSACTIONAL VSAM LOGSTREAM #@$C.CICSVR.LGOFLOGS IS MARKED UNAVAILABLE ON SYSTEM:  #@$2 UNAVAILABLE REASON: TRANSACTIONAL VSAM DETECTED A LOCAL LOG ERROR IGW855I TRANSACTIONAL VSAM FAILED TO WRITE TO THE LOG OF LOGS #@$C.CICSVR.LGOFLOGS , WHILE PROCESSING DSN TVSV.VF01D.TRANLOG , FORWARD RECOVERY LOG STREAM #@$C.CICSVR.LOG001. RETURN CODE 00000008 REASON CODE 70060016 IGW839I 07282003 16.38.02 AN ERROR OCCURRED DURING SYSTEM LOGGER OPERATION IXGWRITE FOR SYSTEM LOG STREAM #@$C.CICSVR.LOG001. SYSTEM LOGGER RETURN CODE 00000008 REASON CODE 00000890 IGW839I 07282003 16.38.02 AN ERROR OCCURRED DURING SYSTEM LOGGER 4 OPERATION IXGWRITE FOR SYSTEM LOG STREAM IGWTV002.IGWLOG.SYSLOG. SYSTEM LOGGER RETURN CODE 00000008 REASON CODE 00000890 IGW827I 07282003 16.38.03 A FAILURE HAS OCCURRED WHILE WRITING TO THE SYSTEM LOG IGWTV002.IGWLOG.SYSLOG . ACCESS TO THE SYSTEM LOG HAS BEEN LOST. TRANSACTIONAL VSAM WILL BE TERMINATED. IEF402I IXGLOGR FAILED IN ADDRESS SPACE 001F         SYSTEM ABEND S1C5 - REASON CODE 020004 IXG058E LOGSTREAM CONNECTIONS HAVE BEEN LOST DUE TO SYSTEM LOGGER TERMINATION IXG056I SYSTEM LOGGER ADDRESS SPACE HAS ENDED. 
end example

Example 3-20: Transactional VSAM status after failure

start example
 D SMS,TRANVSAM  IGW800I 16.42.45 DISPLAY SMS,TRANSACTIONAL VSAM DISPLAY SMS,TRANSACTIONAL VSAM - SERVER STATUS   System   TVSNAME  State   Rrs    #Urs     Start     AKP    QtimeOut   -------- -------- ------ ----- -------- --------- -------- --------   #@$2     IGWTV002 DISED  UNREG        1 COLD/WARM     1000      300 DISPLAY SMS,TRANSACTIONAL VSAM - LOGSTREAM STATUS   LogStreamName              State      Type       Connect Status   -------------------------- ---------- ---------- --------------   IGWTV002.IGWLOG.SYSLOG     Disabled   UnDoLog    DisConnected   IGWTV002.IGWSHUNT.SHUNTLOG Disabled   ShuntLog   DisConnected   #@$C.CICSVR.LGOFLOGS       Ena-Unaval LogOfLogs  DisConnected   #@$C.CICSVR.LOG001         Ena-Unaval FrLog      DisConnected 
end example

Example 3-21: System logger restart

start example
 S IXGLOGRS IGW474I DFSMS VSAM RLS  REQUEST TO ENABLE TRANSACTIONAL VSAM  LOGSTREAM IGWTV002.IGWLOG.SYSLOG IS ACCEPTED IGW471I DFSMS VSAM RLS COMMAND PROCESSOR 608  ON SYSTEM:  #@$2  IS WAITING FOR A RESPONSE   FROM TRANSACTIONAL VSAM:  IGWTV002  COMMAND REQUESTED:   ENABLE  TRANSACTIONAL VSAM:  IGWTV002  IGW473I DFSMS VSAM RLS COMMAND PROCESSOR  ON SYSTEM:  #@$2  IS WAITING FOR A RESPONSE FROM TRANSACTIONAL VSAM:  IGWTV002  COMMAND REQUESTED:  ENABLE  LOGSTREAM:  IGWTV002.IGWLOG.SYSLOG  IGW473I DFSMS VSAM RLS COMMAND PROCESSOR  ON SYSTEM:  #@$2  IS WAITING FOR A RESPONSE FROM TRANSACTIONAL VSAM:  IGWTV002  COMMAND REQUESTED:  ENABLE  LOGSTREAM:  IGWTV002.IGWSHUNT.SHUNTLOG  IGW860I TRANSACTIONAL VSAM HAS SUCCESSFULLY REGISTERED WITH RLS  IGW888I TRANSACTIONAL VSAM PERMITNONRLSUPDATE EXIT NOT LOADED FOR INSTANCE IGWTV002 ON SYSTEM #@$2 IGW848I 07282003 16.44.24 SYSTEM UNDO LOG IGWTV002.IGWLOG.SYSLOG INITIALIZATION HAS STARTED IXL014I IXLCONN REQUEST FOR STRUCTURE LOG_IGWLOG_001 629 WAS SUCCESSFUL.  JOBNAME: IXGLOGR ASID: 0033 CONNECTOR NAME: IXGLOGR_#@$2 CFNAME: FACIL06 IGW474I DFSMS VSAM RLS IS CONNECTING TO TRANSACTIONAL VSAM  LOGSTREAM IGWTV002.IGWLOG.SYSLOG SYSTEM NAME:              #@$2 TRANSACTIONAL VSAM INSTANCE NAME:  IGWTV002 IGW848I 07282003 16.44.26 SYSTEM UNDO LOG IGWTV002.IGWLOG.SYSLOG INITIALIZATION HAS ENDED 
end example

3.4.5 Subsystem failure

When the Transactional VSAM instance fails, all the new request (put/get/open) against the VSAM files receives a return code documenting that Transactional VSAM is not available. The application code needs to handle these return codes. For a complete list of these return codes, refer to DFSMStvs Administration Guide, GC26-7483-00.

For the in-flight requests, retained locks are generated against the records that were held by that unit of works. Refer to DFSMStvs Planning and Operating Guide, SC26-7348-00 to understand how to resolve a situation with retain locks.



 < Day Day Up > 



Systems Programmer's Guide to--Z. OS System Logger
ASP.NET for Web Designers
ISBN: 738489433
EAN: 2147483647
Year: 2002
Pages: 99
Authors: Peter Ladka

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net