| < Day Day Up > |
|
In this section, we discuss how Transactional VSAM reacts to failures that might happen in the logger environment.
Transactional VSAM reacts to missing data in a different log stream based on the type of log stream that is affected.
First of all, Transactional VSAM only recognizes that there are missing data on the IGWLOG and IGWSHUNT log stream because these are the only log streams where TVS retrieves data from. If any data is lost on the log-of-logs or in any of the forward-recovery log, TVS will not detect it. It is CICSVR or similar that sees the missing data condition asynchronously from the time that the problem occurred.
The impact of having missing data on either IGWLOG or IGWSHUNT to a Transactional VSAM subsystem, is that it will not be able to retrieve data to perform any backout requests. When Transactional VSAM recognizes that there missing data, it comes down and requests a cold start. For information about how to cold start Transactional VSAM, refer to DFSMStvs Planning and Operating Guide, SC26-7348-00. The following is a syslog sample of the scenario:
Example 3-15: Missing data in IGWLOG log stream
IGW839I 07282003 14.03.32 AN ERROR OCCURRED DURING SYSTEM LOGGER OPERATION IXGBRWSE READBLOCK FOR SYSTEM LOG STREAM IGWTV002.IGWLOG.SYSLOG. SYSTEM LOGGER RETURN CODE 00000008 REASON CODE 00000804 IGW822I 07282003 14.03.32 THE SYSTEM LOGGER FAILED TO LOCATE A BLOCKID REQUESTED BY TRANSACTIONAL VSAM. MISSING BLOCKID 0000000133C3B4C8 CHAIN HISTORY POINT 0000000133BF5F00 IGW833I A FAILURE OCCURRED WHILE READING FROM SYSTEM LOG IGWTV002.IGWLOG.SYSLOG . THE REQUESTED DATA COULD NOT BE FOUND. TRANSACTIONAL VSAM WILL BE QUIESCED SO INFLIGHT TASKS CAN COMPLETE. NEXT TRANSACTIONAL VSAM START MUST BE COLD IGW835I A FAILURE TO READ DATA FROM SYSTEM LOG IGWTV002.IGWLOG.SYSLOG DURING BACKOUT HAS CAUSED JOB **AKP** STEP **AKP** UNIT OF RECOVERY B9C98A7B7E7491440000000701020000 TO BE SHUNTED IGW473I DFSMS VSAM RLS REQUEST TO QUIESCE TRANSACTIONAL VSAM LOGSTREAM IGWTV002.IGWLOG.SYSLOG IS ACCEPTED QUIESCE REASON: TRANSACTIONAL VSAM LOGGER DETECTED AN I/O ERROR *** TRANSACTIONAL VSAM COLD START IS REQUIRED *** IGW834I TRANSACTIONAL VSAM RECEIVED A LOST DATA WARNING FOR SYSTEM LOG IGWTV002.IGWLOG.SYSLOG. TRANSACTIONAL VSAM WILL BE QUIESCED SO INFLIGHT TASKS CAN COMPLETE. NEXT TRANSACTIONAL VSAM START MUST BE COLD IGW10114I TVS003C G003 B9C98A7B7E7491440000000701020000 4 FAILURE OCCURRED DURING BACKOUT PROCESSING WHILE TRYING TO START A BROWSE OF THE UNDO LOG. LOGGER RETURN CODE (00000008) LOGGER REASON CODE (7003000A) ATR306I RESOURCE MANAGER IGWTV002020031960915105983550000 CAUSED A HEURISTIC-MIXED CONDITION FOR URID = B9C98A7B7E7491440000000701020000.
The impact of having missing data in the log-of-log will not be so dramatic. It only affects performance during the CICSVR operations. DFSMStvs continues to function without this log stream.
For a forward-recovery log stream, a missing data condition prevents the possibility to reconstruct the file by applying these change to the image copy. DFSMStvs closes the data set. To recover the condition, you need to close all the data sets associated with that log stream, delete the log stream, take a data sets backup, define the log stream and e-open the data sets.
There are two types of full conditions: one is temporary and it is when the interim storage gets filled up and the second one is permanent when the complete log stream (interim plus offloads) is full.
In the first case, you will see message IGW838I to inform you of the delay but this situation should also resolve as soon as data gets offloaded to DASD. If you see this message very often, you should re-evaluate your logger configuration and ensure it is tuned properly.
The following is a sample of the syslog message for this scenario:
Example 3-16: Interim storage full condition
IGW838I 07282003 15.32.14 A TEMPORARY ERROR CONDITION OCCURRED DURING SYSTEM LOGGER OPERATION IXGWRITE FOR LOG STREAM IGWTV002.IGWLOG.SYSLOG.SYSTEM LOGGER RETURN CODE 00000008 REASON CODE 00000860
The permanent condition, where the log stream is completely full, can happen in the following situations:
When there is no more space on the DASD pool to allocate space for the next offload data set
When the DSEXTENT has been saturated
If this situation happens against either the IGWLOG or IGWSHUNT log stream, Transactional VSAM cannot continue and it will come down. The following are the syslog messages generated in this scenario. While Transactional VSAM is down, system logger will keep an internal connection to the log stream to preserve the copy of the data that cannot be offloaded. You can resolve the problem either by adding more space to the offload data set DASD pool or by switching to new LOGR data set formatted with a bigger DSEXTENT value. Once you have resolved the problem, you need to restart Transactional VSAM. At restart time, Transactional VSAM reconnects to the log stream and the offload process resumes.
Example 3-17: Log stream full condition
IXG251I IKJ56245I DATA SET IXGLOGR.IGWTV002.IGWLOG.SYSLOG.A0000582 NOT ALLOCATED, NOT ENOUGH SPACE ON VOLUMES+ IXG251I IKJ56245I USE DELETE COMMAND TO DELETE UNUSED DATA SETS IXG301I SYSTEM LOGGER FAILED TO OFFLOAD DATA FOR LOG STREAM IGWTV002.IGWLOG.SYSLOG IN STRUCTURE LOG_IGWLOG_001. RETURN CODE: 00000008 REASON CODE: 00000805 DIAG1: 00000004 DIAG2: 4714041D DIAG3: 0107001B DIAG4: 00000000 IGW839I 07282003 15.56.37 AN ERROR OCCURRED DURING SYSTEM LOGGER OPERATION IXGWRITE FOR SYSTEM LOG STREAM IGWTV002.IGWLOG.SYSLOG. SYSTEM LOGGER RETURN CODE 00000008 REASON CODE 0000085D IGW827I 07282003 15.56.37 A FAILURE HAS OCCURRED WHILE WRITING TO THE SYSTEM LOG IGWTV002.IGWLOG.SYSLOG . ACCESS TO THE SYSTEM LOG HAS BEEN LOST. TRANSACTIONAL VSAM WILL BE TERMINATED. IGW471I DFSMS VSAM RLS REQUEST TO DISABLE TRANSACTIONAL VSAM INSTANCE IGWTV002 IS ACCEPTED. DISABLE REASON: TRANACTIONAL VSAM DETECTED SYSTEM LOGGER ENVIRONMENTAL ERROR IGW10111I TVSD17C GD17 B9C9A3C77E7480000000D1FB01020000 FAILURE OCCURRED DURING PREPARE PROCESSING WHILE TRYING TO WRITE A COMMIT RECORD TO THE UNDO LOG. LOGGER RETURN CODE (00000008) LOGGER REASON CODE (7002001F) THIS UR WILL BE SHUNTED ATR306I RESOURCE MANAGER IGWTV002020031960915105983550000 CAUSED A HEURISTIC-MIXED CONDITION FOR URID = B9C9A3C77E7480000000D1FB01020000.
If this situation happens against a forward-recovery log stream, Transactional VSAM closes the data set(s) associated with the log stream since it cannot process the log data.
If this happens against the log-of-logs log stream, Transactional VSAM disables the use of the log stream but it continues to function.
In case of a coupling facility failure or a connectivity to a coupling facility failure, system logger detects the loss of connectivity to the single instance of the structure. Then the system that recognized the failure condition initiates a rebuild for the structure, according to the PREFLIST specified in the structure definition for the CFRM policy.
While the recovery is taking place, Transactional VSAM will not be able to access the log streams being rebuilt and receives return code 861 temporary unavailable from system logger. This situation should resolve as soon as the rebuild completes and the log stream is available in the new structure. This temporary error condition will be handled by Transactional VSAM and not propagated to the applications. Following is a sample of the log messages expected during this scenario:
Example 3-18: Log stream rebuild log messages
IGW838I 07282003 16.48.22 A TEMPORARY ERROR CONDITION OCCURRED DURING SYSTEM LOGGER OPERATION IXGWRITE FOR LOG STREAM IGWTV002.IGWLOG.SYSLOG . SYSTEM LOGGER RETURN CODE 00000008 REASON CODE 00000861
As a result of the system logger address space going down, Transactional VSAM reacts in the following ways:
Since it is not be able to access the IGWLOG and IGWSHUNT, the instance of Transactional VSAM running on the system where system logger went down will disconnect from the log streams and then terminate
It will disconnect from the other log streams, forward recovery and log-of-logs, but these log streams are still available on the other systems in the sysplex.
When the system logger address space, Transactional VSAM automatically restarts and reconnects to the log streams. Following is a syslog of the scenario:
Example 3-19: System Logger failure messages
IGW473I DFSMS VSAM RLS REQUEST TO MAKE TRANSACTIONAL VSAM LOGSTREAM #@$C.CICSVR.LGOFLOGS UNAVAILABLE FOR USE ON THIS SYSTEM IS ACCEPTED. TRANSACTIONAL VSAM LOGSTREAM #@$C.CICSVR.LGOFLOGS IS MARKED UNAVAILABLE ON SYSTEM: #@$2 UNAVAILABLE REASON: TRANSACTIONAL VSAM DETECTED A LOCAL LOG ERROR IGW855I TRANSACTIONAL VSAM FAILED TO WRITE TO THE LOG OF LOGS #@$C.CICSVR.LGOFLOGS , WHILE PROCESSING DSN TVSV.VF01D.TRANLOG , FORWARD RECOVERY LOG STREAM #@$C.CICSVR.LOG001. RETURN CODE 00000008 REASON CODE 70060016 IGW839I 07282003 16.38.02 AN ERROR OCCURRED DURING SYSTEM LOGGER OPERATION IXGWRITE FOR SYSTEM LOG STREAM #@$C.CICSVR.LOG001. SYSTEM LOGGER RETURN CODE 00000008 REASON CODE 00000890 IGW839I 07282003 16.38.02 AN ERROR OCCURRED DURING SYSTEM LOGGER 4 OPERATION IXGWRITE FOR SYSTEM LOG STREAM IGWTV002.IGWLOG.SYSLOG. SYSTEM LOGGER RETURN CODE 00000008 REASON CODE 00000890 IGW827I 07282003 16.38.03 A FAILURE HAS OCCURRED WHILE WRITING TO THE SYSTEM LOG IGWTV002.IGWLOG.SYSLOG . ACCESS TO THE SYSTEM LOG HAS BEEN LOST. TRANSACTIONAL VSAM WILL BE TERMINATED. IEF402I IXGLOGR FAILED IN ADDRESS SPACE 001F SYSTEM ABEND S1C5 - REASON CODE 020004 IXG058E LOGSTREAM CONNECTIONS HAVE BEEN LOST DUE TO SYSTEM LOGGER TERMINATION IXG056I SYSTEM LOGGER ADDRESS SPACE HAS ENDED.
Example 3-20: Transactional VSAM status after failure
D SMS,TRANVSAM IGW800I 16.42.45 DISPLAY SMS,TRANSACTIONAL VSAM DISPLAY SMS,TRANSACTIONAL VSAM - SERVER STATUS System TVSNAME State Rrs #Urs Start AKP QtimeOut -------- -------- ------ ----- -------- --------- -------- -------- #@$2 IGWTV002 DISED UNREG 1 COLD/WARM 1000 300 DISPLAY SMS,TRANSACTIONAL VSAM - LOGSTREAM STATUS LogStreamName State Type Connect Status -------------------------- ---------- ---------- -------------- IGWTV002.IGWLOG.SYSLOG Disabled UnDoLog DisConnected IGWTV002.IGWSHUNT.SHUNTLOG Disabled ShuntLog DisConnected #@$C.CICSVR.LGOFLOGS Ena-Unaval LogOfLogs DisConnected #@$C.CICSVR.LOG001 Ena-Unaval FrLog DisConnected
Example 3-21: System logger restart
S IXGLOGRS IGW474I DFSMS VSAM RLS REQUEST TO ENABLE TRANSACTIONAL VSAM LOGSTREAM IGWTV002.IGWLOG.SYSLOG IS ACCEPTED IGW471I DFSMS VSAM RLS COMMAND PROCESSOR 608 ON SYSTEM: #@$2 IS WAITING FOR A RESPONSE FROM TRANSACTIONAL VSAM: IGWTV002 COMMAND REQUESTED: ENABLE TRANSACTIONAL VSAM: IGWTV002 IGW473I DFSMS VSAM RLS COMMAND PROCESSOR ON SYSTEM: #@$2 IS WAITING FOR A RESPONSE FROM TRANSACTIONAL VSAM: IGWTV002 COMMAND REQUESTED: ENABLE LOGSTREAM: IGWTV002.IGWLOG.SYSLOG IGW473I DFSMS VSAM RLS COMMAND PROCESSOR ON SYSTEM: #@$2 IS WAITING FOR A RESPONSE FROM TRANSACTIONAL VSAM: IGWTV002 COMMAND REQUESTED: ENABLE LOGSTREAM: IGWTV002.IGWSHUNT.SHUNTLOG IGW860I TRANSACTIONAL VSAM HAS SUCCESSFULLY REGISTERED WITH RLS IGW888I TRANSACTIONAL VSAM PERMITNONRLSUPDATE EXIT NOT LOADED FOR INSTANCE IGWTV002 ON SYSTEM #@$2 IGW848I 07282003 16.44.24 SYSTEM UNDO LOG IGWTV002.IGWLOG.SYSLOG INITIALIZATION HAS STARTED IXL014I IXLCONN REQUEST FOR STRUCTURE LOG_IGWLOG_001 629 WAS SUCCESSFUL. JOBNAME: IXGLOGR ASID: 0033 CONNECTOR NAME: IXGLOGR_#@$2 CFNAME: FACIL06 IGW474I DFSMS VSAM RLS IS CONNECTING TO TRANSACTIONAL VSAM LOGSTREAM IGWTV002.IGWLOG.SYSLOG SYSTEM NAME: #@$2 TRANSACTIONAL VSAM INSTANCE NAME: IGWTV002 IGW848I 07282003 16.44.26 SYSTEM UNDO LOG IGWTV002.IGWLOG.SYSLOG INITIALIZATION HAS ENDED
When the Transactional VSAM instance fails, all the new request (put/get/open) against the VSAM files receives a return code documenting that Transactional VSAM is not available. The application code needs to handle these return codes. For a complete list of these return codes, refer to DFSMStvs Administration Guide, GC26-7483-00.
For the in-flight requests, retained locks are generated against the records that were held by that unit of works. Refer to DFSMStvs Planning and Operating Guide, SC26-7348-00 to understand how to resolve a situation with retain locks.
| < Day Day Up > |
|