7.8 ESE database errors

---

Three types of errors can occur in an Exchange database:

• Application (Exchange or ESE) level

• JET (database) level

• Physical or page level

Table 7.7 describes the most common errors. See Microsoft Knowledge Base article 314917 for more information.

Table 7.7: Exchange Database Errors

Error	Meaning	Indication
-1018 (JET_errReadVerifyFailure)	The checksum read from disk is not the same as the checksum that was originally written, or the page number does not match the page number that is expected.	Page-level corruption may have occurred.
-1019 (JET_errPageNotInitialized)	A page that is expected to be in use is not initialized or empty.	The page may be corrupt.
-1022 (JET_errDiskIO)	A generic error indicating that the requested page cannot be read by ESE.	A physical disk error is probable.

A -1018 error occurs when the ESE engine makes a request to Windows to fetch a particular page from the database and ESE then returns unexpected data, normally a different page number or checksum. ESE flags the error to prevent further access to the page and to let administrators know that action is required. Continuing to attempt to access a damaged page (or damaged section of a database) usually results in even more data loss than may occur if you immediately notice a problem and take appropriate action. For example, if you take a backup of a flawed database, you will not be able to restore it.

Many different underlying problems can cause a wrong page to be returned, including a disk or controller error or faulty SCSI cable termination. In the past, Microsoft has also pointed the finger at antivirus software and questioned whether the methods used to access the Store to detect incoming viruses are the root cause of some -1018 errors, especially when vendors use nonapproved APIs for the Store. While no conclusive answer to this question has ever been determined, you can eliminate the antivirus issue, since the major antivirus vendors now support Microsoft's VS (Virus Scanning) API. Thus, if a problem occurs because of an antivirus product, the issue is now likely to be in Microsoft's own code rather than in the addon product.

Microsoft has worked hard to eliminate false errors that caused much concern in earlier versions and to institute better problem reporting for errors. For example, Figure 7.25 shows how Exchange reports a problem reading a page. In this instance, the Store requested a certain page number and ESE returned a page whose page number was different. While this is not a -1018 error, a condition now exists that will eventually result in a -1018 error if its underlying cause is not some transient hardware condition. In either case, the sheer fact that the Store is having difficulty in reading pages accurately is enough for a system administrator to become concerned.

Figure 7.25: Error 475 precedes a potential -1018.

Another example is the retry logic introduced in Exchange 5.5 SP2 (and used since) in an attempt to solve the problem where a temporary hardware glitch would cause a -1018 error to be reported. Instead of immediately accepting that a problem exists, the Store pauses and then enters a retry pattern, where it attempts to read the failed page up to 16 times before concluding that a problem is real. The Store makes no further attempt to reference the page after the failure occurs, since this might cause further corruption.

Even if a retried read is successful, the Store still flags a -1018 error to the event log along with details of the failed page and the number of read attempts, because you must investigate even a transient problem to ensure that it is not an indication of a deep-rooted problem that will eventually corrupt a database. You should check for loose SCSI cables, controller boards, and any other connections to ensure that they are not the root cause. Also, make sure that firmware upgrades for disks and controllers are applied so that known bugs are not maintained.

While failing hardware is often the cause of transient errors, sometimes they are associated with times of heavy system load where the software cannot keep up with the demand. However, you will not see this type of problem with current versions of Exchange. Such -1018 errors may also be generated during online backups. The Store verifies checksums as it streams each 4-KB page out to the backup media and halts the backup if a page fails a checksum test to ensure that the backup does not contain corrupt data.

ESE signals -1019 errors when it attempts to read a page that it expects to contain data only to find that the page is not initialized or it is empty. Pointers within the database may be corrupt, so that a link exists to an incorrect page or a bug has caused bad data to be written. ESE generates a -1022 error when it cannot read a page because of an I/O problem. This is a catchall error, which usually means that the database is damaged for one reason or another and you need to repair it. This is when a solid knowledge of Exchange disaster recovery techniques is invaluable.

7.8.1 Hard and soft recoveries

Recovery is the process that the Store goes through to make a database consistent. Databases become inconsistent when the Store process terminates abruptly without having the chance to commit all outstanding transactions into the database. Replaying transactions after a power outage or similar failure is referred to as a "soft recovery." The database is available and just needs to be updated from the transaction logs. As outlined previously, ESE can determine what transactions are outstanding by reference to the checkpoint file and the transaction logs. A hard recovery means that a hardware failure has meant that you have to recover data from a backup. Exchange performs different processing during hard and soft recoveries.

Soft recovery is an automatic process that requires no human intervention. ESE always checks that its databases are consistent before it commences normal operations. ESE takes the following steps each time a client (such as the Store) starts an ESE instance with the JetInit function:

• ESE locates the current transaction log by reference to the checkpoint file.

• ESE verifies the signature in the transaction log to ensure that it matches the database.

• ESE consults the checkpoint file to detect whether it needs to roll forward any transactions.

• Normal operations begin and client connections are accepted.

The same sequence takes place when a soft recovery occurs. Many system administrators are probably unaware that it has happened, although evidence exists if you care to look for it in the application event log.

Soft recoveries

An inconsistent database must go through a successful soft recovery process before the Store is able to mount it. If the soft recovery does not work, you have to run ESEUTIL to repair the database and then attempt to mount the fixed database. Apart from entries in the application event log, the vast majority of soft recoveries pass without the need for administrator intervention, since the Store checks to see whether it must perform a soft recovery for each database every time the Information Store service restarts.

During a soft recovery, the Store checks the database header to determine the log files that are required to make the database consistent. This information is held in the "Log Required" field, which records the log generations that are needed.^[3] The Store then checks the signature in each log to ensure that it belongs to the database and then searches the logs for transactions that have not been committed to the database. ESE reads each page referenced in a transaction and compares its timestamp. If the timestamp on the page is earlier than the timestamp in the log record, ESE replays the operation. If the timestamp is equal or higher, ESE ignores the transaction. While it processes transactions, ESE tracks the transactions that it begins and the ones that it fully commits. At the end of the pass, any uncommitted transactions will have all of their operations rolled back. Adjusting the database in this manner is sometimes referred to as "physical redo, logical undo." ESE must process all transactions before the Store can allow clients to connect.

Note that programs that operate on the database at low level, such as the ESE maintenance utility, also check transaction logs for outstanding transactions before they commence operation. The only exception to this rule is when you run ESEUTIL/P to repair a database. When you repair an inconsistent database, ESE ignores the log files and they cannot be used afterward, since the internal structure of the database is changed as ESEUTIL repairs the database to make it consistent.

As can be seen in Figure 7.12, transaction logs contain hard-coded paths to the database. If you move the database to a new location before a recovery operation begins, ESE ignores the logs when the Information Store service restarts. If you attempt a recovery using logs that do not match the location of the database, ESE flags a -550 error. From a hardware perspective, the time taken to recover a log depends on the speed of the CPU and disk I/O subsystem, but even the smallest system should be able to recover logs at better than one log/minute. Surprisingly, it is faster to process a few messages with large attachments rather than a set of smaller messages. This is because ESE has fewer pages to read and process. The more pages to read, the longer recovery will take. The speed of current CPUs largely disguises this fact. Naturally, recoveries that do not have to actually process (or redo) many transactions, because the database was largely up-to-date when the crash occurred, are much faster than recoveries that have to process thousands of outstanding transactions. This is because each redo operation results in a "dirty" page, which must be written out to disk, thereby increasing the I/O load. Any time a log generates many redo operations, you will see a spike in the I/O load on the server.

Not all log files are necessarily required during a recovery operation. Some logs contain details of transactions that have been fully committed to disk and exist in a database. ESE uses the checkpoint to determine whether a transaction has not been fully committed. It is often a good idea to copy all of the transaction logs to a holding directory before beginning any recovery operations, just in case you run into a problem with the recovery and have to go back and start over again.

Recovery can only occur if the target database is in a consistent state when the operation begins. Do not expect success if you restore a corrupted database. You will cause further damage if ESE attempts to replay a transaction for a record on a page that does not exist. In this case, ESE proceeds to locate whatever data is in the specified location and delete it. The replay operation will then proceed and perform whatever instruction is in the log file. For instance, ESE might insert new data into the page. The new data has no relationship to the other contents of the database that depended on the original data, so the result is a logical corruption. The requested record might not exist because ESE never wrote the page to the database due to a hardware or software problem. Perhaps physical corruption occurred after the page was written and the header had been altered.

The problems that occur after you restore a database only to find it is corrupt are all the justification you need to verify backups on a regular basis. You should further protect your system by monitoring the event log to discover and respond to any database errors that ESE reports in the application event log. Never attempt to replay transaction logs against a database other than the original. You cannot, for instance, take transaction logs from one server and replay them on another in an attempt to recover messages after a crash. You should not rename transaction logs or attempt to create a missing log to complete a set (perhaps by copying another log file). The checkpoint file should stop any attempt to circumvent the protection mechanisms built into database recovery, but only if it is available. If you delete the checkpoint and then mess around with transaction logs, you are playing with fire. It is quite possible that the ESE engine will limp its way through the logs, cheerfully corrupting your database to an unrecoverable state, and then report "Recovery complete." Of course, recovery is not complete and all you have managed to create is a useless set of data.

The current transaction log must be available before any recovery operation can succeed. If a hardware failure corrupts this file, it may be possible to rename the previous log to become the current transaction log and proceed with the recovery, but at your own risk. It is highly likely that modifications made by transactions span the corrupt log file and the file that you rename. Some of the modifications may have been committed into the database, but when you run the recovery, you will end up in a situation where incomplete transactions are committed. There is no way to complete the transaction due to the missing log. There is no way to redo or undo these transactions, so you will end up with a corrupt database. In this situation, no clue will exist in the event log, since ESE has been "tricked" by renaming the transaction log and has no idea that transactions are incomplete. The effect of the corruption may only become evident later when further transactions attempt to manipulate the affected pages.

You can experience much the same type of corruption if you use write- back caching on a controller that does not protect its cache if a failure occurs. Write-back caching tells an application that a write operation is complete when data is written into the cache. In this case, ESE will assume that it can flush pages and will proceed to do so. If the controller then fails, some data will remain in the cache. This is not a problem if the controller protects the cache or you can transfer the cache to a replacement controller, but it certainly is if you replace the cache along with the controller. In this case, when you restart the system the cache is empty and ESE cannot recover the outstanding transactions, which results in a severely corrupted database. Never use a controller that does not support a fully protected and removable cache equipped with good battery backup, and always remember to move the cache to the new controller after installation.

Hard recoveries

Hard recoveries need human intervention. Exchange cannot deal with a hard disk failure on its own. After you replace the failed disk, you need to restore the last full backup as well as any incremental backups that have occurred since. Essentially, you need to assemble a complete set of transaction logs containing all the transactions that have occurred since you took the last full backup. The aim is to provide Exchange with the opportunity to replay details of every transaction that has occurred since the last full backup, and you need a complete set of logs for this to be possible. When you have recovered the databases and all of the necessary transaction logs, you can restart the Store service, which then begins to replay any outstanding transactions found in the logs.

To make things as simple as possible, I normally recommend that installations take a full backup each night to avoid the requirement for multiple restore operations. Given the speed of the backup hardware and software available today, servers that use less than 50 GB for the Store databases should always take a full daily backup. Larger databases deserve more attention, because the time required to take a backup is usually much longer, and a substantial investment in backup hardware may be required to shorten the backup period. It is best practice in many large companies to take steps to attempt to keep any online backup to less than four hours. The logic here is that if the backup takes four hours, then a restore will take eight hours (restores usually take twice as long as backups). Eight hours is one working day, and service metrics will take a very large hit if you cannot restore a server to good health in one day. You can use a simple equation here. You know the size of the databases that must be backed up-let us say that the total size of the databases amounts to 60 GB. The backup hardware and software combination must therefore be capable of processing data at 15 GB/hour to remain inside the desired four-hour period. If this is not possible, you need to upgrade the backup hardware or software (or perhaps both) until the goal is met. Striped arrays of DLTs provide the best performance in terms of throughput, but they are expensive. The true value of the hardware investment here is often not realized until the second day of a disaster recovery, at which time it is too late to rescue the career of the unhappy system administrator.

Exchange does not allow soft recoveries to proceed against databases that need hard recoveries. You might, for instance, restore a backup and then run ESEUTIL /P to repair the database and make it consistent again. After ESEUTIL is complete, the database header shows the database to be consistent, but recall that up to Exchange 2000 SP2, a backup contains the database and the patch file that holds information about pages that user activity has changed after the backup application originally wrote the pages to the backup media. Running ESEUTIL /P ignores the patch file, so ESE will never apply the details contained there back to the database, which provides a potential to create a logical corruption. This sequence of events is unusual, and the removal of the patch file from backup operations in SP2 simplifies matters, but it certainly can happen if unwary administrators think they are doing the right thing to "fix" the database after a restore. Exchange protects itself by noting whether a hard recovery is required and issuing error -544 if you attempt to repair a database after a restore.

Because hard recoveries rely on the availability of log files, it is obvious that if the log files are on the same drive as the database you run the risk that any failure will affect both database and log files. Without log files any recovery becomes impossible. This is a fundamental reason for you to separate the database and transaction logs on any Exchange server intended for production work.

^[3] . The database stores log generations in decimal. You have to convert these values to hex to determine the log file names that are needed.

---