10.7 Exploring the deleted items cache

---

When a user deletes an item, the Store moves the item into the Deleted Items folder. The item remains in the Deleted Items folder until the user makes an explicit choice to empty the folder or the client automatically empties the folder the next time the user exits. Clients such as Outlook and Outlook Express control this behavior through the option shown in Figure 10.30. Some users like to keep items in the Deleted Items folder for quite a long period, perhaps because they like to have the luxury to take time to review these items before they make the final decision to delete them. In some cases, I have known users to hold thousands of items in the Deleted Items folder. Some insist on synchronizing the contents of the Deleted Items folder to their offline Store-perhaps in order to review the contents offline. Given the transient nature of the majority of email, it is best to encourage users to make the decision to keep clean mailboxes by emptying the Deleted Items folder when they exit their client. You can give users "help" in this respect by using tools such as PROFGEN or ProfileMaker (www.autoprof.com) to generate profiles with a preset option to empty the Deleted Items folder when Outlook clients exit.

Figure 10.30: Options in Outlook (left) and Outlook Express (right) to control the Deleted Items folder.

Up to Exchange 5.5, the story stopped here. Once deleted, items stayed deleted. However, users make mistakes and sometimes delete items when they should not. When this happened, administrators had a simple choice:

Either tell the user that they could do nothing to recover the deleted item or go ahead and restore the mailbox database on another server, find the item, export it to a PST, and provide it to the user. Given these options, most administrators chose to decline the chance to restore databases and users suffered-unless they held a sufficiently high position in the organization to convince the administrator to change his or her mind. All Exchange servers now support the option to maintain a cache of deleted items that remains in the Store until a set deleted retention period expires, at which point the Store removes the items from the database. This is a great feature for both users and administrators alike because it saves work for everyone.

10.7.1 Recovering items

Pushing work down to users is always popular with administrators. Appeals to the help desk to recover an important message can now be bounced back to the user, but only if he or she has the right client. MAPI clients from Outlook 98 onward include the option to recover deleted items from the cache through the Recover Deleted Items option from the Tools menu. From Exchange 2000 SP2 onward, OWA also includes the Recover Deleted Items option. You can only recover items when you are connected to a server. You then see a list of items in the cache (Figure 10.31), and, as you can see, you can recover all your deleted spam, just in case you missed reading some interesting information.

Figure 10.31: Recovering deleted items.

You can still recover deleted items if you use Outlook 2003 in cached Exchange mode, since this operation forces Outlook to connect to Exchange. However, recovered items do not appear in the Deleted Items folder immediately, because Exchange recovers them into the server-based folder, and there is a time lag before the client synchronizes the recovered items into the cache.

Before you can recover deleted items for public folders, the properties of the Public Folder Store must include a deleted item retention period, and the user who attempts to recover the items must have write permission for the public folder (see Knowledge Base article 811358 for full details). Note that you cannot use a public folder "favorite" to recover items, since this is merely a pointer to the actual folder. Instead, select the public folder from the hierarchy and then select the option to begin the recovery, which you accomplish by selecting the desired items and then clicking on the recover item. Outlook then moves the recovered items back into either the Deleted Items folder or the public folder.

If someone does not use a recent Outlook or OWA client, an administrator can log on to his or her mailbox with Outlook and recover the items. Certainly, this strategy can work for a limited period or if you only have a small number of clients in this situation and you have a plan to upgrade to the latest client software. However, it is not viable when you have to manage large populations of IMAP or POP clients.

By default, the Deleted Items folder is the only folder enabled for item recovery. However, clients can delete an item in place, meaning that the item does not progress through the Deleted Items folder before it enters the cache (use the SHIFT/Delete combination to do this with Outlook). If users are accustomed to delete items in place, you may want to enable deleted items recovery for every folder to allow users to recover items in their original folders. To enable deleted items recovery in all folders, create a new DWORD value at the following registry location and set its value to 1-this works for all versions from Outlook 98 through Outlook 2003:

HKEY_LOCAL_MACHINE\Software\Microsoft\Exchange\Client\ Options\DumpsterAlwaysOn

While you can force the Outlook user interface to allow item recovery from every folder, you are not able to filter deleted items, so the Recover Deleted Items option only shows items originally deleted from a specific folder. In other words, when you select the Recover Deleted Items option, you see the complete contents of the cache-items deleted from all folders. This proves that Exchange keeps all deleted items in a single cache and does not maintain a separate cache for each folder.

10.7.2 Setting a deleted item retention period

The Store uses a flag (a MAPI property) to enable "soft" or two-phase deletes. When a client empties the Deleted Items folder, the Store hides the items from view by setting the flag, which has the effect of placing the items into the Deleted Items cache, the collection of items within the Store that have the deleted item flag set. The cache does not exist in a separate area in the Store. Instead, the items remain in the Deleted Items folder, but you cannot see them because the Store does not reveal their existence unless you use the "Recover Deleted Items" option. The Deleted Items folder does not exist for public folders, but you can still recover a deleted item by opening the public folder and then using the Recover Deleted Items option.

Deleted items remain in the cache until their deleted item retention period passes. A property of a Store database defines the deleted item retention period. You can set the deleted items retention period by individually setting the properties of each Store (Figure 10.32) or by applying a system policy to Stores on one or more servers that you select through ESM. The latter method is preferable in large organizations, since it allows you to impose the same policy on a large number of Stores at one time. It is nice to have the opportunity to control settings on an individual Store too, since you might have a situation where you need to apply a different policy. For example, you might decide that the Mailbox Store that holds executive mailboxes has a retention period of 90 days, just to ensure that you can always recover messages belonging to people who might be able to force a database restore. Note the checkbox in Figure 10.32 that refers to backups. You can opt to permanently remove items even if you have not backed up the Store, or you can keep them in the cache until you perform a successful backup. The second option is preferable, since you never want to leave yourself in a situation where you cannot recover an item.

Figure 10.32: Setting the deleted items retention period.

You can also define specific retention periods for selected mailboxes by setting a property of the user account through AD Users and Computers (Figure 10.33). Because of the time-intensive nature of setting individual limits on mailboxes, this is not a recommended option and should only be taken when necessary. For example, during a legal discovery action, lawyers may compel you to retain all messages sent and received by particular users, such as the officers of the company or those involved in particular deals. You can either group all the affected users into a single Mailbox Store, apply the desired retention period to the Store, or edit each user's properties. The latter option is preferable if the target accounts use mailboxes on a set of distributed servers, perhaps spread across a wide geographical area.

Figure 10.33: Setting a specific retention period for a mailbox.

Note that the Store does not calculate the size of items in the deleted items cache when it applies quotas to user mailboxes. Some users exploit this fact when they come close to exceeding their quota by selecting large items from their mailboxes, deleting the items, emptying the Deleted Items folder, and then performing whatever actions they wish. Later on, when their quota frees up, they can recover the items from the cache.

Figure 10.34 shows how to add Stores to a system policy.

Figure 10.34: Adding Stores to a system policy.

10.7.3 Cleaning the cache

The background Store maintenance tasks performed by the System Attendant remove items in the cache after their retention period expires. The exact schedule that controls background processing is set through a database property, and it is usually started nightly just after midnight. In database terms, removing items from the deleted items cache is known as "hard deletes," because the Store removes the items from its database tables and you cannot recover them later without performing a database restore. You can get an idea of the clean-up processing done by the Store by looking for event 1207, which reports the number of items and their size in the cache before and after scanning for items with an expired retention period. As an example, Figure 10.35 shows that clean-up processing removed 487 items, which occupied 10,036 KB.

Figure 10.35: Items removed from the deleted items cache.

10.7.4 Sizing the cache

As soon as the deleted items cache appeared in Exchange, administrators began to worry about the impact of the cache on database sizes. After all, if the cache holds items instead of removing them from the Store, database sizes must increase. The question is, how much extra space will the cache occupy? Note that the maximum size of the cache is 4 GB per store, although you are hardly likely to accumulate so many deleted items unless your retention period is excessive.

The cache begins to grow as soon as you install a server and it stabilizes over time. You can expect the cache to stabilize in size after twice the retention period. In other words, if you set the retention period to be a month, the cache should be stable in about two months. Once stabilized, the size of the cache will vary, but not dramatically unless user behavior changes. For example, if you double the default mailbox quota, users do not have to delete items to stay within the old quota and the deleted item cache will reduce. However, the size of the database grows to reflect the new quota.

Obviously, the retention period is the biggest influence on the cache size-the longer the retention period, the larger the cache. Most installations use retention periods between 7 and 15 days, the logic being that if someone does not realize that he or she has deleted an item in error in that time, the item probably is not very important. Other influences on cache size include:

• Mailbox quotas: Smaller quotas force users to delete items more quickly, so the cache tends to grow.

• Message volume: Higher message volumes create faster item turnover in user mailboxes, so the cache tends to grow.

• Organizational culture: If you train users to read messages and make a decision to delete or keep, the cache tends to grow (but users can work within lower quotas, so databases are smaller). Caches tend to be smaller when users keep everything and only delete items when necessary. You can use the Mailbox Manager to assist users by implementing a policy that automatically cleans out obsolete items from folders on a regular basis.

These rules of thumb have varying impact from company to company. You can make an accurate assessment by taking data from production servers to gain a view of actual cache sizes. You can then use the data to predict the likely impact on the cache size if something changes. Exchange provides a set of Performance Monitor counters to track the size of the deleted items cache. These are:

• MSExchangeIS Mailbox-total count of recoverable items

• MSExchangeIS Mailbox-total size of recoverable items

• MSExchangeIS Public-total count of recoverable items

• MSExchangeIS Public-total size of recoverable items

The Store maintains separate counters for every active database on a server as well as overall totals for Mailbox and Public Stores. As with other performance data, the Store updates these counters on an ongoing basis, so you can track the growth of the cache over time. Figure 10.36 shows the Performance Monitor displaying the size of the deleted items cache for a Mailbox Store-200,895 KB is in the cache, equating to 10,052 items (reported by the count of recoverable items). Unless the server is very active, with users deleting items frequently, viewing cache data in this way is relatively uninteresting. The Store updates the counters as clients delete items, so you can see the cache size changing in real time. Treat the information as a snapshot that provides a quick overview of the cache or a data point that you can observe and record on an ongoing basis to track the size of the cache over time.

Figure 10.36: Using Performance Monitor to track the size of the deleted items cache.

It is more interesting to look at the deleted item cache by user. You can do this through ESM by including the "Deleted Items (KB)" column in the data displayed in the console. This column is not one of the default column set, so you need to modify the view. Select a Mailbox Store, right-click, and then select "View" and "Choose Columns" to work with the columns displayed by ESM. Figure 10.37 shows how to add the column to the view. You may also want to move the columns around so that you can compare the size of the deleted items against the overall mailbox size. Figure 10.38 shows the resulting view.

Figure 10.37: Adding the deleted items column to a display.

Figure 10.38: Using ESM to view deleted items by mailbox.

After ESM displays the data, you can capture it by clicking on the context-sensitive menu, selecting the "Export List" option, and choosing the target file name and type-normally CSV (Comma-Separated Values). You can use Excel and other programs to create many different analyses from the captured data, but the essential data we require is a sum of the deleted items in each mailbox.

After you know the total size of the cache for an individual Mailbox Store, you can calculate it as a percentage of the overall database size. Remember to include the size of the EDB and STM files in the calculation. For example, the Mailbox Store illustrated in Figure 10.38 uses a ten-day default retention period and allows large mailbox quotas; the total size of the cache as reported by ESM is 437,178 KB. Using the file sizes reported by Windows Explorer, we calculate the cache as follows:

Size of mailbox EDB:	8.89 GB
Size of mailbox STM:	0.46 GB
Size of database:	9.34 GB
Total count of deleted items in mailboxes (reported by ESM):	0.54 GB
Percentage size of cache:	5.76%

The problem with this calculation is that it compares absolute database size on disk against the reported size of the cache, so "white space" is included in the calculation. White space is the set of unused pages in the database that used to hold deleted items. The Store reuses deleted pages to hold new items, but there is always a certain amount of unused space in any Mailbox or Public Store. You can reduce the amount to zero by rebuilding a database with the ESEUTIL utility, but it starts to accumulate as soon as the database goes back into use. Expect to see white space occupy between 5 percent to 15 percent of a database's reported size on disk, with larger percentages in databases where you have moved mailboxes to another database and you have not rebuilt the database. In our example, the reported size of all mailboxes in the database is 3.54 GB, so the deleted items cache actually represents 15.2 percent of this amount. You can decide to use this calculation to assess the likely impact of the deleted item cache on database size, but most administrators compare the reported size of the cache against the database size on disk, since they are more concerned about the absolute increase in database size.

With these points in mind, you can use the guidelines outlined in Table 10.4 to estimate the potential size of the deleted items cache on an Exchange server given a range of mailbox quotas. The percentage of the database occupied by the cache then varies according to user discipline. If users delete messages quickly, the cache will be at the lower end of the scale, but if they tend to retain messages, it is likely to be at the upper end.

Table 10.4: Estimating the Size of the Deleted Items Cache

Cache Size	Large Mailboxes (200 MB+)	Medium Mailboxes (100-200 MB)	Small Mailboxes (50-100 MB)
5-10%	10 days	10 days	20 days
10-20%	20 days	25 days	35 days
15-20%	30 days	40 days	50 days

It is important to emphasize that these are just guideline figures and your mileage will vary. Every organization is different, and the hard data you gather from your own servers will provide guidelines that are more accurate. Known instances exist where the size of the deleted items cache is much larger than the 30 percent maximum shown here, but these are the exceptions that prove the rule. Remember that the larger the cache, the longer maintenance operations such as background Store defragmentation will take, and this is the reason why you should be concerned about the size.

The deleted items cache is a tremendous help to administrators, because it removes the need to recover databases if users delete items in error. Compared with its benefits, the additional burden in disk space requirement is not onerous.

---