Database File Structure

Each Exchange 2003 database consists of two files: the rich text file (ending in .EDB), which holds e-mail messages and Message Application Programming Interface (MAPI) content; and the native content “streaming” file (ending in .STM), which holds all non-MAPI information. Hence, a mailbox store (formerly known as a private information store in Exchange 5.5 and earlier) now consists of an .EDB and an .STM file pair. The default name for the first mailbox store is Priv1.edb and Priv1.stm (Figure 2-1). Similarly, a public folder store (formerly known as a public information store in Exchange 5.5) now comprises the files (named by default) Pub1.edb and Pub1.stm (Figure 2-2). Each database incorporates both files, and Exchange Server 2003 treats the files as one unit. When Exchange reports the size of the store, it gives the combined size of the rich text file, the native content file, and the transaction logs. Both types of data are stored in an Extensible Storage Engine (ESE) database format. (ESE is discussed later in this chapter.)

Figure 2-1: The Database tab for a mailbox store property sheet.

Figure 2-2: The Database tab for a Public folder store property sheet.

Rich Text File

The rich text file holds messages from MAPI clients such as Microsoft Outlook. MAPI clients access these messages without a conversion process running on the server. The rich text file is the same as the Exchange 5.5 information store. It is an .EDB file that uses transaction logging just as it did in Exchange 5.5.

On-Demand Content Conversion

When a MAPI client attempts to read a message from the rich text file, no conversion is necessary when the message format was originally rich text or plain text, since the message is in the client’s native format. However, if another type of client, such as an HTTP client, attempts to read a rich text or plain text message from the rich text file, Exchange converts the message to the requested format. This process, in which messages are converted for dissimilar clients, is known as on-demand conversion.

When a MAPI client attempts to read a message that was originally formatted as an HTML message, portions of that message might reside in the .STM file. If they do, this message will also need to be converted in the memory space of the Exchange server.

Exchange Server 2003 does not automatically convert data when it writes the data to the database. Instead, client actions, such as a dissimilar client requesting data from the rich text file, cause the data to be converted. This is known as deferred content conversion. For example, suppose that a Web-based client posts a message to an Exchange server. That message is stored in the native content file. If another client requests this message over TCP port 80, Exchange streams it out of the native content file without converting it. However, if an Outlook client requests that message, that client is attempting to read information from a database that is not in its native format. In this case, Exchange converts the message in the memory of the server and then passes it to the client. The message is not moved from the native content file to the rich text file before being sent to the Outlook client.

If the Outlook client makes a change to the message and then saves it, the message is copied from the native content file to the rich text file and then deleted from the native content file. During the copy process, the updated message is converted from its native content to rich text before being written to the rich text file.

Native Content File

In Exchange 5.5, messages are always written to the database in Microsoft Database Encapsulated Format (MDBEF). If a message in non-MDBEF format needs to be written to the database, the Imail process converts it to MDBEF format so that it can be written to the information store.

In Exchange Server 2003, the native content file holds all non-MAPI messages in their native format, including HTTP and IMAP4. Native content files can include audio, video, voice, HTML-formatted messages and other multimedia formats. This file simply holds the raw data with no overhead, such as compression or B-tree overhead. In addition, the page checksums and space-usage information (information that tells ESE which pages in the database are used and which ones are free) are stored in the .EDB file.

Messages delivered from the native content file are streamed to the client. The streaming is fast due to the presence of the Win32 kernel mode component ExIFS, the Exchange Installable File System. (See the next section, “How Streaming Works,” for more about this component.)

Any Win32-based client is able to access the data in the native content file over server message block (SMB) architecture. This capability means that any kind of data can be placed in the native content file, and it will be accessible to your LAN-based clients as regular file shares as well as being available over standard Internet protocols such as HTTP.

How Streaming Works

Figure 2-3 illustrates how streaming works in Exchange Server 2003. Let’s assume that a Post Office Protocol 3 (POP3) client requests a message from the native content file. The POP3 client connects to the POP3 server in Internet Information Services (IIS), and the server asks the Store process (Store.exe) to manifest a handle to the message inside the native content file. The Store process negotiates a handle for the message with the kernel mode Exchange Installable File System (ExIFS) driver. The ExIFS driver locks the message, creates the handle, and presents the handle back to the Store process. It is also worth noting that page checksums are not verified during outbound transmission. Once manifested, the handle is handed back to the POP3 virtual server across the epoxy layer in user mode. (The “Front-End/Back-End Servers” section, later in this chapter, discusses the epoxy layer.) The POP3 virtual server then issues a TRANSMITFILE command, which is a high-performance API that uses both the handle and sockets to transmit the file. The TRANSMITFILE command is issued to the Auxiliary Function Driver (AFD), which essentially acts on behalf of Winsock, streaming the file out of the NT Cache Manager by talking to the ExIFS driver. It is important to note that at no time does the streaming data enter user mode, which makes this architecture very fast and very reliable. The handle contains the list of database pages that hold the information being requested. During the locking phase, ESE reserves the pages that hold the information and gives them to ExIFS.

Figure 2-3: Streaming architecture of the native content file.

IIS can write to the native content file as well. ExIFS will make the native content file appear to IIS as multiple, yet virtual, files. When IIS wants to write to the native content file—as would be the case, for example, when an inbound message has a graphics attachment—ExIFS creates virtual files to which IIS can write. Then ExIFS streams this data into the native content file and passes the list of pages to the Store process. ESE then commits the pages by logging the information in the transaction logs. Page checksums are stored in the rich text file so that only data is held in the native content file.

ExIFS requests database space from ESE when necessary and allocates space for new messages from its reserved space. This allocation makes the write operation faster than if the space was not previously allocated.

Single-Instance Message Store

Exchange 2003 databases continue to support the Single-Instance Message Store (SIS) feature, meaning that a message sent to multiple recipients is stored only once as long as all the recipients are located in the same database. SIS is not maintained when a mailbox is moved to a different database, even if it still resides in the same storage group. Moreover, SIS does not span multiple databases in a single storage group.

Here is an example of how SIS works. John, the Exchange administrator at (a fictional) company named Trains by Dave, Inc., has deployed two storage groups consisting of four databases each. Each group contains two mailbox stores and two public folder stores. Mary, a user in John’s network, sends a 1- MB message to a distribution group of 40 recipients, all of them residing in the first storage group, with 30 of them on mailbox store 1 and the other 10 on mailbox store 2.

Without SIS, the message would be copied 42 times (40 copies for 40 users plus 1 copy for the transaction log plus 1 copy in the Sent Items folder of the sender), requiring a whopping 42 MB of total disk space to store the message. However, as Figure 2-4 shows, with SIS only three copies of the message are held: one in the database of mailbox store 1, one in the database of mailbox store 2, and one temporarily in the transaction log. Hence, sending this message to these 40 recipients requires only 3 MB of total disk space, saving 38 MB.

Figure 2-4: How the Single-Instance Message Store feature works.

Storage Groups and Multiple Databases

A storage group consists of an instance in memory of the Extensible Storage Engine (ESE) transaction logging system (described later in this chapter), the set of transaction logs, and their corresponding databases in the group. The ESE instance is managed by the Store.exe process, which runs as a single process. Each storage group can hold up to 5 databases. There can be up to 4 storage groups per server, and each group can hold up to 5 databases, so the limit is 20 databases per server. Each database can be either mounted (started) or dismounted (stopped). Although a corrupt database cannot be mounted, it also cannot stop the Store.exe process from running or stop other stores in the storage group from being mounted or dismounted. (Exchange Server 2003 introduces a new Recovery Storage Group disaster recovery function. This feature will be discussed in Chapter 28, “Troubleshooting Exchange Server 2003.”)

Having multiple databases in multiple storage groups gives you great flexibility regarding how you manage your users and databases. For instance, you can group your users together by department or location and create their accounts in a common database. You can also control your user-to-database ratio, so as your company grows, you can keep your databases at a predetermined size and simply create another database for new users when necessary.

Also, the databases are very scalable, meaning that a single database can have from one mailbox up to (theoretically) an unlimited number of mailboxes. Neither extreme is recommended, however, and the actual limit to the number of mailboxes in a database will depend on factors such as hardware capacity or the amount of time necessary to back up or restore your databases. In addition, databases can be scheduled individually for backup, and you can perform multiple backups on multiple databases to multiple tape drives simultaneously. This feature helps reduce the window of time necessary to back up your databases.