Preventing MFT Fragmentation

---

On NTFS, the Master File Table (MFT) is the map of each and every file on the volume and is itself a file. Every time a new file is made, a new record is created in the MFT file. As more files are added, the MFT expands. Unfortunately, files that are constantly growing, such as the MFT, are most susceptible to the extremes of fragmentation. Because the MFT is such an important file, to combat any tendency to fragment, Microsoft reserved space on the disk immediately after the MFT that is referred to as the MFT Zone. Approximately one eighth of an NTFS volume is reserved for the MFT Zone. The theory is that the MFT Zone is reserved space on the disk for the MFT to expand, thus preventing or at least minimizing MFT fragmentation.

Despite this precaution, the MFT does fragment on occasion. This is because files still get written into the MFT Zone under certain circumstances. For example, when a disk is full, files are stored in the MFT Zone. Now, suppose that some files get deleted from this volume but not the ones stored in the MFT Zone. Despite space being available on the disk, files still reside in the MFT Zone. When more files are stored onto this disk, the MFT must expand to store these new files, but the files in the MFT Zone block the way. As a result, the MFT becomes fragmented.

Several attempts have been made to eradicate MFT fragmentation. One method is to bypass the operating system and the special APIs in order to defragment the MFT online (Microsoft technical authorities have told me that they consider this approach dangerous). A safer alternative is offline defragmentation during rebooting.

In addition, some tools actually work to prevent the MFT from becoming fragmented in the first place. If the MFT is fragmented when the defragmentation software is first installed on a machine, it is quickly defragmented at boot time. From that point on, a monitoring process works to ensure that the MFT does not again become fragmented, thus reducing the number of times required to defragment it. In practice, boot-time defragmentation will have to be run from time to time, but the frequency is something that should tie in with other maintenance actions that require reboot of Windows-based systems.

The MFT is actually the heart of the NTFS file system. It is essentially an index to all of the files on an NTFS volume, containing the file name, a list of file attributes, and a pointer to the file. The data for each file is contained in one record in the MFT, referred to as the file record. If a file is fragmented, then more than one pointer is required — one for each fragment, in fact. When the file system accesses a file, it must first go through the MFT to obtain the location of that file or the location of that file's various fragments, as well as that file's various attributes. As one might surmise, an additional MFT performance barrier occurs due simply to regular file fragmentation. In that the file system must access the MFT to obtain the location of every single file fragment and then locate the file fragments themselves, there is a lot of "double work" involved.

But, how does the MFT itself get fragmented and what can be done about it? First, it should be said that NTFS has a built-in feature that under some conditions prevents MFT fragmentation from occurring; for example, as noted previously, the MFT is created with a preallocated expansion space into which it can expand without fragmenting. Under a couple of conditions, however, MFT fragmentation does occur, one of which is very common.

Converting FAT to NTFS

The most common way in which the MFT gets fragmented is when a FAT partition is converted to NTFS. A FAT partition ordinarily is converted to NTFS when such a conversion is specifically needed or desired, but it also occurs automatically during Windows NT installation, if the NTFS format is chosen. The partition created during installation is a FAT partition, and even if the NTFS format is chosen during installation, the partition is still created as FAT and is only converted to NTFS after the first boot. When a FAT partition is converted to NTFS, an MFT is created. If the contiguous free space is large enough, the MFT is made contiguous, with contiguous preallocated expansion space. However, because the MFT itself plus the preallocated expansion space comprises about 12 percent of the partition, not enough contiguous free space is usually available and the MFT is created fragmented.

MFT and Full or Heavily Fragmented Disks or Partitions

One caveat about the MFT preallocated space must be noted. NTFS, having been developed to efficiently use every bit of space on a disk or partition, will utilize the preallocated MFT space for normal files if the disk or partition becomes full or heavily fragmented. Simply said, if the disk runs out of regular space for files or file fragments, NTFS will turn to the MFT preallocated space and begin writing files and file fragments to this space. When this occurs, the MFT can only expand by continuing its growth in another space on the disk, a space not adjacent to the MFT. Thus, MFT fragmentation begins; when that space runs out, another space will have to be found, and so on.

Effect of MFT Fragmentation on Performance

When the MFT is fragmented, access to directories and files slows down tremendously as the file system works its way through the fragmented MFT to access the pointers and file attributes necessary to access every file. Combined with regular file fragmentation, this is a performance nightmare — the file system has to make multiple I/Os to access the MFT and then make multiple I/Os to access the file itself. Until a couple of years ago, it was not possible to safely defragment the MFT, as it could not be moved safely online while the disk or partition was live and in use; hence, one could defragment a disk or partition and clean up regular fragmentation but MFT fragmentation would remain. (The author has personal experience with a medium-use disk that was regularly defragmented, yet ended up with an MFT in over 9000 fragments.) Before Diskeeper 5.0, nothing could be done about MFT fragmentation, and simply opening Windows Explorer, for example, could take two to three minutes. Diskeeper's Frag Guard feature prevents fragmentation of the MFT. For the first time, a defragmenter has taken a proactive approach to eliminating fragmentation.

---