File System Constructs and Functions

In addition to the basic structures of file systems just discussed, several different constructs and functions are common in file systems today:

• Superblock

• Name space

• Metadata

• Access permissions

• Interface semantics

• Locking

• Buffers

• Journaling

These primary file system functions are discussed in the following sections.

Superblock

File systems usually have a reserved location in a storage address space to keep internal information about their operation, configuration, and status. This reserved location is called the superblock in UNIX file systems and the master file table in Windows server file systems.

The Name Space

The file system's name space provides a method for interpreting and displaying file and directory information to end users and applications. Some of the major usability differences between different file systems result from having different name space implementations. For instance, different name spaces might treat uppercase and lowercase characters differently, have different reserved characters, and allow different numbers of characters.

NOTE

For you networking people who want a networking analogy, the name space is the presentation layer of the storage stack. I used to talk about name spaces as the data representation function, but I've stopped doing that because people seem to prefer using the term name space. I still think that "data representation" is the correct abstract notion. After all, you can have operating systems with GUIs and sound capabilities (really!) where visual icons and audio leitmotivs can be associated with data files in an abstract way that could hardly be thought of as a "name" made out of letters. The file "°," formerly known as "prints.dat," immediately comes to mind. However, for this book, I'm going with the traditional and conceptually limited term name space.

Metadata and Attributes

As file systems hold increasing amounts of data, it is essential that mechanisms be found that help manage data in meaningful ways. The main file system facility for supporting data management is metadata. Metadata is descriptive information retained by the file system that describes inherent characteristics, control information, or usage history of a file. In other words, it's data about data.

Traditionally metadata has been implemented as file system attributes that indicate such things as the size of a file; various date and time stamps for when files were created, opened, and modified; and whether the file needs backing up. Metadata normally is used by system or data management functions like backup processes.

Metadata can be stored many different ways. It can be stored with the file or it can be stored in a separate and independent storage area. Chapter 16, "New Directions in Network Filing: Clustered File Systems, Distributed File Systems, and Network Storage for Databases," shows how some distributed file system products are structuring metadata differently.

NOTE

Some discussions of file systems and metadata include the layout reference system as part of the file system's metadata. That's fine, but I prefer to think of the reference system as a different entity that is often stored and used independently of qualitative descriptions of the data.

Access Permissions

File systems usually have some form of access permission scheme that determines what data different users and applications can access. As users attempt to open files, their IDs are checked against the security data associated with the file. If the user is authorized to work with the file, he or she gains the appropriate level of access. It is possible to keep files from being updated or viewed that way.

This is the most common method of providing security in storage networks. SCSI storing processes do not have any security whatsoever. SAN networking technology can provide network access security, although this is not commonly done currently.

Interface Semantics

Access control can also be provided through the file system application interface using file system semantics. The semantics of a file system describe the methods by which data is exchanged between applications and a file system and can include control over which users or applications get to work with data.

For example, a file system may have several options for opening files that can be used by application vendors for a number of reasons. The semantics open with deny-write or open with deny-open can be used when first opening a file to ensure that other workers or application processes cannot interfere with the work being done by the person or application that opened it initially. This is why you may have seen a message telling you that you cannot access a certain file because another user is working on it.

Locking

As illustrated in the preceding section, locking is a mechanism that allows file systems to support exclusive or prioritized access to a file. Locking allows the first user to finish working on a file without risking interference from other users or applications.

Locking is not necessarily done the same on every file system, and there are significant differences in the locking mechanisms used on UNIX and Windows systems. Chapter 16, which covers network attached storage, discusses some of these differences in greater detail.

File System Buffers and Cache

Memory I/O operations are approximately five to six orders of magnitude faster than disk I/O. Considering that file system performance has a major impact on overall system performance, it is not surprising that file systems load parts of their data in memory to improve performance.

File systems also keep application data in memory buffers to improve performance. In essence this is equivalent to caching, but it is done by the file system, which has detailed information about all the storage addresses where a file is stored. Unlike block-based caching, which has no file or directory context to work from, file system caching has 100 percent accuracy.

Operating systems can provide a significant amount of available system memory to be used as file system cache buffer memory. The way this memory is allocated depends on operating differences between operating systems, the amount of memory in the system, and the number of applications running.

File system buffering and caching create some consistency difficulties for storage management applications such as block-based backup and point-in-time copy. If there is data stored in host memory that has not been written to disk and there is associated data (data linked by the file system's reference system) that has already been written to disk, there is a chance that the system could suddenly fail unexpectedly, resulting in inconsistent data. Different operating systems provide various levels of support that allow storage management functions to synchronize inconsistent data. The main methods are discussed in the following section.

File System Checking and Journaling

The requirement to maintain file system consistency between stored file data and the file system's reference system is normally dealt with in one of two ways:

• File system consistency checking

• Journaling

In general, file system consistency is a problem only when an abnormal system stop occurs due to such unexpected occurrences as a sudden loss of power, a natural disaster, or an operating system bug. Most file systems have information in the superblock that indicates if the file system followed an orderly shutdown, maintaining file system consistency. When a file system starts, it reads the superblock to determine if an orderly shutdown process occurred. If it didn't, data consistency is at risk.

The file system's reference system contains information that is duplicated for redundancy. Using these redundant reference system structures, a process called a file system check (FS check) can determine whether the file system is consistent. If not, the FS check process can correct the problem to restore the file system to a usable state.

The FS check process has to verify and correlate all stored data with its internal information and reference system. The FS check process takes a long time to run for very large file systems. (A file system is large if it has a lot of files in it, and file system size is independent of the size of the storage address space it manages.) FS check operations can take hours, or even days for very large file system.

The time needed to run FS check is obviously a problem for IT organizations trying to run 24/7 operations. As systems and file systems increase in size, the time needed to check file system consistency becomes a larger liability. For that reason, file systems have adopted journaling techniques that simplify consistency detection.

File system journaling is based on database journaling concepts that maintain the referential integrity of data. Before data is written to disk, a journal entry is written, indicating the nature of the write operation. After the data is written, another journal entry is made that indicates the write completed correctly. If a journaled file system starts and discovers from its superblock that data might be inconsistent, it can look at the journal for any incomplete writes and roll back its reference system to a state that can be guaranteed to have consistency.

Obviously the overhead of journaling slows performance. However, most would agree that it is well worth it in order to allow fast recoveries from system failures.

Scaling File Systems

One of the main problems administrators face is storage scalability. When file systems reach a certain threshold of filled capacity, they tend to become fragmented and perform poorly. Those thresholds depend on a number of variables, including the application, the particular file system used, and the configuration of underlying storage.

Virtualization and volume management technologies are used to increase the size of storage address spaces, but unless the file system adjusts to use the increased storage capacity, there is no benefit to be had from scalable storage. Therefore, it is essential to have a way to increase the boundaries of the file system that manages the underlying storage address space.

Copying Files into a New File System on New Storage

One way to increase the size of a file system is to copy its contents to a new, larger storage address space. This is a fairly straightforward approach that has the side benefit of defragmenting file data as it is being written to its new location. Although it might not seem obvious, this process actually creates a new file system that manages the allocation of the new, larger storage address spaces for existing files.

The problem of using this approach is maintaining consistency with updates that occur in the original file system as data is being copied to the new file system. This is done by either quiescing the system so that it does not receive updates or through the use of a data migration facility that includes copy-on-write capabilities. Eventually, all the data is copied to the new file system, and the system can cut over to start using it for all functions.

The cut-over process can be performed manually or be accomplished by products that automate the process. Because manual processes disrupt system and application processes, automated data migration products may be well worth their cost.

File System Expansion

When a file system is created, it establishes an internal structure and reference system to fit the size of the storage address space. Unless special processes are run, there is no way for the file system to know that the storage address space it manages may have been increased. Some file systems include these processes and can expand their boundaries to accommodate increases in the storage address space they currently manage.

There are two methods: static expansion and dynamic expansion. Static expansion is done when the system is quiesced. The file system is given the new dimensions and may redistribute data over the new, larger storage address space. This redistribution process is a bit like defragmentation, but it is easier to accomplish because there is more free space to work with.

Dynamic expansion is done while the file system is running (it's always a good idea to stop application processing anyway). It appends the new storage address space to the end of the previous address space. File system expansion is a fairly complicated task given the complexity of the reference systems used in file systems and the absolute requirement to maintain data consistency.

The ability to dynamically expand a file system depends on the capabilities of the operating system, the file system, and a volume manager that can clearly identify all the various components that need to be integrated in the process.