Hack54.Make Files Easier to Find with Extended Attributes

Hack 54. Make Files Easier to Find with Extended Attributes

Define file- and directory-specific metadata to make it easier to find critical data.

Most projects and users organize their files by taking advantage of the inherently hierarchical nature of the Linux filesystem. Conceptually related items are given meaningful names and stored in hierarchical directories with equally memorable or evocative names. But unfortunately, file and directory names and structures that were memorable at the time of creation are not always equally memorable a month or two later, when you're desperately looking for a specific file.

Extended attributes are name/value pairs that you can associate with any file or directory in a Linux filesystem. These are a special type of metadata, which is the term for data about data, such as modification and access times, user and group ownership, protections, and so on. Extended attributes can be associated with any object in a Linux filesystem that has an inode. The names of extended attributes can be up to 256 bytes long, are usually standard ASCII text, and (like standard Linux strings) are terminated by the first NULL byte. The value of an attribute can be up to 64 KB of arbitrary data in any format.

Extended attributes are often used for system purposes, by tagging files with metadata about who can access them, and under what circumstances. This hack discusses how extended attributes can be extremely useful for any user or system administrator who wants to tag important files and directories with information that makes them easier to find, work with, track modifications to, and so on.

5.10.1. Getting and Installing Extended Attribute Support

Extended attributes are currently supported by all major Linux filesystems for desktop and server use, including ext2, ext3, reiserfs, JFS, and XFS. They are not currently supported in NFS filesystems, even if they are being set and used in the actual filesystem being exported via NFSthe code that transfers file data and metadata across the network does not currently understand extended attributes.

Like the ACLs [Hack #53], using extended attributes also requires that your kernel supports them, that the filesystems in which you want to use them are mounted with extended attribute support, and that the user-space utilities for displaying and setting the values of different extended attributes (attr, getfattr, and setfattr) are compiled and installed on your system.

The JFS and XFS filesystems automatically support extended attributes in the 2.6 Linux kernel. If you are using an earlier version of the kernel, see http://acl.bestbits.at for links to any patches needed to add extended attribute support to the kernel you are using.

5.10.1.1. Configuring your kernel for extended attributes.

Most modern Linux distributions provide support for extended attributes in the default kernels that they deliver. If you have access to the configuration file used to build your kernel, you can use the grep utility to check that the FS_XATTR configuration variable associated with your ext2, ext3,or reiserfs filesystem is set to y, as in the following example:

 $ grep FS_XATTR /boot/config-2.6.8-24.16-default CONFIG_EXT2_FS_XATTR=y CONFIG_EXT3_FS_XATTR=y CONFIG_REISERFS_FS_XATTR=y 

If the FS_XATTR value associated with the type of filesystem you are using is set to n, you will have to enable that configuration variable, save the updated kernel configuration, and recompile your kernel in order to use extended attributes. Extended attributes must be separately enabled for each type of filesystem you are using (with the exception of the XFS journaling filesystem, which inherently supports them). The kernel configuration options that enable them are located on the File Systems pane in your favorite kernel configuration editor (make xconfig, make menuconfig, and so on).

5.10.1.2. Configuring fstab for extended attributes.

Once you are running a kernel with support for extended attributes, you will need to make sure that the filesystems in which you want to use extended attributes are mounted with extended attribute support enabled. Check your /etc/fstab file to verify this. Filesystems mounted with extended attribute support will have the user_ xattr keyword in the mount options portions of their entries in the file. In the following example, the reiserfs filesystem on /dev/sda6 is mounted with extended attribute support, while the ext3 filesystem on /dev/hda1 is not:

 /dev/sda6   /usr   reiserfs   noatime,user_xattr   1 2 /dev/hda1   /opt2  ext3       defaults             0 0 

If your kernel supports extended attributes, you can edit this file to enable extended attribute support when you initially mount a filesystem by adding the user_xattr keyword to that filesystem's mount options, as in the following example:

 /dev/hda1   /opt2   ext3      defaults,user_xattr 0 0 

After updating this file, you can enable extended attribute support in currently mounted filesystems without rebooting by executing a command like the following, which would enable extended attribute support in the example ext3 filesystem /dev/hda1:

 # mount -o remount,user_xattr /dev/hda1 

5.10.1.3. Installing user-space applications for extended attributes.

The last step in using extended attributes on your system is to make sure that the user-space applications that enable you to display and set attributes are present. The Linux attr package provides the following three utilities for extended creation, modification, and examination:

attr

Lets you set, get, or remove an extended attribute on any filesystem object(s) represented by an inode (files, directories, symbolic links, and so on)

getfattr

Lets you examine extended attributes for any filesystem object(s)

setfacl

Lets you set extended attributes for any filesystem object(s)

If your system uses a package management system, you can query that system's database to see if the attr package and its associated library, libattr, are installed. The following is an example query on a system that uses RPM:

 # rpm -qa | grep attr libattr-2.4.16-2 attr-2.4.16-2 

You can also look for the utilities themselves, using the which command:

 # which attr /usr/bin/attr # which getfattr /usr/bin/getfattr # which setfattr /usr/bin/setfattr 

If the attr package is not installed and the binaries are not present on your system, you can find the source code or binary packages for your system by following links from http://acl.bestbits.at. You must install this package before continuing with the rest of this hack.

5.10.2. Displaying Extended Attributes and Their Values

Both the attr and getfattr commands enable you to display the name and value of a specific extended attribute on a given file or files. To look for the value of the extended attribute backup for the file hack_attrs.txt, I could use either of the following commands:

 $ attr -g backup hack_attrs.txt Attribute "backup" had a 3 byte value for hack_attrs.txt: yes $ getfattr -n user.backup hack_attrs.txt # file: hack_attrs.txt user.backup="yes" 

As you'd expect, querying the attributes of a file that does not have any returns nothing.

Note that these two commands require slightly different attribute syntax, which is confusing at best. User-defined extended attributes in the ext2, ext3, JFS, and reiserfs filesystems are always prepended with the string "user". The attr command is an older command for retrieving extended attributes and is primarily intended for use with extended attributes in the XFS journaling filesystem; therefore, it follows slightly different (older) syntax conventions. As a general rule, you should always use the getfattr command to accurately query extended attributes across different filesystems.

It is sometimes useful to query all of the extended attributes on a specific file, which you can do with the getfattr command's -d option:

 $ getfattr -d hack_attrs.txt # file: hack_attrs.txt user.backup="yes" user.status="In progress" 

If you are only interested in seeing the value of an extended attribute without any additional explanation, you can use the getfattr command's --only-values option, as in the following example:

 $ getfattr -n user.backup --only-values hack_attrs.txt yes$ 

Note that the output of this command does not have a newline appended, so any scripts from which you invoke this command must take into account the fact that its output will appear to include the bash prompt of the user who executed the script (here, a $).

5.10.3. Setting Extended Attributes

System capabilities such as SELinux make internal use of extended attributes. These attributes are not viewable or settable by normal users. However, normal users can take advantage of extended attributes to provide convenient meta-information about the files that they are working on, simplifying searching and eliminating the "now what was I working on?" problem.

Setting extended attribute values can be done with either the attr or setfattr commands. The following are examples of setting the user.status attribute to the value "In progress" using each of these commands:

 $ attr -s status -V "In Progress" hack_attrs.txt Attribute "status" set to a 11 byte value for hack_attrs.txt: In Progress $ setfattr -n status -v "In progress" hack_attrs.txt setfattr: hack_attrs.txt: Operation not supported $ setfattr -n user.status -v "In progress" hack_attrs.txt 

As you'll note from the second of these examples, the setfattr command explicitly requires that you identify the attribute you are setting as being a user-defined extended attribute. Attempting to set an attribute in any other attribute namespace, or to create a namespace, results in the "Operation not supported" error shown in this example. The third example command shows the correct syntax for setting a user-defined extended attribute with setfattr.

5.10.4. Removing Extended Attributes

The ability to set user-defined extended attributes isn't completely useful for associating user-defined metadata unless you can also remove existing attributes. You can do this with both the attr and setfattr commands, though the setfattr command is recommended because its syntax is consistent with the values returned by the getfattr command. For example, you can remove the user.status attribute from the file hack_attrs.txt using either of the following commands:

 $ attr -r status hack_attrs.txt $ setfattr -x user.status hack_attrs.txt 

5.10.5. Searching Using Extended Attributes

Extended attributes are inherently interesting, but the proof of their value is in using them cleverly. On my systems, I use a shell script called find_by_attr to query extended attributes and display a list of files that contain a specific extended attribute. The script is the following:

 #!/bin/bash # # Simple script to find files by attribute and value # - Bill von Hagen (wvh) # if [ $# -lt 3 ] ; then    echo "Usage: find_by_attr attribute value files…"    exit -1 fi attr=$1 val=$2 shift 2 # # Set IFS to TAB to allow files with space in their names # IFS=' ' for file in $* ; do     result=`getfattr -d "$file" | grep $attr | \                   sed -e "s;user\.$attr=;;" -e "s;$attr=;;" -e 's;";;g'`    if [ x$result = x$val ] ; then           echo $file    fi     done 

Using the getfattr command's -d option to dump all file/directory attributes and then searching the output for the attribute specified as the first argument with a value of the second argument may seem like overkill, but I did this to eliminate error messages from files on which the specified attribute has not been set.

You can invoke this script either from the command line, by specifying explicit filenames, or as the target of a find command, as in the following example:

 $ find . -exec find_by_attr backup yes {} \; 2>/tmp/find_by_attr.$$.err 

Note that you'll want to redirect standard error to a temporary file (as shown here) or to /dev/null in order to eliminate any noise from filenames that can't be resolved, such as bad symlinks and the like.

I use this script in combination with the user.backup extended attribute that I used in the previous command examples to identify critical files that I back up daily. This makes it easy for me to use the script as input to a backup command to save an archive of the files that I am actively working on.

Extended attributes are powerful tools that are immediately useful to users and system administrators. They serve as the foundation for Linux desktop search tools such as the GNOME project's Beagle tool, and they provide an infinitely flexible way of passing information to other programs, identifying specific files to multiple users, and accomplishing many other tasks. Once you become familiar with using extended attributes, you'll find yourself using them in more and varied ways. As SUSE says, "Have a lot of fun!"