Project12.Power Globbing

Project 12. Power Globbing

"Globbing is great, but how do I customize the way it works to my own liking?"

This project shows you how to extend your globbing skills beyond the basics. It gets Bash to reveal the list of filenames before they are processed, customizes the way Bash expands filename patterns, and reveals more powerful pattern-matching operators.

Check Completion

Imagine typing the command

$ rm some-filename-pattern

and getting the filename pattern wrong or forgetting about some essential files that happen to match it. Not good. It's wise to dry-run a pattern first, using a neutral command such as echo or ls.

$ ls some-filename-pattern <visually check the list of files> $ rm some-filename-pattern

Even better is to use Bash's glob list expansion. Type the pattern, press and release Control-x, and then press g to get Bash to list the expansion.

Let's dry-run an rm command and then issue it:

$ rm b?g<Control-x>g bag  bfg  big  bug $ rm b?g

Alternatively, type Control-x * to get Bash to expand directly onto the command line.

$ rm b?g<Control-x>* $ rm bag bfg big bug

If the dry run looks good, press Return to execute the command. Otherwise, press Control-u to delete the command line.

(This is similar to the more familiar technique of simply pressing the Tab key. Tab differs in that it either completes if a single match is found, or, after a second Tab, lists all the matches, if many matches are found.)

Customize Completion

Bash's globbing can be controlled and customized through shell options, which are settings that control the way Bash works. You can switch them on and off with the shopt command, specifying the command-line option -s (set) to switch on an option or -u (unset) to switch off an option. Issued without command-line options, shopt displays the current settings of all the shell options. Only the options relevant to globbing are shown in the example below (by filtering out unwanted options with grep). By default, all options are off.

$ shopt | grep glob dotglob         off extglob         off nocaseglob      off nullglob        off

Tip

Type help shopt to learn more about Bash shell options.

Set nocaseglob

You may make globbing case insensitive (that is, get it to ignore case) by setting the shell option nocaseglob. (This does not affect tabbed completion in cases where globbing is not involved.)

With option nocaseglob switched off (the default setting), the pattern *a does not match files ending in A.

$ echo *a smalla

If we switch on nocaseglob by typing

$ shopt -s nocaseglob

we'll see that globbing no longer considers case.

$ echo *a bigA smalla

Set nullglob

When no files match a pattern, Bash leaves the pattern unexpanded. In this example, there are no files beginning in zzz.

$ ls zzz* ls: zzz*: No such file or directory

Learn More

Project 45 covers Bash shell options and the shopt command in more detail, also showing how to make tabbed completion case insensitive.

This is fine for interactive use, where you probably want to know that no files match, but often not what is wanted in a shell script. The example script below will try to process a nonexistent file called bz*.

First, we try a pattern that does match and expands to a list of filenames. The for loop simply echoes the name of each file to the screen for illustrative purposes.

$ for file in ba*; do echo "Processing file: $file"; done Processing file: baag Processing file: bag Processing file: bags

Next, we try a pattern that is not matched by any file. Bash does not expand the pattern but leaves it verbatim on the command line.

$ for file in bz*; do echo "Processing file: $file"; done Processing file: bz*

We'd prefer bz* to expand to nothing. Perhaps this point isn't obvious to you right now, but when you start writing shell scripts you'll understand that this is not usually what's wanted. (It doesn't matter either way in this simple example, because we are just echoing the name, but in a real scenario we will process the filenames in some way.) To this end, Bash provides the shell option nullglob. Switch on this option by typing

$ shopt -s nullglob

Now rerun the previous example. The for loop no longer tries to process bz* but exits silently, having processed no files.

$ for file in bz*; do echo "Processing file: $file"; done

Now switch the option off again.

$ shopt -u nullglob

One word of warning: Although nullglob works well in this example, it can cause unexpected results for other commands. When used with the ls command, for example, expansion to nothing causes ls to list all files in the current directory. To be safe, turn nullglob on when it is required and off again when it's not required.

The dotglob Option

The star pattern-matching operator is not matched by filenames that begin with a dot (hidden files). This is usually what's wanted, and the dot should be matched explicitly when required. For example:

$ file .*

Change this behavior by setting the shell option dotglob, after which star will be expanded to include hidden files (but not the dot and dot-dot directory entries).

Without dotglob we get

$ echo * baag bag bags bfg bg big blag bug

Turn on dotglob, and we get

$ shopt -s dotglob $ echo * .hidden baag bag bags bfg bg big blag bug

Learn More

Projects 9 and 10 cover basic shell scripting, and the projects in Chapter 9 cover more advanced scripting.

Set GLOBIGNORE

You may tell Bash to ignore particular filenames when expanding filename patterns by setting the shell variable (not option) GLOBIGNORE. It should contain a colon-separated list of filenames and filename patterns. Bash will perform the expansion as normal and then remove any filenames that match any of those in the GLOBIGNORE pattern list.

This is best illustrated by an example. First, let's see which files match.

$ echo * baag bag bag.tmp bags bfg bg big blag bug bug.bak

Now tell Bash to ignore all .tmp and .bak files.

$ GLOBIGNORE="*.tmp:*.bak" $ echo * .hidden baag bag bags bfg bg big blag bug

The .tmp and .bak files are no longer matched. Notice, however, that .hidden shows up (even if dotglob is turned off). This is because setting GLOBIGNORE turns on dotglob. When GLOBIGNORE is set, Bash presumes you'll use it rather than dotglob when you want to ignore hidden files. To ignore hidden files, add .* to the ignore list.

$ GLOBIGNORE=".*:*.tmp:*.bak" $ echo * baag bag bags bfg bg big blag bug

Learn More

Refer to Project 4 to learn about shell variables.

Activate Extended Globbing

The final shell option listed in "Customize Completion" is extglob, which enables extended globbing. Extended globbing lets you list alternative patterns to match, and specify how many occurrences of each pattern to match. The alternative patterns are regular filename patterns and may include any of the usual pattern-matching operators, like star, query, ranges, and classes.

The alternative matching patterns are enclosed in parentheses and separated by a vertical bar (|). One of four operators (@, *, ?, and +) must be placed immediately before the enclosing parentheses to specify how many repetitions of each pattern are necessary to match.

First, let's match exactly one from a number of patterns. The at (@) symbol says to match exactly one of the patterns.

Let's create some test files.

$ touch monmonday monday tuesday $ touch wednesday thursday friday day

Next, enable extended globbing.

$ shopt -s extglob

Finally, write a pattern to match monday and also t followed by anything and then day.

$ ls @(mon|t*)day monday      thursday   tuesday

Another might match monday and also t or u or v or w followed by anything and then day.

$ ls @(mon|[t-w]*)day monday   thursday    tuesday   wednesday

Notice that monmonday does not match. Only one repetition of the pattern is allowed, specified by the @ pattern prefix.

The star prefix allows zero or more repetitions of the pattern to match. Notice that both monmonday and day now match.

$ ls *(mon|t*)day day        monday     monmonday thursday  tuesday

The query prefix allows zero or one repetitions of the pattern to match, so we still see day but not monmonday.

$ ls ?(mon|t*)day day        monday     thursday  tuesday

Finally, the plus prefix allows one or more patterns to match, so we now see monmonday but not day.

$ ls +(mon|t*)day monday      monmonday thursday   tuesday

Note

Although monmonday does not match, a stuttering tttttttttuesday does match, but only because t* is allowing the t to repeat, not the @ prefix.

Here are some more examples in which we assume the following files.

$ echo *.txt a.txt a1.txt a11.txt a111.txt a12.txt a2.txt a21.txt  a22.txt a222.txt a3.txt a33.txt a333.txt $ echo a?(1|2).txt a.txt a1.txt a2.txt $ echo a*(1|2).txt a.txt a1.txt a11.txt a111.txt a12.txt a2.txt a21.txt  a22.txt a222.txt $ echo a+(1|2).txt a1.txt a11.txt a111.txt a12.txt a2.txt a21.txt  a22.txt a222.txt

Finally, the exclamation point matches anything except one of the patterns.

$ echo a!(1|2).txt a.txt a11.txt a111.txt a12.txt a21.txt  a22.txt a222.txt a3.txt a33.txt a333.txt

The easiest way to understand extended globbing is to play with it.

Learn More

Refer to "Bash Initialization" in Project 4 to learn more about the Bash startup sequence.

Make It So

To set shell options and shell variables permanently, place the commands in a Bash startup file. To change settings for all users, put the commands in /etc/bashrc and ensure that /etc/profile sources /etc/bashrc. To change settings for just yourself, put the commands in ~/.bashrc and ensure that ~/.bash_profile sources ~/.bashrc.