3.7 Objective 7: Making Use of Regular Expressions In Objective 3, filename globbing with wildcards is described, which enables us to list or find files with common elements (i.e., filenames or file extensions) at once. File globs make use of special characters such as *, which have special meanings in the context of the command line. There are a handful of shell wildcard characters understood by bash, enough to handle the relatively simple problem of globbing filenames. Other problems aren't so simple, and extending the glob concept into any generic text form (files, text streams, program string variables, etc.) can open up a wide new range of capability. This is done using regular expressions. Two tools that are important for the LPIC Level 1 exams and that make use of regular expressions are grep and sed. These tools are useful for text searches. There are many other tools that make use of regular expressions, including the awk, Perl, and Python languages and other utilities, but you don't need to be concerned with them for the purpose of the LPIC Level 1 exams. 3.7.1 Using grep A long time ago, as the idea of regular expressions was catching on, the line editor ed contained a command to display lines of a file being edited that matched a given regular expression. The command is: g/regular expression/p That is, "on a global basis, print the current line when a match for regular expression is found," or more simply, "global regular expression print." This function was so useful that it was made into a standalone utility named, appropriately, grep. Later, the regular expression grammar of grep was expanded in a new command called egrep (for "extended grep"). You'll find both commands on your Linux system today, and they differ slightly in the way they handle regular expressions. For the purposes of Exam 101, we'll stick with grep, which can also make use of the "extended" regular expressions when used with the -E option. You will find some form of grep on just about every Unix or Unix-like system available.
Syntaxgrep [options] regex [files] Description Search files or standard input for lines containing a match to regular expression regex. By default, matching lines will be displayed and nonmatching lines will not be displayed. When multiple files are specified, grep displays the filename as a prefix to the output lines (use the -h option to suppress filename prefixes). Frequently used options - -c
-
Display only a count of matched lines, but not the lines themselves. - -h
-
Display matched lines, but do not include filenames for multiple file input. - -i
-
Ignore uppercase and lowercase distinctions, allowing abc to match both abc and ABC. - -n
-
Display matched lines prefixed with their line numbers. When used with multiple files, both the filename and line number are prefixed. - -v
-
Print all lines that do not match regex. This is an important and useful option. You'll want to use regular expressions, not only to select information but also to eliminate information. Using -v inverts the output this way. Examples Since regular expressions can contain both metacharacters and literals, grep can be used with an entirely literal regex. For example, to find all lines in file1 that contain either Linux or linux, you could use grep like this: $ grep -i linux file1 In this example, the regex is simply "linux." The uppercase L in "Linux" is matched by the command-line option -i. This is fine for literal expressions that are common. However, in situations in which regex includes regular expression metacharacters that are also shell special characters (such as $ or *), the regex must be quoted to prevent shell expansion and pass the metacharacters on to grep. As a simplistic example of this, suppose you have files in your local directory named abc, abc1, and abc2. When combined with bash's echo expression, the abc* wildcard expression lists all files that begin with abc, as follows: $ echo abc* abc abc1 abc2 Now suppose that these files contain lines with the strings abc, abcc, abccc, and so on, and you wish to use grep to find them. You can use the shell wildcard expression abc* to expand to all the abc files as displayed with echo above, and you'd use an identical regular expression abc* to find all occurrences of lines containing abc, abcc, abccc, etc. Without using quotes to prevent shell expansion, the command would be: $ grep abc* abc* After shell expansion, this yields: $ grep abc abc1 abc2 abc abc1 abc2 # no! This is not what you intended! grep would search for the literal expression abc, because it appears as the first command argument. Instead, quote the regular expression with single or double quotes to protect it: $ grep 'abc*' abc* or: $ grep "abc*" abc* After expansion, both examples yield the same results: $ grep abc* abc abc1 abc2 Now this is what you're after. The three files abc, abc1, and abc2 will be searched for the regular expression abc*. It is good to stay in the habit of quoting regular expressions on the command line to avoid these problems -- they won't be at all obvious because the shell expansion is invisible to you unless you use the echo command. | The use of grep and its options is common. You should be familiar with what each option does, as well as the concept of piping the results of other commands into grep for matching. | 3.7.2 Using sed In Objective 2, we introduce sed, the stream editor. In that section, we talk about how sed uses addresses to locate text upon which it will operate. Among the addressing mechanisms mentioned is the use of regular expressions delimited between slash characters. Let's recap how sed can be invoked.
Syntaxsed [options] 'command1' [files] sed [options] -e 'command1' [-e 'command2'] [files] sed [options] -f script [files] Description Note that command1 is contained within single quotes. This is necessary for the same reasons as with grep. The text in command1 must be protected from evaluation and expansion by the shell. The address part of a sed command may contain regular expressions, which are enclosed in slashes. For example, to show the contents of file1 except for blank lines, the sed delete (d) command could be invoked like this: $ sed '/^$/ d' file1 In this case, the regular expression ^$ matches blank lines and the d command removes those matching lines from sed's output. 3.7.2.1 Quoting As shown in the examples for grep and sed, it is necessary to quote regular expression metacharacters if you wish to preserve their special meaning. Failing to do this can lead to unexpected results when the shell interprets the metacharacters as file globbing characters. There are three forms of quoting you may use to preserve special characters: - \ (an unquoted backslash character)
-
By applying a backslash before a special character, it will not be interpreted by the shell but will be passed through unaltered to the command you're entering. For example, the * metacharacter may be used in a regular expression like this: $ grep abc\* abc abc1 abc2 Here, files abc, abc1, and abc2 are searched for the regular expression abc*. - Single quotes
-
Surrounding metacharacters with the single-quote character also protects them from interpretation by the shell. All characters inside a pair of single quotes are assumed to have their literal value. - Double quotes
-
Surrounding metacharacters with the double-quote character has the same effect as single quotes, with the exception of the $, ' (single quote), and \ (backslash) characters. Both $ and ' retain their special meaning within double quotes. The backslash retains its special meaning when followed by $, ', another backslash, or a newline. In general, single quotes are safest for preserving regular expressions. | Pay special attention to quoting methods used to preserve special characters, because the various forms don't necessarily yield the same result. | 3.7.3 Regular Expressions Linux offers many tools for system administrators to use for processing text. Many, such as sed and the awk and Perl languages, are capable of automatically editing multiple files, providing you with a wide range of text-processing capability. To harness that capability, you need to be able to define and delineate specific text segments from within files, text streams, and string variables. Once the text you're after is identified, you can use one of these tools or languages to do useful things to it. These tools and others understand a loosely defined pattern language. The language and the patterns themselves are collectively called regular expressions (often abbreviated just regexp or regex). While regular expressions are similar in concept to file globs, many more special characters exist for regular expressions, extending the utility and capability of tools that understand them. Regular expressions are the topic of entire books (such as Jeffrey E. F. Friedl's excellent and very readable Mastering Regular Expressions, published by O'Reilly & Associates). Exam 101 requires the use of simple regular expressions and related tools, specifically to perform searches from text sources. This section covers only the basics of regular expressions, but it goes without saying that their power warrants a full understanding. Digging deeper into the regular expression world is highly recommended when you have the chance. 3.7.3.1 Regular expression syntax It would not be unreasonable to assume that some specification defines how regular expressions are constructed. Unfortunately, there isn't one. Regular expressions have been incorporated as a feature in a number of tools over the years, with varying degrees of consistency and completeness. The result is a cart-before-the-horse scenario, in which utilities and languages have defined their own flavor of regular expression syntax, each with its own extensions and idiosyncrasies. Formally defining the regular expression syntax came later, as did efforts to make it more consistent. Regular expressions are defined by arranging strings of text, or patterns. Those patterns are composed of two types of characters: - Metacharacters
-
Like the special file globbing characters, regular expression metacharacters take on a special meaning in the context of the tool in which they're used. There are a few metacharacters that are generally thought of to be among the "extended set" of metacharacters, specifically those introduced into egrep after grep was created. Now, most of those can also be handled by grep using the -E option. Examples of metacharacters include the ^ symbol, which means "the beginning of a line," and the $ symbol, which means "the end of a line." A complete listing of metacharacters follows in Table 3-6, Table 3-7, and Table 3-8. - Literals
-
Everything that is not a metacharacter is just plain text, or literal text. It is often helpful to consider regular expressions as their own language, where literal text acts as words and phrases. The "grammar" of the language is defined by the use of metacharacters. The two are combined according to specific rules (which, as mentioned earlier, may differ slightly among various tools) to communicate ideas and get real work done. When you construct regular expressions, you use metacharacters and literals to specify three basic ideas about your input text: - Position anchors
-
A position anchor is used to specify the position of one or more character sets in relation to the entire line of text (such as the beginning of a line). - Character sets
-
A character set matches text. It could be a series of literals, metacharacters that match individual or multiple characters, or combinations of these. - Quantity modifiers
-
Quantity modifiers follow a character set and indicate the number of times the set should be repeated. These characters "give elasticity" to a regular expression by allowing the matches to have variable length. The next section lists commonly used metacharacters. The examples given with the metacharacters are very basic, intended just to demonstrate the use of the metacharacter in question. More involved regular expressions are covered later. 3.7.4 Regular Expression Examples Now that the gory details are out of the way, here are some examples of simple regular expression usage that you may find useful. 3.7.4.1 Anchors Anchors are used to describe position information. Table 3-6 lists anchor characters. Table 3-6. Regular Expression Position Anchors ^ | Match at the beginning of a line. This interpretation makes sense only when the ^ character is at the lefthand side of the regex. | $ | Match at the end of a line. This interpretation makes sense only when the $ character is at the righthand side of the regex. |
Example 1 Display all lines from file1 where the string "Linux" appears at the start of the line: $ grep '^Linux' file1 Example 2 Display lines in file1 where the last character is an "x": $ grep 'x$' file1 Display the number of empty lines in file1 by finding lines with nothing between the beginning and the end: $ grep -c '^$' file1 Display all lines from file1 containing only the word "null" by itself: $ grep '^null$' file1 3.7.4.2 Groups and ranges Characters can be placed into groups and ranges to make regular expressions more efficient, as shown in Table 3-7. Table 3-7. Regular Expression Character Sets [abc] [a-z] | Single-character groups and ranges. In the first form, match any single character from among the enclosed characters a, b, or c. In the second form, match any single character from among the range of characters bounded by a and z. The brackets are for grouping only and are not matched themselves. | | [^abc] [^a-z] | Inverse match. Match any single character not among the enclosed characters a, b, and c or in the range a-z. Be careful not to confuse this inversion with the anchor character ^, described earlier. | | \<word\> | Match words. Words are essentially defined as being character sets surrounded by whitespace and adjacent to the start of line, the end of line, or punctuation marks. The backslashes are required and enable this interpretation of < and >. | | . (the single dot) | Match any single character except a newline. | | \ | As mentioned in the section on quoting earlier, turn off (escape) the special meaning of the character that follows, turning metacharacters in to literals. |
Example 1 Display all lines from file1 containing either "Linux," "linux," "TurboLinux," and so on: $ grep '[Ll]inux' file1 Example 2 Display all lines from file1 which contain three adjacent digits: $ grep '[0-9][0-9][0-9]' file1 Example 3 Display all lines from file1 beginning with any single character other than a digit: $ grep '^[^0-9]' file1 Example 4 Display all lines from file1 that contain the whole word "Linux" or "linux," but not "LinuxOS" or "TurboLinux": $ grep '\<[Ll]inux\>' file1 Example 5 Display all lines from file1 with five or more characters on a line (excluding the newline character): $ grep '.....' file1 Example 6 Display all nonblank lines from file1 (i.e., that have at least one character): $ grep '.' file1 Example 7 Display all lines from file1 that contain a period (normally a metacharacter) using escape: $ grep '\.' file1 3.7.4.3 Modifiers Modifiers change the meaning of other characters in a regular expression. Table 3-8 lists these modifiers. Table 3-8. Regular Expression Modifiers * | Match an unknown number (zero or more) of the single character (or single-character regex) that precedes it. | ? | Match zero or one instance of the preceding regex. This modifier is an "extended" feature and available in grep only when the -E command-line option is used. | + | Match one or more instances of the preceding regex. This modifier is an "extended" feature and available in grep only when the -E command-line option is used. | | \{n,m\} | Match a range of occurrences of the single character or regex that precedes this construct. \{n\} matches n occurrences,\{n,\} matches at least n occurrences, and \{n,m\} matches any number of occurrences between n and m, inclusively. The backslashes are required and enable this interpretation of{ and }. | | | Alternation. Match either the regex specified before or after the vertical bar. This modifier is an "extended" feature and available in grep only when the -E command-line option is used. |
Example 1 Display all lines from file1 that contain "ab," "abc," "abcc," "abccc," and so on: $ grep 'abc*' file1 Example 2 Display all lines from file1 that contain "abc," "abcc," "abccc," and so on, but not "ab": $ grep 'abcc*' file1 Example 3 Display all lines from file1 that contain two or more adjacent digits: $ grep '[0-9][0-9][0-9]*' file1 Example 4 Display lines from file1 that contain "file" (because ? can match zero occurrences), file1, or file2: $ grep -E 'file[12]?' file1 Example 5 Display all lines from file1 containing at least one digit: $ grep -E '[0-9]+' file1 Example 6 Display all lines from file1 that contain "111," "1111," or "11111" on a line by itself: $ grep '^1\{3,5\}$' file1 Example 7 Display all lines from file1 that contain any three-, four-, or five-digit number: $ grep '\<[0-9]\{3,5\}\>' file1 Example 8 Display all lines from file1 that contain "Happy," "happy," "Sad," "sad," "Angry," or "angry": $ grep -E '[Hh]appy|[Ss]ad|[Aa]ngry' file1 3.7.4.4 Basic regular expression patterns
Example 1 Match any letter: [A-Za-z] Example 2 Match any symbol (not a letter or digit): [^0-9A-Za-z] Example 3 Match an uppercase letter, followed by zero or more lowercase letters: [A-Z][a-z]* Example 4 Match a U.S. Social Security Number (123-45-6789) by specifying groups of three, two, and four digits separated by dashes: [0-9]\{3\}-[0-9]\{2\}-[0-9]\{4\} Example 5 Match a dollar amount, using an escaped dollar sign, zero or more spaces or digits, an escaped period, and two more digits: \$[ 0-9]*\.[0-9]\{2\} Example 6 Match the month of June and its abbreviation, "Jun." The question mark matches zero or one instance of the e : June? 3.7.4.5 Using regular expressions as addresses in sed These examples are commands you would issue to sed. For example, the commands could take the place of command1 in this usage: $ sed [options] 'command1' [files] These commands could also appear in a standalone sed script.
Example 1 Delete blank lines: /^$/d Example 2 Delete any line that doesn't contain #keepme:: /#keepme/!d Example 3 Delete lines containing only whitespace (spaces or tabs). In this example, tab means the single tab character and is preceded by a single space: /^[ tab]*$/d Example 4 Delete lines beginning with periods or pound signs: /^[.#]/d Example 5 Substitute a single space for any number of spaces wherever they occur on the line: s/ */ /g Example 6 Substitute def for abc from line 11 to 20, wherever it occurs on the line: 11,20s/abc/@@@/g Example 7 Translate the characters a, b, and c to the @ character from line 11 to 20, wherever they occur on the line: 11,20y/abc/@@@/ | Make certain you are clear about the difference between file globbing and the use of regular expressions. |  | 