Project80.Understand Shell Quoting

Project 80. Understand Shell Quoting

"How do I selectively turn off the shell's interpretation and expansion of special characters?"

This project explores the art of quoting in the Bash shell. It shows how we force Bash to interpret characters literally in situations where they normally would be considered special characters.

Recognize Special Characters

The Bash shell expands a command line before the command line is executed. During the expansion phase, all special characterssuch as wildcards, redirection symbols, and the dollar symbol used in variable expansionare interpreted and replaced by their expansion text. To invoke a command and pass it text that includes any of those characters used in their literal senses (as in the strings M*A*S*H and $64,000 Question), the special characters must be quoted or escaped to prevent interpretation.

Before we can employ quoting, we need to know which characters must be quoted. Table 9.1 is a handy reference listing all the special characters, and character combinations, that the shell is likely to interpret.

Table 9.1. Shell Special Characters

Symbol	Expansion or Interpretation
#	Introduce a comment
;	Separate commands
{...} (...)	Introduce a command block and subshell
&& ||	Logical AND and OR operators (placed between commands)
~	Home directory
/	Directory or filename separator
$var	Variable expansion
`...` $(...)	Execute a command and substitute the output
$((...)) ((...))	Evaluate an integer expression and condition
' " \	Strong quote, weak quote, escape next character
* [...] ?	Globbing
&	Background execution
< > | !	Redirection and pipelining
!	History expansion

Quote and Escape

Suppose that you want to echo the text

I want $lots

If no quoting is used, $lots will be taken as an instruction to expand the variable lots (which is currently unset). The result would be as follows.

$ echo I want $lots I want

To prevent the dollar special character from being interpreted, escape it in one of three ways. First, precede it with a backslash.

$ echo I want \$lots I want $lots

Second, enclose the entire string in single quotes, which are also called strong quotes because no special characters within them are interpreted.

$ echo 'I want $lots' I want $lots

Third, use double quotes, also called weak quotes because most, but not all, special characters they enclose are escaped. The exceptions are

• The dollar symbol in all three forms: $var, $(...), and $((...))

• The ! symbol in history expansion

In this example, we cannot use double quotes.

A First Escape Trick

Suppose that you want to echo a line such as

I want $1000000 (a lot of $)

We'll assume that the number of dollars is not fixed but is held in the variable lots. To illustrate the different forms of quoting, we'll examine what happens when each form is employed, starting with none.

$ echo I want $$lots (a lot of $) -bash: syntax error near unexpected token `('

Employing single quotes prevents all forms of expansion.

$ echo 'I want $$lots (a lot of $)' I want $$lots (a lot of $)

To achieve the intended result, we must employ double quotes, which prevent the parentheses from being interpreted but allow expansion of variable $lots.

$ echo "I want $$lots (a lot of $)" I want 2324lots (a lot of $)

Closer, but this didn't quite work. It's still necessary to escape the first dollar symbol. If we don't, it attaches itself to the second dollar symbol and causes the shell to expand the special variable $$.

$ echo "I want \$$lots (a lot of $)" I want $1000000 (a lot of $)

A More Daring Escape

Let's look at a trickier example. How might we quote this?

$ echo $5 - That's ok

We cannot use double quotes, because we don't want to interpret $5. Single quotes won't work either, because the text itself contains a single quote acting as an apostrophe. A first attempt might have us escaping the apostrophe.

$ echo '$5 - That\'s ok' > Control-c

Tip

Don't be tempted to skip quoting because a command appears to work correctly. If your command attempts to pass the text note.* to grep unquoted, for example, and no matching filenames exist in the current directory, the shell will not expand it. Your unquoted command will workuntil the day you create a file with a name such as note.1.

This fails because inside single quotes, no special characters are interpreted, including backslash. Hence, Bash sees the apostrophe as the closing quote and the last single quote as an unterminated open quote.

The simplest method involves converting the expression to two strings enclosed in single quotes, with the (unenclosed) apostrophe between them. Then we escape the apostrophe by using either a backslash or double quotes, as shown in the next two examples.

$ echo '$5 - That'\''s ok' $5 - That's ok $ echo '$5 - That'"'"'s ok' $5 - That's ok

We could also use the following technique where two quoted parts are run consecutively.

$ echo '$5'" - That's ok" $5 - That's ok

Consecutive Quotes

Suppose that we use the awk command to filter field number 4 (written as $4 in awk scripting) from the output of a ps command. We type the following, employing single quotes to escape $4 from the shell because we want it to be interpreted by awk.

$ ps xc | awk '{print $4}'

Suppose now that we want to do the same thing, but using the field number stored in a shell variable called field.

$ field=4 $ ps xc | awk '{print $$field}'

This won't work, of course. So how do we both allow Bash to expand $field to 4 and escape the first dollar so we pass, literally, $4 to awk? In this simple example, there are several ways, but you can apply a general solution to almost all quoting problems of this nature. It may seem trivial now, but remember it for the future; I've seen many people completely stumped trying to solve quoting dilemmas that are amenable to this particular solution.

We simply start and stop quoted regions as necessary. The first quoted region is '{print $'; the second is '}'. $field is not quoted and, therefore, is expanded by the shell.

$ ps xc | awk '{print $'$field'}'

Although not necessary in this example, the general rule would have quoted each region to prevent problems with spaces in expanded parameters.

$ ps xc | awk '{print $'"$field"'}'

Ensure that you don't include spaces between the quoted regions.

Multi-Level Quoting

Let's write a command that uses grep to search a file for the sequence a*. Here's our test file.

$ cat file This line contains a* This line does not

Learn More

Project 23 shows how to use the grep command.

Because star is a special character in regular expressions, we must escape it, passing \* to grep. Star and backslash are also special characters to the shell and must be escaped from it too, as \\ and \*.

Learn More

Projects 39 and 40 explore the ps command in detail.

Therefore, we form the following command.

$ grep a\\\* file This line contains a*

Quoting within Command Evaluation

Here's a tip that might save much head-scratching. Suppose that we have a command substitution such as

$(ps xc | grep "$target")

The variable $target may expand to include spaces, so we must double-quote it for the grep command to work correctly. If we then use the command substitution as a parameter to another command, we must enclose the whole substitution in double quotes. A naive attempt has us type the following.

$ grep "$(ps xc | grep "$target")" processes.txt

This shouldn't work, because as we have seen in previous examples, the expression forms two quoted regions: "$(ps xc | grep " and ")". Surprisingly, it does work, because Bash processes a command substitution ($(...)) as an independent syntactical element. It processes $(ps xc | grep "$target") and then considers the outer expression grep "..." processes.txt.