Bash variable expansion behavior when using single or double spaces

I had a problem in a bash backup script caused by the following behavior that I don't understand.

First, we create a parameter holding a string with 2 consecutive spaces (right before '<'):

me@mysystem:~$ TEST="a string with 2 spaces right <--HERE!" me@mysystem:~$ echo $TEST a string with 2 spaces right <--HERE! 

I can grep a space:

me@mysystem:~$ echo ${TEST} | grep " " a string with 2 spaces right <--HERE! 

I can grep a space followed by another letter:

me@mysystem:~$ echo ${TEST} | grep " s" a string with 2 spaces right <--HERE! 

I can grep the part after the 2 spaces:

me@mysystem:~$ echo ${TEST} | grep "HERE" a string with 2 spaces right <--HERE! 

I can do a variable expansion replacing both spaces with an underscore:

me@mysystem:~$ echo ${TEST// /_} a_string_with_2_spaces_right__<--HERE! 

But grepping the double space requires quoting the variable:

me@mysystem:~$ echo $TEST | grep " "; echo "Done." Done. me@mysystem:~$ echo ${TEST} | grep " "; echo "Done." Done. me@mysystem:~$ echo "$TEST" | grep " "; echo "Done." a string with 2 spaces right <--HERE! Done. me@mysystem:~$ echo "${TEST}" | grep " "; echo "Done." a string with 2 spaces right <--HERE! Done. 

I don't understand why this is the case.

Long story short: I had an if statement in the script checking whether ${FILE} represents an existing file, and that apparently also stops interpreting the variable when encountering a space unless quoted. So I'm not sure anymore what ${} does or doesn't actually take care off.

It's best to always quote the variable I guess?

1 Answer

This is a classic case of word splitting, done by shell, on unquoted variable expansion.

Basically when you do, $TEST (without quotes ("$TEST")), the shell:

• Performs word splitting, based on the value of IFS environment variable (default: space, tab, newline), and splits the whole expansion into separate words

• so as you have two consecutive spaces, the word splitting (and also pathname (glob) expansion if any of *, ?, [] is present) is done, and when the output string is rebuilt the words are joined by space, as a result you have only one space between right and <--HERE!. This would be true for any combination of IFS characters.

In a nutshell, quote your variable expansions:

"${variable}"