# What makes the bash interpreter utility interpret differently standard and smart double quotes, present in a shell script, created in TextEdit?

While learning bash scripting by going through a Beginner's Guide, I tried writing some lines of code in the .sh file, created in TextEdit on macOS Yosemite and running it using the command bash \path\to\script\file\example_script.sh in bash Terminal.

Line of code that I first tested:

echo “The path to my home directory is: $HOME”  Standard output (stdout) in Terminal: “The path to my home directory is: ??  instead of getting: The path to my home directory is: /Users/Ri$hi


Then, I got to know about the curious case of "smart quotes" from a stackexchange, and played with some funny combos, like below:

Line of code that I tested later:

Scenario 1:

echo “The path to my home directory is: $HOME (foo) bar”  stdout: -bash: syntax error near unexpected token ('  Scenario 2: echo "The path to my home directory is:$HOME (foo) bar"


stdout:

The path to my home directory is: /Users/Ri$hi (foo) bar  Scenario 3: echo “The path to my home directory is:$HOME” “(foo)” “bar”


stdout:

-bash: syntax error near unexpected token ('


Scenario 4:

echo "The path to my home directory is: $HOME" “(foo)” “bar”  stdout: -bash: syntax error near unexpected token ('  Scenario 5: echo “The path to my home directory is:$HOME” "(foo)" “bar”


stdout:

“The path to my home directory is: ?? (foo) “bar”


So, I thought why not find out the reason interactively on this forum.

NOTE: Terminal always displays standard double quote " when Shift+" is pressed, but it allows displaying smart double quotes “ ”through +C, +V operation.

• I'll admit that I did not read the entire question, because it does not matter as you cannot use smart quotes in a bash script! Always use proper code formatting and punctuation when scripting! Use ' and or " as appropriate where applicable. I also strongly suggest you never use TextEdit, and instead, use an appropriate code editor, e.g. Sublime Text. – user3439894 Aug 30 '17 at 18:40
• Also, always run your bash script code through ShellCheck before executing it! :) Additionally, when using RegEx, you'll find Regex101 - online regex editor and debugger useful in testing your regular expressions. – user3439894 Aug 30 '17 at 19:27
• You can use TextEdit in a pinch, so long as you set the format to Plain Text. Personally I use BBEdit and TextWrangler. – IconDaemon Aug 30 '17 at 19:33
• @IconDaemon, TextEdit in Plain Text mode will still insert smart quotes when typing plain quotes unless you turn off that feature. While I did suggest Sublime Text, a paid app, nonetheless there are free code editors available that are far better then using TextEdit (under any circumstances IMO). – user3439894 Aug 30 '17 at 19:54
• @user3439894 - Agreed. – IconDaemon Aug 30 '17 at 20:44

There are two things going on here: First, bash recognizes the plain ASCII double-quote, " (character code 0x22) as a double-quote; it does not recognize the fancy unicode left double-quote, “ (unicode U+201C, UTF-8 encoding 0xe2809c) and the corresponding right double-quote, ” (unicode U+201D, UTF-8 encoding 0xe2809d) as anything other than random sequences of bytes (or maybe as random characters, if it's using a UTF-8 locale). This is the fundamental thing to realize: as far as bash is concerned, “ and ” are not actually quotes, they're just things that happen to look like a quotes when they're printed out.

The second complication is that the unicode double-quotes are multibyte characters, so if bash isn't in a UTF-8 locale it may treat some of the bytes differently than others(!)

To see the effect of the first thing, try replacing each occurrence of a double-quote with the string WIBBLE -- another arbitrary sequence that has no special meaning to the shell:

$echo "The path to my home directory is:$HOME bar"
The path to my home directory is: /Users/gordon bar
$echo “The path to my home directory is:$HOME bar”
“The path to my home directory is: /Users/gordon bar”
$echo WIBBLEThe path to my home directory is:$HOME barWIBBLE
WIBBLEThe path to my home directory is: /Users/gordon barWIBBLE


In the first command (with ASCII double-quotes), the quotes are parsed and removed by bash before the argument(s) are passed to the echo command, and hence not printed. In the second and third (with fancy double-quotes and WIBBLE in place of plain quotes), they're just treated as part of the strings to be passed to echo, so echo prints them as part of its output.

$echo "The path to my home directory is:$HOME (foo) bar"
The path to my home directory is: /Users/gordon (foo) bar
$echo “The path to my home directory is:$HOME (foo) bar”
-bash: syntax error near unexpected token ('
$echo WIBBLEThe path to my home directory is:$HOME (foo) barWIBBLE
-bash: syntax error near unexpected token ('


In the second and third commands (with fancy double-quotes and WIBBLE), bash sees parentheses in a non-quoted portion of the command (remember: as far as bash is concerned, fancy quotes are not actually quotes), in a place where they aren't allowed by shell syntax, and therefore complains.

$echo “The path to my home directory is:$HOME”
“The path to my home directory is: ??
$echo WIBBLEThe path to my home directory is:$HOMEWIBBLE
WIBBLEThe path to my home directory is:


Here, something weirder is happening. In the second command, it's looking for a variable named HOMEWIBBLE, not finding it, so replacing it with blank. In the case of the first one, with the fancy double-quotes, it looks to me like it's treating each byte of the UTF-8 encoding of ” as a separate character, treating the first as part of the variable name (again causing the variable not to be found), and then just passing the second and third bytes through, giving an invalid UTF-8 character, which gets printed as ??. Using a hex dump to get a better idea what's going on gives this:

$echo “$HOME”
“??
$echo “$HOME” | xxd -g1
00000000: e2 80 9c 80 9d 0a                                ......


Note that the first “ goes through fine, and shows up in the hex dump as e2 80 9c (the expected UTF-8 encoded fancy double-quote), but after that is just 80 9d -- the first e2 of the second fancy quote got eaten somehow! (BTW, the 0a at the end is a linefeed, marking the end of the output.) To see what's happening, let me define a shell variable as HOME+the first byte of the encoding of ”, and watch what happens:

$eval$'HOME\xe2=foo'
$echo “$HOME”
“foo??
$echo “$HOME” | xxd -g1
00000000: e2 80 9c 66 6f 6f 80 9d 0a                       ...foo...


...So there's what's going on: it's treating the first byte of the double-quote's encoding as part of the variable name, substituting it (if defined), and then just passing through the orphaned second and third bytes, leaving invalid UTF-8. I'm not sure if this is a bash bug, oddity of its parsing, or what.

Anyway, the details are rather messy, but the takeaway should be clear: don't use fancy quotes in your shell scripts; they won't work right. And the same applies to fancy single-quotes and other unicode punctuation marks.

• Nice answer +1, I just didn't want to go to the trouble of explaining why, when the answer was already out on the Internet and one needs to use plain quotes anyway. – user3439894 Aug 30 '17 at 19:57
• @gordon-davisson Also, while running $echo ”$HOME” | xxd -g1 I'm getting 0000000: e2 80 9d 66 6f 6f 80 9d 0a, i.e. 7 zeros initially, instead of 8, in your case. What could be the reason behind it? – Rishi Khanna Sep 1 '17 at 6:29
• I have a new found respect for the power of WIBBLE - this is amazingly useful and comprehensive. +1 and some. – bmike Sep 1 '17 at 14:53
• @RishiKhanna That's weird; the initial zeroes aren't based on the output from echo, they're added by xxd to keep track of where you are (needed when dumping large files). So they shouldn't be related to bash or the fancy quotes... BTW, I'd originally copied that final command wrong (fixed now) -- but the 00000000` part was right. – Gordon Davisson Sep 1 '17 at 14:54