Issues with Unicode symbols in some applications running in Terminal.app and iTerm2

Question

Darwin DeoGloria 13.3.0 Darwin Kernel Version 13.3.0: Tue Jun  3 21:27:35 PDT 2014; root:xnu-2422.110.17~1/RELEASE_X86_64 x86_64

Fresh user account. Terminal.app and iTerm2 with UTF-8 set. Accented characters work fine in all applications ("å,ä,ö,é"). But some (wide?) unicode chars is not showing in some command line applications. I first discovered it in weechat, but in the bash shell it works fine.

In stock Mavericks bash:

DeoGloria:~ test$ echo -e "\xE2\x9A\x91"
⚑

Copying and pasting the flag char with mouse.

DeoGloria:~ test$ ⚑
-bash: ⚑: command not found

bash is working as expected. Starting default zsh, no config done

DeoGloria:~ test$ zsh
DeoGloria:~ test$ echo -e "\xE2\x9A\x91"
⚑

Showing alright. But not when I paste:

DeoGloria% <2691>
zsh: command not found: \u2691

Locale settings:

DeoGloria% locale
LANG="sv_SE.UTF-8"
LC_COLLATE="sv_SE.UTF-8"
LC_CTYPE="sv_SE.UTF-8"
LC_MESSAGES="sv_SE.UTF-8"
LC_MONETARY="sv_SE.UTF-8"
LC_NUMERIC="sv_SE.UTF-8"
LC_TIME="sv_SE.UTF-8"
LC_ALL=

Starting emacs or vi from zsh and pasting "⚑" works fine. It doesn't work in nano. I get an non-visual character.

I suspected some issues with ncurses, but bash, zsh, nano, emacs are all linked with

/usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)

I'm out of ideas. Any pointers?

Answer 1

Interesting question! I did some tests on my side.

zsh

xXx:~/ $ <2691>                                                                                                                                                                                              
zsh: command not found: \u2691
xXx:~/ $🇧🇸                                                                                                                                                                                                                                                                                         
zsh: command not found: 🇧🇸
xXx:~/ $ ℃                                                                                                                                                                                                                                                                                                                                                 
zsh: command not found: ℃

I did also several tests with other shells and results are not the same:

sh

sh-3.2$ ⚑
sh: ⚑: command not found

bash

Ok, that should be the same as sh:

bash-3.2$ ⚑
bash: ⚑: command not found

csh/tcsh

[LeBouc-xXx:~] xXx% \U+2691
⚑: Command not found.

---

Apparently, all that shells are not set with the same input/output options. I tried with xterm/terminal and iTerm2 to see if I could be a software dependant problem. That's the same everywhere.

I don't have the time to search/read that but you might find some useful informations about the zsh line editor here.

If you find it, keep me up-to-date, I would be pleased to have the final answer to your question.