When looking at man page, some characters produced by `man` are not rendered

Question

When I do in my terminal a, for instance,

man tr

I can see that some of the texts are rendered in different colours, based on the control characters which man produces in its output, while some are not.

For instance, the terminal displays

NAME
 tr <E2><80><93> translate characters

from which I conclude, that the hex sequence E28093 is not interpreted. My guess is that this this is a certain unicode character sequence (maybe for a long dash), but I don't understand why it is not shown.

Here is a screenshot of my terminal.

My TERM variable is set to xterm-256color and the LANG environment variable is set to en_GB.utf8 and I don't have neither a PAGER nor a MANPAGER variable defined, which means that man pipes to less.

UPDATE : However, I have defined the variable LESS, which provides default options for less. It's value is -A --quit-if-one-screen --LONG-PROMPT -X -R; but I don't see, how this could explain the behaviour. Furthermore, even when I explicitly use a different pager, i.e.

MANPAGER=more man tr

the output is the same.

Unicode characters seem to be displayed fine. For instance, if I do in the command line (which is zsh) a

echo い

I see as output an い.

This happens with both Terminal.app and iTerm2. The font being used is Monaco,

BTW, on request by user mmmmmm, I encluse here a screenshot of the output produced by man -d tr:

UPDATE

I did a

man tr | tee tmp/man_tr.txt
xxd tmp/mat_tr.txt

to see the exact hex code produced by tr. The first couple of lines look like this:

00000000: 5452 2831 2920 2020 2020 2020 2020 2020  TR(1)           
00000010: 2020 2020 2020 2020 2020 2020 2047 656e               Gen
00000020: 6572 616c 2043 6f6d 6d61 6e64 7320 4d61  eral Commands Ma
00000030: 6e75 616c 2020 2020 2020 2020 2020 2020  nual            
00000040: 2020 2020 2020 2020 2020 2054 5228 3129             TR(1)
00000050: 0a0a 4e08 4e41 0841 4d08 4d45 0845 0a20  ..N.NA.AM.ME.E. 
00000060: 2020 2020 7408 7472 0872 20e2 8093 2074      t.tr.r ... t
00000070: 7261 6e73 6c61 7465 2063 6861 7261 6374  ranslate charact
00000080: 6572 730a 0a53 0853 5908 594e 084e 4f08  ers..S.SY.YN.NO.
00000090: 4f50 0850 5308 5349 0849 5308 530a 2020  OP.PS.SI.IS.S.  
000000a0: 2020 2074 0874 7208 7220 5b2d 082d 4308     t.tr.r [-.-C.
000000b0: 4363 0863 7308 7375 0875 5d20 5f08 735f  Cc.cs.su.u] _.s_

We can see the ominous byte sequence e28093 occuring in the line at offset 60. What on earth is this? It can't be a UTF-8 sequence (there are no 3-byte-sequences starting with E2).

Furthermore these ghost codes don't appear on every man page. I see it with i.e. man cat, but they don't show with man man, man zsh or man bash..... But there is a system in it:

I have installed via Homebrew the Gnu tools for the Mac. For installing them, there are two options: (1) Have them replace the original files, i.e. GNU-tr replaces the MacOS-tr, or (2) leave the original binaries unchanged and install the GNU tools under a new name, i.e. the GNU-tr is now named /usr/local/bin/gtr, while the original tr is still available as /usr/bin/tr. I have decided to go for the second option, i.e. have the GNU-versions along with the BSD-versions, but in different directories. Since their name differs, there should be no problem with PATH either. BTW, I have not defined the variable MANPATH either, but again this should not matter since the programs are named differently (tr vs. gtr).

Now when I do a

man gtr

I see the GNU tr manpage, formatted nicely; but when I just do a

man tr

I see the BSD tr manpage, with the E2-hex-sequence in it. At least this shows, under what condition it happens - the whole mess is related to the GNU utilities -, although I still don't understand, WHY it happens.

Answer 1

The "mysterious hex sequence" is actually just the UTF-8 bytes for an en dash - in hex that is 0xE28093.

The en dash is supposed to be there - it's in the man page. The output of those bytes (0xE28093) from man is also as it is supposed to be on a standard installation.

The problem on your installation could be with the terminal software or its configuration. When using Terminal.app, check its Settings and make sure that under the Profiles tab in the Advanced sub-tab, below the International header, it should say "Text encoding: Unicode (UTF-8)".

According to your comments, you have already that the same bytes just output using cat results in the correct dash character.

Therefore the problem is most probably with your pageer settings. You could try removing -X and -R from the LESS environment variable. Just set the variable to empty and try the man command to see if there's an effect.

Answer 2

I just found (with the valuable help of the comments by mmmmmm and jksoergaard) that my pagers (less and more) have problems displaying UTF8 multibyte characters, and that this is unrelated to man. Therefore I think we can regard this question as closed. However, I will research the issue of multibyte characters and pagers (maybe they simply don't work together), and post a new question on this topic, if necessary.

At least for less, an explanation seems to be here. It does not mention more, but I can man the man-page work with less, if I set

export LESSCHARSET=utf-8

UPDATE : As pointed out by user mmmmmm, /usr/bin/more and /usr/bin/less point to the same inode, which explains why more also behaves in this way.