1

When I do in my terminal a, for instance,

man tr

I can see that some of the texts are rendered in different colours, based on the control characters which man produces in its output, while some are not.

For instance, the terminal displays

NAME
 tr <E2><80><93> translate characters

from which I conclude, that the hex sequence E28093 is not interpreted. My guess is that this this is a certain unicode character sequence (maybe for a long dash), but I don't understand why it is not shown.

Here is a screenshot screenshot of my terminal.

My TERM variable is set to xterm-256color and the LANG environment variable is set to en_GB.utf8 and I don't have neither a PAGER nor a MANPAGER variable defined, which means that man pipes to less.

UPDATE : However, I have defined the variable LESS, which provides default options for less. It's value is -A --quit-if-one-screen --LONG-PROMPT -X -R; but I don't see, how this could explain the behaviour. Furthermore, even when I explicitly use a different pager, i.e.

MANPAGER=more man tr

the output is the same.

Unicode characters seem to be displayed fine. For instance, if I do in the command line (which is zsh) a

echo い

I see as output an .

This happens with both Terminal.app and iTerm2. The font being used is Monaco,

BTW, on request by user mmmmmm, I encluse here a screenshot of the output produced by man -d tr:

screenshot

UPDATE

I did a

man tr | tee tmp/man_tr.txt
xxd tmp/mat_tr.txt

to see the exact hex code produced by tr. The first couple of lines look like this:

00000000: 5452 2831 2920 2020 2020 2020 2020 2020  TR(1)           
00000010: 2020 2020 2020 2020 2020 2020 2047 656e               Gen
00000020: 6572 616c 2043 6f6d 6d61 6e64 7320 4d61  eral Commands Ma
00000030: 6e75 616c 2020 2020 2020 2020 2020 2020  nual            
00000040: 2020 2020 2020 2020 2020 2054 5228 3129             TR(1)
00000050: 0a0a 4e08 4e41 0841 4d08 4d45 0845 0a20  ..N.NA.AM.ME.E. 
00000060: 2020 2020 7408 7472 0872 20e2 8093 2074      t.tr.r ... t
00000070: 7261 6e73 6c61 7465 2063 6861 7261 6374  ranslate charact
00000080: 6572 730a 0a53 0853 5908 594e 084e 4f08  ers..S.SY.YN.NO.
00000090: 4f50 0850 5308 5349 0849 5308 530a 2020  OP.PS.SI.IS.S.  
000000a0: 2020 2074 0874 7208 7220 5b2d 082d 4308     t.tr.r [-.-C.
000000b0: 4363 0863 7308 7375 0875 5d20 5f08 735f  Cc.cs.su.u] _.s_

We can see the ominous byte sequence e28093 occuring in the line at offset 60. What on earth is this? It can't be a UTF-8 sequence (there are no 3-byte-sequences starting with E2).

Furthermore these ghost codes don't appear on every man page. I see it with i.e. man cat, but they don't show with man man, man zsh or man bash..... But there is a system in it:

I have installed via Homebrew the Gnu tools for the Mac. For installing them, there are two options: (1) Have them replace the original files, i.e. GNU-tr replaces the MacOS-tr, or (2) leave the original binaries unchanged and install the GNU tools under a new name, i.e. the GNU-tr is now named /usr/local/bin/gtr, while the original tr is still available as /usr/bin/tr. I have decided to go for the second option, i.e. have the GNU-versions along with the BSD-versions, but in different directories. Since their name differs, there should be no problem with PATH either. BTW, I have not defined the variable MANPATH either, but again this should not matter since the programs are named differently (tr vs. gtr).

Now when I do a

man gtr

I see the GNU tr manpage, formatted nicely; but when I just do a

man tr

I see the BSD tr manpage, with the E2-hex-sequence in it. At least this shows, under what condition it happens - the whole mess is related to the GNU utilities -, although I still don't understand, WHY it happens.

11
  • 1
    Do you have an environment variable MANPAGER and if so what is its value?
    – mmmmmm
    May 22 at 10:24
  • 1
    What does man -d tr show as the command?
    – mmmmmm
    May 22 at 11:05
  • @mmmmmm : No MANPAGER, but a variable LESS; see my update to my question. May 22 at 11:33
  • Unfortunately I can't read that last image could you please paste the relevant line - the last one as text [ and also what does which man show? and the version of macOS
    – mmmmmm
    May 22 at 12:06
  • The last one says: _ man: not executing command: /usr/bin/mandoc -W unsupp,stop /usr/share/man/man1/tr.1 2>&1 >/dev/null_ May 22 at 12:44

2 Answers 2

3

The "mysterious hex sequence" is actually just the UTF-8 bytes for an en dash - in hex that is 0xE28093.

The en dash is supposed to be there - it's in the man page. The output of those bytes (0xE28093) from man is also as it is supposed to be on a standard installation.

The problem on your installation could be with the terminal software or its configuration. When using Terminal.app, check its Settings and make sure that under the Profiles tab in the Advanced sub-tab, below the International header, it should say "Text encoding: Unicode (UTF-8)".

According to your comments, you have already that the same bytes just output using cat results in the correct dash character.

Therefore the problem is most probably with your pageer settings. You could try removing -X and -R from the LESS environment variable. Just set the variable to empty and try the man command to see if there's an effect.

8
  • May I ask how you identified this sequence as en-dash? Initially, I also suspected that this is some kind of dash, but when I enter this sequence in this search engine, it did not find anything unusual. May 22 at 14:17
  • Of course, the terminal settings (for instance the font set) is a potential suspect. However, after saving the man page to a text file and verifying that it still contains E28093, I can do a cat of this text file and the output looks fine. Now that you found out that it describes an en-dash, I can indeed see, when looking closely, that the dash at this position looks slightly different (tr – translate characters). This proves at least that the terminal does not have a problem printing this character. This means that the pager is the culprit. May 22 at 14:20
  • I also tried to unset the variable LESS, but no effect. Plus, it also doesn't explain, why using the pager more does not show the problem. May 22 at 14:23
  • You're using a Unicode search engine. In UTF-8, some characters are encoded using multi-byte sequences. The 0xE2 is the start of such a multi-byte sequence.
    – jksoegaard
    May 22 at 14:23
  • 1
    The settings might be odd, but it does not explain, why we have the same effect with different settings or a different pager....... May 22 at 14:32
2

I just found (with the valuable help of the comments by mmmmmm and jksoergaard) that my pagers (less and more) have problems displaying UTF8 multibyte characters, and that this is unrelated to man. Therefore I think we can regard this question as closed. However, I will research the issue of multibyte characters and pagers (maybe they simply don't work together), and post a new question on this topic, if necessary.

At least for less, an explanation seems to be here. It does not mention more, but I can man the man-page work with less, if I set

export LESSCHARSET=utf-8

UPDATE : As pointed out by user mmmmmm, /usr/bin/more and /usr/bin/less point to the same inode, which explains why more also behaves in this way.

2
  • In macOS the Apple more and less are the same (they are hard linked together)
    – mmmmmm
    May 24 at 8:50
  • @mmmmmm : Indeed, they have the same inode number! I will put this into my answer. Thanks a lot! May 24 at 11:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .