man command, and output decoration (bold/underline/...)

Question

When I do, say, a

man zsh

I get the zsh man page, parts of the text are in bold or underline, which makes it for easier reading. man is doing this for instance by inserting 0x08 control codes in its output, which then is interpreted by the Terminal for rendering underlined text.

When I do a

man zsh | cat

I of course don't see such decorations anymore, but just the plain text. This makes sense, because the program reading from the pipe, is usually not interested in seeing those control codes.

I assume that man checks whether the stdout is a tty or a pipe/file, an in the latter case it does not produce the output.

Now comes the puzzling part:

When I do a

man zsh | less
man zsh | xxd
man zsh | awk ....

I do still see the decorations (underline etc.) at the receiving end of the pipe. How is this possilble? man does not know whether the process it pipes the output to, wants to see the control code, and cat and head certainly don't have the ability to filter out such decorations.

Answer 1

The theory

The man program itself, doesn't talk directly to the terminal. It produces output which is text with a small amount of formatting. Another program, a pager, generally less, handles the user interaction (scrolling, searching, etc.) and applies some basic formatting.

In addition to basic pagination (breaking lines, indenting, etc.), man can use three ways to make text stand out: capital letters, bold and underline. Capital letters are just different characters from lowercase letters. For bold and underline, man uses the backspace (BS) control character:

• The three-character sequence c BS c indicates that the character c should be displayed in bold.
• The three-character sequence _ BS c (where _ is the underscore character) indicates that the character c should be displayed underlined.

(There is no provision for bold underscore, and I don't know offhand which way _ BS _ would be interpreted.)

These sequences are inspired from what you could do with a typewriter, and in particular with a teletype which was a common interface to computers when the man program was originally written: if you print a character, then use the BS control to move back left and print that character again, you get a bolder character. If you use BS to overstrike a _ with another character, that other character is underlined.

Character-oriented video terminals don't overstrike characters in this way, and modern terminal emulator programs present a similar interface. Terminals interpret the BS character as an instruction to move left by one position. When displaying a character at a position, the terminal completely overwrites whatever was previously there.

So it's the job of the pager to interpret convert c BS c and _ BS c to instructions for the terminal to display c in bold or underline. The way to do this depends on the terminal capabilities, although in practice all modern terminals use the same escape sequences for basic functionality like bold and underline.

The practice

When you run man zsh, the man command invokes less. less converts BS-based bold/underline into formatting escape sequences that the terminal understands. So you see the text formatted as intended.

When you run man zsh | cat, the man program detects that its output is not a terminal, so it doesn't run less: it just prints the output that it would have passed to less. cat just passes its input through unchanged. The terminal sees sequences like c BS c and _ BS c, and in either case, BS erases the previous character so the visual effect is just like a plain c.

When you run man zsh | less or man zsh | cat | less, the less command receives the same input that it would receive if man had invoked less directly. However, the behavior may be different because man can invoke less with some non-default command options. (From a quick check, this doesn't appear to happen on macOS.)

You can disable less's handling of backspace by passing the -u command line option. Run either LESS=-u man man or man man | less -u to see what happens if less just passes the BS character unchanged to the terminal: the visual effect is the same as when not running less (but less still does pagination). Run LESS=-U man man or man man | less -U to see a visual rendering of the BS character as^H in reverse video.

Other ways to see the BS characters include:

• man man | cat -v which translates each BS character into the two-character sequence ^H.
• man man | hd which shows a hex dump of the text. You'll see sequences like 4e 08 4e 41 08 41 4d 08 4d 45 08 — 08=backspace, 4e=N, 41=A, 4d=M, 45=E so this is NAME in bold.
• man man >out.txt and open out.txt in an editor. How the editor shows control characters depends on the editor and possibly on its configuration.

If you want to print your own, you can pass \b or \010 in the first argument of the printf terminal command to print a backspace character. Try these:

printf 'Normal\nB\bBo\bol\bld\bd\n_\bU_\bn_\bd_\be_\br_\bl_\bi_\bn_\be\n'
printf 'Normal\nB\bBo\bol\bld\bd\n_\bU_\bn_\bd_\be_\br_\bl_\bi_\bn_\be\n' | cat
printf 'Normal\nB\bBo\bol\bld\bd\n_\bU_\bn_\bd_\be_\br_\bl_\bi_\bn_\be\n' | cat -v
printf 'Normal\nB\bBo\bol\bld\bd\n_\bU_\bn_\bd_\be_\br_\bl_\bi_\bn_\be\n' | hd
printf 'Normal\nB\bBo\bol\bld\bd\n_\bU_\bn_\bd_\be_\br_\bl_\bi_\bn_\be\n' | less
printf 'Normal\nB\bBo\bol\bld\bd\n_\bU_\bn_\bd_\be_\br_\bl_\bi_\bn_\be\n' | less -u
printf 'Normal\nB\bBo\bol\bld\bd\n_\bU_\bn_\bd_\be_\br_\bl_\bi_\bn_\be\n' | less -U

Answer 2

By default, cat strips control characters when the output goes to a terminal. From man cat:

-e  Display non-printing characters (see the -v option), and display a dollar
    sign (‘$’) at the end of each line.

-t  Display non-printing characters (see the -v option), and display tab
    characters as ‘^I’.

-v  Display non-printing characters so they are visible.  Control characters
    print as ‘^X’ for control-X; the delete character (octal 0177) prints
    as ‘^?’.  Non-ASCII characters (with the high bit set) are printed as ‘M-’
    (for meta) followed by the character for the low 7 bits.

So, running

• man man | cat will strip control characters,
• man man | cat -e will show them (as will cat -v and cat -t),
• man man | cat | less will pass control characters to less (making the cat part basically a no-op).