How to control buffering for processes in terminal pipe? `stdbuf` doesn’t seem to work

Question

I’m trying to use stdbuf to control buffering for a process in terminal — specifically, using stdbuf -o0 to prevent buffering when piping output onward, roughly as described in this answer — but it doesn’t seem to work as expected, or indeed have any effect. Compare the output of the following commands:

{ echo 'foo' ; sleep 2 ; echo 'boo'} | grep 'oo'
{ echo 'foo' ; sleep 2 ; echo 'boo'} | grep 'oo' | cat
{ echo 'foo' ; sleep 2 ; echo 'boo'} | stdbuf -o0 grep 'oo' | cat
{ echo 'foo' ; sleep 2 ; echo 'boo'} | stdbuf -o1M grep 'oo'

In the first, grep processes and outputs each line as it’s received. In the second, the output appears all together after 2 seconds — as explained here, grep buffers its output when not outputting to a terminal. Both of these are as expected. The third one is the surprise: according to the first linked answer and all other documentation/discussion I can find, stdbuf -o0 should prevent grep buffering its output, so that this ends up acting like the first command. However, on my Macs it acts like the second command — output arrives all together after 2 seconds. Similarly, the fourth command should (as I understand) cause grep to buffer its output, and so act like the second — but it doesn’t, it gives incremental output like the first.

Why is stdbuf not having any effect here — do other Mac users get the same results? And is there an altnernative way to generally prevent buffering in pipes? I’ve tried stdbuf -oL (line buffering), and this similarly doesn’t have any effect. I’ve also tried with gstdbuf, i.e. the version installed by Homebrew’s coreutils, and that also has no effect; both stdbuf and gstdbuf are version 9.5. I’ve tried this on three Macs, variously running Sonoma 14.4.1 and Ventura 13.6.6, and Monterey 12.7.4, with identical results. I’ve tried unbuffer from the expect package; that doesn’t have any effect here either. And I’ve tried with other pipeline targets besides cat; they all seem to show the same behaviour.

The grep-specific option --line-buffered does work, as does replacing grep by ag, which doesn’t buffer for pipes by default. However, my actual use-case is with ripgrep-all (rga) which buffers in pipelines like grep but doesn’t offer a line-buffering or non-buffering option, and this seems to be a general issue with stdbuf and unbuffer (I can’t find any test in which they seem to have any effect), so I’d really like to find a general answer if possible.

Answer 1

The problem is caused by System Integrity Protect (SIP). Below are possible solutions.

Note: Some of the solutions below assume /usr/local/bin occurs before /usr/bin in the PATH variable.

Below is the default PATH for macOS Monterey.

% echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

Note: When appropriate, you may need to enter the command hash -r to empty the hash table after entering certain commands.

• Use the command below to copy grep to a folder not covered by SIP. In this case, this would be the /usr/local/bin directory.

  rm -f /usr/local/bin/grep
  cp /usr/bin/grep /usr/local/bin
  

• Use the command below to install ggrep (GNU grep) 3.11.

  brew install grep
  

  You can then use command below to change the PATH variable.

  PATH="/usr/local/opt/grep/libexec/gnubin:$PATH"
  

  Or, instead leave the path alone and copy the ggrep symbolic link to grep in the /usr/local/bin directory, as shown below.

  rm -f /usr/bin/local/grep
  ln -s "$(readlink /usr/local/bin/ggrep)" /usr/local/bin/grep
  

  Or, again leave the path alone and use just ggrep instead of grep.

---

Below is a more detailed explanation of how setbuf works.

The setbuf command sets one or more environmental variables in an attempt to cause the command given as a positional parameter to use a different library. The functions in this library then act upon one or more of these environmental variables to change the buffering. SIP prevents commands stored in protected directories from being able to use libraries other than the ones specified when the command was created.

The setbuf command is not included with macOS. I installed setbuf by using the command below.

brew install coreutils

Note that setbuf and gsetbuf are both symbolic links to the command given below.

/usr/local/Cellar/coreutils/9.5/bin/gstdbuf

The library associate with setbuf is given below.

/usr/local/Cellar/coreutils/9.5/libexec/coreutils/libstdbuf.so

It is possible to do the same operations as setbuf without actually using setbuf or gsetbuf. For example, the commands below can be executed without using setbuf.

{ echo 'foo' ; sleep 2 ; echo 'boo'} | stdbuf -o0 grep 'oo' | cat

To demonstrate this, I will first use setbuf to get the desired environmental variable and value, as shown below.

% diff <(env) <(stdbuf -o0 env)              
0a1
> _STDBUF_O=0
24c25
< _=/usr/bin/env
---
> _=/usr/local/bin/stdbuf

Next, grep is copied to a directory not covered by SIP, as shown below.

rm -f /usr/local/bin/grep
cp /usr/bin/grep /usr/local/bin

Finally, the commands below do the same without setbuf.

Note: The description for DYLD_INSERT_LIBRARIES can be found by entering man dyld.

{ echo 'foo' ; sleep 2 ; echo 'boo'} | DYLD_INSERT_LIBRARIES=/usr/local/Cellar/coreutils/9.5/libexec/coreutils/libstdbuf.so _STDBUF_O=0 grep 'oo' | cat

Or use the commands below.

export DYLD_INSERT_LIBRARIES=/usr/local/Cellar/coreutils/9.5/libexec/coreutils/libstdbuf.so
{ echo 'foo' ; sleep 2 ; echo 'boo'} | _STDBUF_O=0 grep 'oo' | cat

What is happening here is that grep is using the functions in libstdbuf.so. These functions are using the variable _STDBUF_O to determine that no buffer is to be used.

---

To see which additional environmental variables are being passed to grep in the above example, I created the file name myenv.c containing the following source code.

#include <stdio.h>
int main(int argc, char *argv[], char *envp[]) {
    while (*envp) printf("%s\n",*envp++);
    return 0;
}

I used the gcc command shown below to create the executable command myenv. This myenv command prints out the environmental variables.

gcc myenv.c -o myenv

To see which additional environmental variables are being passed, I entered the following command. The output is also shown below.

% diff <(./myenv) <(stdbuf -o0 ./myenv)
0a1,2
> DYLD_INSERT_LIBRARIES=/usr/local/Cellar/coreutils/9.5/libexec/coreutils/libstdbuf.so
> _STDBUF_O=0
24c26,27
< _=/Users/davidanderson/myenv/./myenv
---
> _=/usr/local/bin/stdbuf
> DYLD_FORCE_FLAT_NAMESPACE=y

This output shows DYLD_FORCE_FLAT_NAMESPACE as an additional environmental variable, which I did not include in the above example. I do not think this variable is needed for macOS Monterey and newer versions of macOS, since the variable is not defined in the output of the man dyld command. However, I did find the DYLD_FORCE_FLAT_NAMESPACE variable defined in the man dyld command output for macOS Catalina and macOS High Sierra. So older versions of macOS and OS X may need to have this variable defined.

---

References

• The setbuf.c source code on GitHub
• The man page for dyld(1)
• DYLD_INSERT_LIBRARIES DYLIB injection in macOS / OSX
• Allow DYLD Environment Variables Entitlement