5

The tail command has become unusably slow for large files (e.g. 1 GB text file).

EXAMPLE:

$ tail -265000 file.txt > last_265000.txt

takes 10+ minutes, whereas

$ head -265000 file.txt > first_265000.txt

is basically instantaneous.

My workaround is to use the tac command (installed via brew install coretools) to reverse the file lines and use head:

$ tac file.txt | head -265000 | tac > last_265000.txt

I don't remember this being an issue in my prior macs. Tried switching shell from zsh to bash, but same deal.


Specs: MacBook Pro (13-inch, M1, 2020) Big Sur

The text file: https://www1.nyc.gov/assets/finance/downloads/tar/tc1_22.zip

6
  • How much is it having to page to do that? Presumably that kind of command needs the whole file in RAM, perhaps even with a shadow copy. Most people seem to think 8GB RAM is enough. It isn't ;) Try quitting your biggest users first; see if it speeds up.
    – Tetsujin
    Jan 12 at 17:49
  • Running this on a 1 GB file with 94'354'400 lines took 4 seconds on a M1 mini (which already has memory pressure, so free RAM may not be the issue here).
    – nohillside
    Jan 12 at 18:20
  • 1
    tail seeks to the file's end and reads backwards, so memory needs should be minimal.
    – JRFerguson
    Jan 12 at 20:40
  • 1
    What is in the file?
    – Gilby
    Jan 13 at 2:49
  • Picking up the idea by Gilby: I would first create a file consisting of, say, 10 million lines, each consisting of the same 100-byte-string. Do a time measurement with this file. If this file is fast, the reason must be in the concrete contents of your file.txt. If the test file is also slow, we know at least that it really is a problem with tail and/or your platform. In the latter case, I would install the gnu tools and try GNU-tail for a comparision. Jan 13 at 8:54

1 Answer 1

4

Summary of below, so far:

  • /usr/bin/tail in Big Sur is very slow when compared to similar utilities.
  • When presented with a text file with long lines, it can become unusably slow.
  • We don't know if this is unique to this version of macOS.
  • We don't know if this is on Apple's radar.
  • gtail from coreutils is an effective replacement.

It looks like the default tail in /usr/bin is very slow when a large line count is requested, and is orders of magnitude worse when given a binary file:

text file:

> time tail -3 threeGB.txt > /dev/null
tail -3 threeGB.txt > /dev/null  0.00s user 0.01s system 89% cpu 0.011 total


> time tail -26500 threeGB.txt > /dev/null
tail -26500 threeGB.txt > /dev/null  1.31s user 1.85s system 98% cpu 3.195 total

> time gtail -26500 threeGB.txt > /dev/null
gtail -26500 threeGB.txt > /dev/null  0.00s user 0.00s system 42% cpu 0.018 total


> tail -265000 threeGB.txt > /dev/null
tail -265000 threeGB.txt > /dev/null  12.80s user 17.76s system 99% cpu 30.700 total

> gtail -265000 threeGB.txt > /dev/null
gtail -265000 threeGB.txt > /dev/null  0.02s user 0.02s system 87% cpu 0.038 total

binary file:

> time tail -3 twoGB.mpg > /dev/null    
tail -3 twoGB.mpg > /dev/null  0.00s user 0.00s system 40% cpu 0.011 total

> time tail -26500 twoGB.mpg > /dev/null 
tail -26500 twoGB.mpg > /dev/null  338.10s user 265.03s system 78% cpu 12:52.17 total

> time gtail -26500 twoGB.mpg > /dev/null
gtail -26500 twoGB.mpg > /dev/null  0.01s user 0.04s system 24% cpu 0.193 total


(MacBook Pro 2017, Intel i7, 16GB, Big Sur, internal ssd)

I don't know if this is a new problem with Big Sur. It could be that this wasn't previously noticed due to some combination of file size, file type, and line count.

A workaround, as demonstrated above: use gtail from coreutils. If you want, you can link it into your PATH as tail (this post has some info for that).


Added: testing with the OP's text file, using the same MacBook Pro listed above.

> time tail -265000 tc1_22.TXT >/dev/null
tail -265000 tc1_22.TXT > /dev/null  577.89s user 808.23s system 99% cpu 23:11.05 total

> time gtail -265000 tc1_22.TXT >/dev/null
gtail -265000 tc1_22.TXT > /dev/null  0.46s user 0.66s system 97% cpu 1.154 total

The file is ASCII text, CRLF, with tab-delimited fields. Significantly, each line has 1,900 characters; the performance here fits in with the pattern found by @John Palmieri.

9
  • It dosn't make a lot of sense to run tail on a binary file... Also, while 3s for tail is slower than the 0.02 seconds for gtail, it's far from the +10 minutes the OP seems to have.
    – nohillside
    Jan 13 at 11:03
  • 1
    @Yarin - I added that to the post. As you can see, for the text file it was slower still, but not at the scale you're experiencing. For the binary file, I stopped it after twenty minutes. I haven't been able to find a text variant that runs as slowly as the binary file. Attempted so far: UTF-8, UTF-16, CRLF (dos-style linefeeds).
    – Gairfowl
    Jan 13 at 19:42
  • 2
    I did a few of my own experiments, and the timing appears to vary depending on the length of the lines. With short lines (50 characters), the tail command is much faster — just over 1 second as compared to 32 seconds — than with longer lines (1200 characters). Jan 13 at 22:02
  • 1
    Hi, just curious if anybody opened an issue for this with apple.
    – Axel
    Jan 16 at 8:07
  • 2
    @Axel - not that I'm aware of. If you're familiar with the process and want to try it, or know how to search their bug list, please add that to the post.
    – Gairfowl
    Jan 16 at 15:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .