Why is lsof on OS X so ridiculously slow?

Question

I can't figure out why lsof on my Mac (10.8.2, MacBook Pro) is so slow.

On my Mac, lsof takes more than a minute:

$ touch /tmp/testfile
$ time lsof /tmp/testfile

real   1m16.483s
user   0m0.029s
sys    1m15.969s

On a typical Linux box, running Ubuntu 12.04, lsof takes 20 ms:

$ touch /tmp/testfile
$ time lsof /tmp/testfile

real   0m0.023s
user   0m0.008s
sys    0m0.012s

The problem persists if I run lsof -n (to avoid DNS lookups). Further, I tried checking which system calls are made by lsof using dtruss, and found that it's calling proc_info tens of thousands of times:

$ sudo dtruss lsof /tmp/testfile 2> /tmp/dump
$ cat /tmp/dump | sort | uniq -c | sort -nr | head
10000 proc_info(0x2, 0x1199, 0x8) = 1272 0
 6876 proc_info(0x2, 0x45, 0x8) = 1272 0
 2360 proc_info(0x2, 0x190D, 0x8) = 1272 0
 1294 proc_info(0x2, 0xFF, 0x8) = 1272 0
 1152 proc_info(0x2, 0x474, 0x8) = 1272 0
 1079 proc_info(0x2, 0x2F, 0x8) = 1272 0
  709 proc_info(0x2, 0xFE, 0x8) = 1272 0
  693 proc_info(0x2, 0x1F, 0x8) = 1272 0
  623 proc_info(0x2, 0x11A, 0x8) = 1272 0
  528 proc_info(0x2, 0xF7, 0x8) = 1272 0

Any ideas? I've run these tests and obtained the same results using both the version of lsof included with OS X (4.85) as well as the latest version from ftp://sunsite.ualberta.ca/pub/Mirror/lsof/ (4.87).

(For the curious, the reason I'm frustrated by this performance is that when I drag images to Evernote, it runs lsof in the process of copying the file, causing my system to hang for a full minute every time I try to insert an image in Evernote.)

Answer 1

As my experience, from Mac OS X 10.7(Lion) to 10.11.5（EI Capitan), the lsof always hang.

To solve to problem, append -n option.

lsof -n

According to manual of lsof, the -n option:

inhibits the conversion of network numbers to host names for network files.  
Inhibiting conversion may make  lsof  run faster.  It is also useful when host 
name lookup is not working properly

EDIT 2018-04-25: If it is still slow, you can try

-O to bypass  the  strategy it uses to avoid being blocked by some kernel operations
-P to inhibits the conversion of port numbers to port names for network files
-l to inhibits  the  conversion of user ID numbers to login names

The ultimate way to find out why so slow is to run "Instruments" tool (from upper right corner Spotlight Search icon) to do a "System Trace" on /usr/sbin/lsof then see graph and sys calls.

Answer 2

I think the biggest part of the problem is that macOS is becoming ever more ridiculous with bloat and unnecessary layers upon layers of wasteful frameworks. This has meant hundreds of extra processes and thousands of extra files being held open, increasing the amount of work lsof has to do by at least an order of magnitude, and perhaps more like two orders.

lsof went from reasonable speed to atrociously slow between 10.6 and 10.13.

Here on a current 10.13.4 system I see the following with just 7 apps open and running (Terminal, Chrome, Calendar, Finder, Adium, IPGadget, and Stickies). (Chrome has 7 windows, with perhaps 10 tabs each.)

# ps ax | wc -l
     401
# time lsof -lnP | wc -l
   10976

real    0m49.684s
user    0m0.250s
sys 0m40.172s

During the run both CPUs are well over 50% system time

Adding -O helps sometimes, especially if lsof hasn't been run lately, but the best I've seen was about 10% savings. Usually it is minuscule and likely not worth the risks outlined in the manual page:

# time lsof -lnPO | wc -l
   10994

real    0m47.482s
user    0m0.249s
sys 0m40.472s

dtruss claims there are over 89,000 calls to proc_info() with my current process load, and those are into the kernel, and as time reports, the vast majority of time spent is in the kernel. I don't know why there are about 8 calls per open file.

Sadly macOS/Darwin doesn't include the ever more useful and efficient BSD fstat command.

Answer 3

I don't have a great answer why your system appears to take a minute longer than my slowest Mac to call proc_info 30 thousand times, but your timing shows that both linux and OS X are in the 10 ms range for user time to run lsof. Can you reproduce that slow time booting in Safe Mode to rule out other loads on your CPU?

I've tried three Macs and the ones running 10.7.5 are about a second faster than my 10.8.2 Mac. The older OS are slower Core 2 Duo processors and I'd think an i7 Mac running the newer OS would be as fast or faster than older OS and CPU, but I'd be wrong.

All the machines make about the same number of proc_info calls, and all the machines have lean user time for the command - but you might be on to a slower overall timing (and I have no clue why yours is so dramatically slower than my Mountain Lion Mac).

11 inch Air (i7) 2011 running Mountain Lion - SSD:

$ system_profiler SPSoftwareDataType
      System Version: OS X 10.8.2 (or something)
      Kernel Version: Darwin 12.3.0
      Secure Virtual Memory: Enabled
$ time lsof /tmp/testfile 

real    0m1.179s
user    0m0.012s
sys     0m1.158s
$ sudo dtruss lsof /tmp/testfile 2> /tmp/dump
$ cat /tmp/dump | sort | uniq -c | sort -nr | head
9310 proc_info(0x2, 0x68, 0x8)           = 1272 0
1220 proc_info(0x2, 0xCEB6, 0x8)                 = 1272 0
$ cat /tmp/dump | cut -c -9 | sort | uniq -c | sort -nr | head
30884 proc_info
 116 write(0x4
  87 read(0x5,
  60 sigaction
  60 setitimer
  35 stat64("/
  30 sigprocma
  30 sigaltsta
  21 close(0x3
  18 close(0x6 

15 inch MacBook Pro running Lion Server - HDD:

$ system_profiler SPSoftwareDataType
      System Version: Mac OS X Server 10.7.5 (11G63)
      Kernel Version: Darwin 11.4.2
$ time lsof /tmp/testfile

real    0m0.329s
user    0m0.005s
sys     0m0.324s

27 inch iMac running Lion - HDD:

$ system_profiler SPSoftwareDataType
      System Version: Mac OS X 10.7.5 (11G63b)
      Kernel Version: Darwin 11.4.2
$ time lsof /tmp/testfile

real    0m0.066s
user    0m0.002s
sys     0m0.065s
$ sudo dtruss lsof /tmp/testfile 2> /tmp/dump
$ cat /tmp/dump | cut -c -9 | sort | uniq -c | sort -nr | head
23034 proc_info
 188 write(0x4
 141 read(0x5,
  96 sigaction
  96 setitimer
  48 sigprocma
  48 sigaltsta
  31 stat64("/
  21 close(0x3
  18 close(0x6