121

Sometimes I see a distnoted process suddenly spin up and chew up 100% CPU (on one core) and a ton of memory, often in the neighborhood of 1.5G or so. This happens a few times a day, starting a month or so ago.

The command line is /usr/sbin/distnoted agent, and it's started by launchd, neither of which help much. It's usually been running for somewhere between 4h and 24h before it spins up and pegs the CPU.

Web searches say distnoted manages notification delivery, and lots of other people report the same problem with it, but I haven't yet found a fix. Some people find that closing a culprit application (e.g. Skype) stops it, but I haven't found a culprit on my machine yet. I'm usually only running a few apps: Emacs (24.2 from Homebrew), Firefox, Adium, and Dash.

I'm on Mavericks on a late 2012 13" Retina MBP. Thanks in advance!

Update:

I've turned on distnoted logging in the system log by touching /var/log/do_dnserver_log, but it doesn't help much. I see lines like these (uid 501 is me, 89 I haven't found yet):

distnoted[80011]: # distnote server agent  absolute time: 48754.144787848   civil time: Wed Nov 20 10:52:03 2013   pid: 80011 uid: 501  root: no
distnoted[20]: # distnote server daemon  absolute time: 2.808112262   civil time: Tue Nov 19 09:52:24 2013   pid: 20 uid: 0  root: yes
distnoted[444]: # distnote server agent  absolute time: 16.656997509   civil time: Tue Nov 19 09:52:38 2013   pid: 444 uid: 501  root: no
distnoted[1271]: # distnote server agent  absolute time: 52.518265717   civil time: Tue Nov 19 09:53:14 2013   pid: 1271 uid: 89  root: no
distnoted[689]: Interruption - exiting now.

I've also run sudo dtruss -p PID on a spun-up distnoted process, and it spews lines like this:

kevent64(0x3, 0x7FFF7C3FD130, 0x1)       = 1 0
workq_kernreturn(0x20, 0x0, 0x1)         = 0 0
workq_kernreturn(0x20, 0x0, 0x1)         = 0 0
kevent64(0x3, 0x7FFF7C3FD130, 0x1)       = 1 0
workq_kernreturn(0x20, 0x0, 0x1)         = 0 0
workq_kernreturn(0x20, 0x0, 0x1)         = 0 0
kevent64(0x3, 0x7FFF7C3FD130, 0x1)       = 1 0
workq_kernreturn(0x20, 0x0, 0x1)         = 0 0
__disable_threadsignal(0x1, 0x0, 0x0)    = 0 0
__disable_threadsignal(0x1, 0x0, 0x0)    = 0 0
__disable_threadsignal(0x1, 0x0, 0x0)    = 0 0
kevent64(0x3, 0x7FFF7C3FD130, 0x1)       = 1 0
workq_kernreturn(0x20, 0x0, 0x1)         = 0 0
...
17
  • Just fishing here, but by any change are you all running flux? For me, they seem to be related. If I quit flux when emacs goes berserk, emacs either crashes or returns to normal. I'm not sure if this a fluke (only happened twice), but if everyone's running it, there might be something to it.
    – user64563
    Dec 10, 2013 at 22:28
  • i'm not running flux, but maybe others are.
    – ryan
    Dec 10, 2013 at 23:47
  • aquaemacs causes this process to flip out on me.
    – marathon
    Mar 22, 2014 at 17:21
  • I had a very similar problem (possibly the same problem) and my problem went away with the 10.9.4 OS update. Jul 30, 2014 at 3:12
  • Noticed this today. The culprit was the OS X (10.9) Google Drive app (1.17.7290.4094). First time I've seen this.
    – jordanpg
    Sep 10, 2014 at 22:25

14 Answers 14

25

Summary from the OP: This was a great tool for debugging. It originally pointed me to Spotlight reindexing the filesystem, but I narrowed down the things it's allowed to index, and I still saw the problem. I ended up setting up a cron job to kill distnoted regularly. See answer farther down.


You can debug distnoted by creating the file /var/log/do_dnserver_log This causes the CFNotificationCenter server (distnoted) to record information about all notifications to the system log.

I would start there, reboot and look at the system log when the CPU spikes up. This should out the culprit easily.

More info on CFNotificationCenter debugging can be found in official Developer docs here: Technical Note TN2124 > CFNotificationCenter

15
  • thanks! good call, i've now done that. i'm not seeing any distnoted entries in /var/log/system.log, but it also hasn't spun up since i started the logging. fingers crossed.
    – ryan
    Nov 22, 2013 at 18:52
  • i am seeing distnoted log lines now, but they're not too useful. sigh. example: Nov 23 07:56:15 hell.local distnoted[2644]: # distnote server agent absolute time: 77.445654904 civil time: Sat Nov 23 07:56:15 2013 pid: 2644 uid: 89 root: no
    – ryan
    Nov 24, 2013 at 22:19
  • Try to attach DTrace script to that process and see what it actually does, start with sudo dtruss -p PID and see what syscalls does the process actually try to do and if there are any failed ones (status is not 0).
    – Temikus
    Nov 25, 2013 at 12:06
  • Also, what is the UID 89 on your system? Does the UID in notifications change? Does the pid 2644 correspond to distnoted or another process?
    – Temikus
    Nov 25, 2013 at 13:47
  • thanks for the ideas! i'm familiar with strace, but i didn't know about dtruss. i'll definitely try that next time. the pids are just the corresponding distnoted process, and the only uids are me and _appserveradm, a built-in system user i don't know much about.
    – ryan
    Nov 26, 2013 at 15:55
33

I've seen this too. Emacs 24.3.1, Mavericks 10.9.

I've found that the distnoted process calms down within seconds after I quit out of Emacs.

I've filed an Emacs bug here: http://permalink.gmane.org/gmane.emacs.bugs/80836

5
  • 2
    Also seen with Emacs v23.4.1.
    – WilliamKF
    Mar 15, 2014 at 0:23
  • 1
    Same here. Never imagined it was caused by Emacs! Thanks Jun 3, 2014 at 7:12
  • 1
    For me, I've been having the converse problem - Emacs starts using all CPU, and killing my user's distnoted clears the problem temporarily. In this case, looking at the Emacs process I see a lot of threads - non-Emacs originated ones - all waiting on the com.apple.root.default-overcommit-priority queue/mutex (run lldb, "process attach --pid <pid>", and then "thread backtrace all" to see them all)
    – jrg
    Jun 23, 2014 at 14:28
  • and this is an interesting read on what all those threads actually are: newosxbook.com/articles/GCD.html (my killing distnoted might be a 'magic feather', and not the thing that brings it back to normal)
    – jrg
    Jun 24, 2014 at 11:51
  • Also seen with Emacs v24.5 on OS X 10.11.3
    – Michael
    Mar 24, 2016 at 19:38
23

I know I'm late to the party but this is a memory leak specific to Cocoa emacs on Mavericks that is fixed in the trunk. For now there is a patch you can use to build emacs 24.3 with just the fix.

https://gist.github.com/anonymous/8553178


4
  • 1
    I updated to a nightly build from the Emacs for Mac OS X (in March) and still have the problem. It appears to happen if I create an interactive session for R or Clojure (programming languages). The distnoted process will slowly climb to GB of RAM and will free it as soon as I exit Emacs.
    – mattrepl
    May 21, 2014 at 22:57
  • Same problem that @mattrepl mentioned. Jun 3, 2014 at 18:39
  • 1
    Homebrew appears to have integrated this patch. So brew reinstall emacs --cocoa --with-gnutls may fix the problem too. It's also supposed to be fixed in 24.4 but that hasn't hit stable yet.
    – mblakele
    Aug 6, 2014 at 20:17
  • Just experienced this problem with Emacs 24.5 (fix was supposed to be in 24.4) .. in my case, Emacs was showing the spinning ball and distnoted was taking almost 400% CPU (per top) and killing -9 emacs wasn't working, but after killing -HUP disnoted emacs responded to the kill.
    – Michael
    Mar 24, 2016 at 19:37
18

I've been having the same problems with distnoted on El Capitan for some time. My solution isn't as harsh as killing it regularly, rather I check for it running out of control (high CPU usage), and then kill it. I use this script:

#!/bin/sh
#
# check for runaway distnoted, kill if necessary
#
PATH=/bin:/usr/bin
export PATH

ps -reo '%cpu,uid,pid,command' | 
    awk -v UID=$UID '
    /distnoted agent$/ && $1 > 100.0 && $2 == UID { 
        system("kill -9 " $3) 
    }
    '

The script is run from cron every minute with this line in crontab:

*   *  *   *  *   sh "$HOME/bin/checkdistnoted"

In practice, the script kills distnotedonce or twice a day, and typically this occurs after backupd starts.

For those not comfortable with the using the OS X shell (command line), the following script will install both the checkdistnoted script and the crontab entry:

#!/bin/sh
#
# install $HOME/bin/checkdistnoted
# setup crontab to run every minute
# 
# MWR Apr 2016
#

INSTALLCMD=bin/checkdistnoted
cd "$HOME"
[ ! -d bin ] && mkdir bin
[ -f $INSTALLCMD ] || {
    cat > $INSTALLCMD <<-"!!"
    #!/bin/sh
    #
    # check for runaway distnoted, kill if necessary
    #

    PATH=/bin:/usr/bin
    export PATH

    ps -reo '%cpu,uid,pid,command' | 
        awk -v UID=$UID '
        /distnoted agent$/ && $1 >= 100.0 && $2 == UID { 
            # kill distnoted agent with >= 100% CPU and owned by me
            system("kill -9 " $3) 
        }
        '
!!
    chmod +x $INSTALLCMD 
    echo installed $INSTALLCMD
}

INSTALLCRON="# check for runaway distnoted every minute:
* * * * * sh \"\$HOME/$INSTALLCMD\""
crontab -l | grep -q '$HOME'/$INSTALLCMD || {
    crontab -l > mycron
    echo "$INSTALLCRON" >> mycron
    crontab mycron
    rm mycron
    echo updated crontab
}

You need to save the above as install_checkdistnoted.sh on your desktop, then run Applications/Utilities/Terminal and type:

cd Desktop
sh install_checkdistnoted.sh 

If it works fully it will print confirmation of each of the steps. The script won't overwrite an existing checkdistnoted script or crontab entry.

3
  • 2
    THANK YOU! Terrific solution that allows me to keep disnoted, but shuts it down when it gets out of control. For other people like me who may not be familiar with the Unixy way of doing things: 1). your home folder won't have a bin directory, create a bin folder under your user name, and put the script in there as a text file named "checkdisnoted". 2). To create the cron entry, run "crontab -e" in terminal, hit the "i" key to get into insert mode, and paste the whole line with the asterisks, then hit "esc" to get back in command mode, and enter ":wq" to save file and exit the editor.
    – mike
    May 10, 2016 at 18:47
  • @Michael Rourke: This is a great solution. However, the installation script contains syntax errors under my Mac's builtin bash "GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin15)". The "||" logic shortcut and "<<-" don't seem to work here.
    – kakyo
    Aug 21, 2016 at 12:09
  • @kakyo - very sorry, the script failed because a tab became spaces - fixed now.
    – MichaelR
    Aug 23, 2016 at 7:52
8

i gave up and took the sledgehammer approach: kill it automatically, every minute. sigh.

i put this in ~/Library/LaunchAgents/org.snarfed.pkill_distnoted.plist:

<plist version="1.0">
<dict>
  <key>Label</key>
  <string>org.snarfed.pkill_distnoted</string>
  <key>ProgramArguments</key>
  <array>
    <string>pkill</string>
    <string>-KILL</string>
    <string>-f</string>
    <string>distnoted</string>
  </array>
  <key>StartInterval</key>
  <integer>60</integer>  <!-- every minute -->
</dict>
</plist>

and then installed it with launchctl load ~/Library/LaunchAgents/org.snarfed.pkill_distnoted.plist.

5
  • 1
    Michael Rourke's approach below is a touch cleaner, as it only kills disnoted when it starts eating cpu.
    – mike
    May 10, 2016 at 18:51
  • @mike but Michael Rourke approach doesn't deal with cases where disnoted is eating RAM.
    – Cœur
    Jan 3, 2019 at 7:29
  • @Cœur - Yes. I have not experienced a problem with disnoted eating RAM. Has that been a problem that you have seen?
    – mike
    Jan 4, 2019 at 4:59
  • 1
    @mike yes, disnoted was eating 63 GB of RAM on my High Sierra yesterday. Even ryan, in his question, states that the process was chewing up a ton of memory.
    – Cœur
    Jan 4, 2019 at 5:01
  • @Cœur - good point! I upvoted them.
    – mike
    Jan 4, 2019 at 5:03
4

I've been doing different combinations of stripping customizations in order to narrow down this behavior; I think it's comint mode. On 10.9 with emacs 24.3.1 from homebrew (or from emacsforosx) the distnoted + emacs leak (they both slowly increase in memory consumption) will happen with one shell-mode buffer open. It won't if you just visit files.

Just wanted to note it here, gmane appears to be down and I keep finding this discussion on my twice weekly search for followups to this issue.

4
  • thanks! i might actually be seeing the same thing. i thought neutering spotlight (the accepted answer) had worked for me, but i'm still seeing runaway distnoteds after all. thanks again for the lead, i may follow this and debug more too.
    – ryan
    Dec 12, 2013 at 18:09
  • I believe it's something to deal with my Emacs process as well. distnoted calmed right down after I killed off Emacs. I have server.el, edit-server.el and a python shell running at all times for the record. Dec 13, 2013 at 6:49
  • Seeing the same thing! Emacs to blame! Mar 25, 2014 at 2:56
  • I don't even know what comint mode is and I have the distnoted problem from emacs at times. So maybe no specific package is to blame.
    – huyz
    Aug 29, 2014 at 19:29
3

This seems to happen when an application somehow makes a wrong use of the notification API provided by macOS. In my case the culprit was iTerm2. After quitting it, the distnoted processes exited. Other culprits that have been identified are Emacs and iTunes.

1
  • 1
    iTerm2 causes it for me as well.
    – ctc
    Jul 18, 2016 at 21:30
2

I think I can only remember 2 occasions where distnoted has gone haywire. On this occasion there were 2 of them sitting to top of cpu list and one was over 400%. It happened shortly after returning to the office and plugging in a couple of external displays - one of which is usb powered - I took a guess that it might be related. I did nothing else to try and fix the problem before pulling the USB display out which brought sanity back instantly. And then plugging it back in resulted in no repeat problem.

Which proves what? No idea!

I plug them in hundreds of times and this is the first time that it occurred to me that it might be related. And since it doesn't happen everytime I plug them it, then it might have something to do with plugging them both in too quickly after each other, or something random like that. Anyhow thought I would share in case other people find it has anything to do with plugging in peripherals (if that is what an external screen is)

2
  • I had a similar situation. When I unplugged my USB display adapter distnoted stopped consuming excessive CPU (according to "top"), and when I plugged it back in, the problem did not immediately reappear.
    – Dalbergia
    Sep 16, 2015 at 18:30
  • This turned out to be the problem for me, too. Thank you! Nov 20, 2018 at 14:23
0

For what it's worth, I was able to fix this problem by disabling my anti-virus software.

0

This happened to me as well, distnoted was going crazy. After closing a bunch of applications, nothing helped.

Then I noticed one of those 'Report to Apple' dialogs from a crashed Python process had been left open all night.

Though it could just be coincidence, after closing the dialog the distnoted process calmed down.

0

I ran into a similar issue with distnoted a few months ago and could not track down why the CPU usage was spiking above 100%. Finally, I added an entry to my crontab to killall distnoted every 2 minutes which solved my problem.

Recently, I have been having an issue with Sublime Text where typing subl path/to/file was failing to open the file correctly in the Sublime Editor. A restart of the app fixed the issue, but it quickly began to happen again.

After racking my brain to no end, I identified the fact that I was killing distnoted process every 2 minutes to why the subl command had mysteriously stopped working.

The conclusion: the super high CPU usage may have been related to sublime. Now that sublime has updated, hopefully my conclusion is correct, CPU usage remains low, and my subl command returns to working as expected now that distnoted is running again without my crontab killing the process every 2 minutes.

0

I've had this problem too, for quite some time now, but intermittently. Apparently distnoted is part of iTunes and has caused problems on Windows as well. When I killed iTunes (which was playing a song), the distonted process that was using 400% of my CPU (I have 4 cores) stopped being a problem.

So my answer, until I know better, is to recommend that you kill iTunes, not distnoted, and let us know what happens.

-1

I also see distnoted go haywire, in my case it seems related to fontd. I have three distnoted running, one for _spotlight, one for _distnote and one for my user.

distnoted   0,0 6:39,85 2   0   101 _distnote   0 bytes 0 bytes     No      -   No  No  No  0 bytes 0 bytes 64 bit
distnoted   0,0 0,05    2   0   642 _spotlight  0 bytes 0 bytes     Yes     -   No  No  No  0 bytes 0 bytes 64 bit
distnoted   82,1    1:19:38,30  49  1   353 nils    0 bytes 0 bytes     No      -   No  No  No  0 bytes 0 bytes 64 bit

Whenever distnoted eats cpu (30-90%), fontworker and fontd eats about 30-60% cpu each. As soon as I kill fontd, distnoted and fontworker for my user calms down. Killing fontworker does nothing. After a couple of minutes when fontd has restarted and been running a while it all starts again.

fontworker  27,2    52,81   4   1   1073    nils    0 bytes 0 bytes     No      -   No  No  No  0 bytes 0 bytes 64 bit
fontd   32,6    1:07,41 6   0   1072    nils    0 bytes 0 bytes     No      -   No  No  No  0 bytes 0 bytes 64 bit

I have no clue on why this is happening…

-2

Peter Buckley is right, I'm wrong. I hate it when that happens.

Don't remove distnoted, the next boot will be no fun at all.

wrong> I took a more sledgehammer approach
wrong> 
wrong>    sudo mv /usr/sbin/distnoted /usr/bin/distnoted.unwanted
wrong>
wrong> This is a work machine and I have no interest in sync'ing with iTunes.

3
  • That's nuts. As is noted in Apple's page about distnoted, distnoted is part of OS X, deals with distributed notifications, and has been around since at least 2005.
    – jfmercer
    Jan 14, 2015 at 4:09
  • Whatever you do, DO NOT move distnoted as ConorR mentioned (and later corrected, thanks!), it's required to boot OSX (10.9.5 in my case). Jan 14, 2015 at 16:20
  • As much as this isn't really an answer, I think it's important that this remain noted somewhere on the page. I almost considered attempting to move distnoted.
    – Zenexer
    May 16, 2018 at 22:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .