3

I upgraded a machine to macOS Big Sur two days ago, and after a day or so my PiHole DNS server randomly stopped working, which upon inspection was caused by gigabytes of logs for DNS queries made by the Big Sur machine overwhelming the poor thing. I monitored DNS requests on the machine with tcpdump:

$ sudo tcpdump port 53 -k NP
xx:xx:xx.xxxx (proc mDNSResponder:32622, eproc apsd:108) IP 192.168.0.xxx.xxxxx > pi.domain: 24541+ A? xxx-courier-x.push-apple.com.akadns.net. (64)
...

and holy crap, apsd, the Apple Push Notification Service daemon, was literally spamming multiple DNS queries for xxx-courier-x.push-apple.com.akadns.net through mDNSResponder every millisecond (the domain is obfuscated a bit here; the 5th level domain is different for different geographical regions).

Note that the DNS queries are answered just fine, the domain isn't blocked by PiHole or anything.

I killed mDNSResponder which made it stop, which might indicate that apsd is the innocent party and mDNSResponder is the culprit, but I can't be sure.

I then rebooted out of caution, and after a day or so, the problem started again.

I've never seen anything like this on previous versions of macOS. Anyone got insight? Or experiencing something similar and have more details to add?

6
  • Are you seeing CPU or just concerned that the log system can’t handle millions of entries a day without any measurable load - apple.stackexchange.com/questions/386721/… – bmike Nov 28 '20 at 4:56
  • @bmike CPU on the Mac isn't much of a concern, I don't recall CPU usage of mDNSResponder spiking when it happens and I don't think I even bothered to check. But my DNS server is a Raspberry Pi with a 64GB SD card as storage medium, so it certainly can't handle logging millions of entries per hour to a SQLite database on crappy storage medium (didn't check but probably a separate transaction per entry); the DNS server fails after a few hours. This is a bug regardless, but probably wouldn't manifest as complete DNS failure if you're just hammering a public DNS server. – 4ae1e1 Nov 28 '20 at 5:02
  • This might be since Pi is blocking. Are you running stock PiHole or decrypting / MITM any traffic to Apple? – bmike Nov 28 '20 at 5:42
  • @bmike Stock PiHole with some pretty lax lists. No, it's not blocking the particular requests, tcpdump shows the DNS responses too right after corresponding requests and all the push-apple.com.akadns.net requests are resolved just fine (stated in the question). Also, even if they are blocked, hammering the DNS server thousands of time per second still isn't reasonable behavior... – 4ae1e1 Nov 28 '20 at 6:07
  • Btw I just added CPU/RAM usage logging to my tool so the next time this happens the CPU usage of mDNSResponder is recorded. github.com/zmwangx/mDNSSupervisor/commit/… My guess is it will cause some load, but again, the CPU load is hardly my concern. – 4ae1e1 Nov 28 '20 at 6:11
1

This keeps coming up every day or two, so eventually I wrote a small utility called mDNSSupervisor to monitor mDNSResponder and restart it whenever it's apparently going berserk. Doesn't solve the underlying issue of course but enough to cure the symptom and be mostly safe from its effects.

You can download the tool if you want from https://github.com/zmwangx/mDNSSupervisor (free, WTFPL licensed). Installation instructions there, too.

A sample from my logs showing it in action:

time="2020-11-28T17:15:06+00:00" level=info msg="1606583706: 15s rolling average 142.8, above threshold 100"
  PID  %CPU    RSS  ELAPSED COMMAND
  246  38.2   7692 14:01:57 /usr/sbin/mDNSResponder
  259   0.0   4768 14:01:57 /usr/sbin/mDNSResponderHelper
time="2020-11-28T17:15:06+00:00" level=info msg="mDNSResponder restarted"
2
  • Nice update. Will post if I get any bright ideas - so far, I can’t reproduce this – bmike Nov 28 '20 at 18:26
  • 1
    Yeah issues like this are typically very elusive... If I really had the energy I might disable SIP and attach a debugger to mDNSResponder live... But I don't... So I'll settle for treating the symptom ;) – 4ae1e1 Nov 28 '20 at 18:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .