I've looked around a lot to try to find the answer to this but have thus far been unsuccessful. I am programming in Python, and I have code that takes a long time to run (hours to months depending on the application) and I am trying to optimize it. On larger thread systems (once I get above ≈8 threads, but especially >18 threads), I run into a situation where the system is using up a large amount of my CPU instead of the actual code I want to run. On a 12-thread machine as I type this, the system is using ≈25–30% of my total CPU. If I try to run it on a 36-thread machine, the system takes >70% of the CPU which is simply unworkable (though running two instances of the code, limiting each to 18 threads, cuts this overhead back, oddly enough).
The Activity Monitor indicates that
taskgated is using at least 10%, while
launchd are also using several percent, each (along with
mds at 1% so I should turn off Spotlight, and sometimes
lsd spikes to >40%, but that's more rare ... note that that process is another launch system daemon).
My older hypothesis was that it was a disk I/O issue, since the code was writing and reading many small files to try to keep track of certain things and recover if I needed to stop it.
My now-working hypothesis, based on what I could find online about the interactions between
launchd, is that this particular chunk of code is spawning a large number of processes and macOS's launching and security daemons are taking large amounts of CPU to make sure those processes are safe. These are things like calling "mv" and "rm" from the command line (os.system(...) in my Python code), and spawning other code that take a second or two to run (such as in a different conda environment when two have conflicting installs of necessary components). I count at least 40 potential spots where this chunk of code is possibly spawning child processes, and I thread it so that it's doing that concurrently however many threads there are (so, on a 12-thread machine, 12x40 over the course of ~10 seconds). It doesn't seem to me like this should be making my system take up so much CPU, but that's my best guess at the moment.
Possibly related, the
taskgated is constantly spitting out to log files, "MacOS error: -67062", which again I've searched for and found no luck with diagnosing the issue. And,
diskarbitrationd is spawning a lot of "<private>" messages in the Console, but its CPU% is around 0.3% so I'm less concerned about that.
My apologies for rambling a bit here, but I'm trying to provide the information I have, and hopefully someone here has an idea. If I can get rid of this 25% or larger issue, that can save months of time.
For what it's worth, I'm running macOS 10.14.5 and ..4 on two desktops, and 10.15.5 on a laptop. Same issue for all. Running in Linux on a nearly identical 36-thread system build does not have the overhead issue (but I really don't want to switch to Linux) which is another reason why I don't think it's a disk I/O problem.