What causes excessive CPU use by taskgated, notifyd, and launchd processes?

Question

I've looked around a lot to try to find the answer to this but have thus far been unsuccessful. I am programming in Python, and I have code that takes a long time to run (hours to months depending on the application) and I am trying to optimize it. On larger thread systems (once I get above ≈8 threads, but especially >18 threads), I run into a situation where the system is using up a large amount of my CPU instead of the actual code I want to run. On a 12-thread machine as I type this, the system is using ≈25–30% of my total CPU. If I try to run it on a 36-thread machine, the system takes >70% of the CPU which is simply unworkable (though running two instances of the code, limiting each to 18 threads, cuts this overhead back, oddly enough).

The Activity Monitor indicates that taskgated is using at least 10%, while notifyd, logd, and launchd are also using several percent, each (along with mds at 1% so I should turn off Spotlight, and sometimes lsd spikes to >40%, but that's more rare ... note that that process is another launch system daemon).

My older hypothesis was that it was a disk I/O issue, since the code was writing and reading many small files to try to keep track of certain things and recover if I needed to stop it.

My now-working hypothesis, based on what I could find online about the interactions between taskgated and launchd, is that this particular chunk of code is spawning a large number of processes and macOS's launching and security daemons are taking large amounts of CPU to make sure those processes are safe. These are things like calling "mv" and "rm" from the command line (os.system(...) in my Python code), and spawning other code that take a second or two to run (such as in a different conda environment when two have conflicting installs of necessary components). I count at least 40 potential spots where this chunk of code is possibly spawning child processes, and I thread it so that it's doing that concurrently however many threads there are (so, on a 12-thread machine, 12x40 over the course of ~10 seconds). It doesn't seem to me like this should be making my system take up so much CPU, but that's my best guess at the moment.

Possibly related, the taskgated is constantly spitting out to log files, "MacOS error: -67062", which again I've searched for and found no luck with diagnosing the issue. And, diskarbitrationd is spawning a lot of "<private>" messages in the Console, but its CPU% is around 0.3% so I'm less concerned about that.

My apologies for rambling a bit here, but I'm trying to provide the information I have, and hopefully someone here has an idea. If I can get rid of this 25% or larger issue, that can save months of time.

For what it's worth, I'm running macOS 10.14.5 and ..4 on two desktops, and 10.15.5 on a laptop. Same issue for all. Running in Linux on a nearly identical 36-thread system build does not have the overhead issue (but I really don't want to switch to Linux) which is another reason why I don't think it's a disk I/O problem.

Answer 1

I think the overhead you are getting with the daemons you are referring to are unavoidable on macOS. For example, launchd is the main process for launching applications and ensures that the processes it launched are kept alive if instructed so. Using more threads on macOS is a well known issue concerning a higher overhead for the kernel. This is why Apple documentation clearly states that you should use them judiciously and sparingly. Moreover, it seems as though macOS is not trusting your script, as is an unsigned “executable” and the error you are getting corresponds to:

security error -67062 Error: 0xFFFEFA0A -67062 code object is not signed at all

Thus, the extra overhead you are seeing is most probably due to Gatekeeper, which is constantly checking what your script is spawning and doing.

Possible (partial) solutions to your problem:

1. Embed your python script in a signed app - this is what Apple recommends in the technical note TN2206.
2. Use Linux instead

Answer 2

Is your question to understand macOS better or have your Python code complete faster? I suspect the latter.

If so, have you profiled your Python code? What does the performance profile show?

macOS

taskgated and launchd are both involved in evaluating and launching processes.

Turn on macOS's server performance mode for raised resource limits.

If a lack of code signing is a suspected cause, you can ad-hoc code sign your – and others' binaries:

sudo codesign -f -s - <full path to bundle or executable>

External Processes

The spawned processes range in longevity, it's anything from a simple "mv" terminal command to running another program that can take seconds to hours, though most are on the shorter side.

On any operating system, avoid calling out to external processes when an in-language call exists. Launching a process and waiting for the process to terminate is expensive, compared to a system call.

See Python - How to move a file? to replace mv with os.rename(), shutil.move(), or os.replace().

Threads

Adding threads to your process will not guarantee that calls down into the operating system will not be queued and handled sequentially.

Threads in Python are posix threads and thus managed by the operating system. Adding threads gives the underlying operating system more work and more influence on how your process performs. In this regard, the difference between Linux and macOS is significant.

Use threads to handle data manipulation and, where possible, pass off file handling work to a dedicated file handling thread. Avoid touching the disk unless absolutely essential to continue the next task. Even then, try and pass data to and from other processes using pipes or inter-process communication (IPC).

Use a solid state drives (SSDs) instead of spinning hard disk drives (HDDs).

Linux

Given you appear to have proven Linux is faster than macOS, use Linux.

Docker

To save months, I would justify 1 - 2 days experimenting with Docker. This approach will allow you to run a light weight instance of Linux on your Mac and avoid the proven costs of macOS. This should ease the cost of spawning processes.

There will be an unfortunate learning curve to Docker but it will be time well spent.

Using Docker will give you a well defined working environment that can be started, stopped, and replicated without ties to the host operating system.

Unavoidable?

Be wary of assuming the 20-25% of system time is not useful, and is avoidable. macOS is a heavy operating system compared to Linux. High Performance Computing uses specific operating systems for a reason. If using Linux is easy and gets the results faster, sinking time into macOS appears unjustifiable.