2

I've looked around a lot to try to find the answer to this but have thus far been unsuccessful. I am programming in Python, and I have code that takes a long time to run (hours to months depending on the application) and I am trying to optimize it. On larger thread systems (once I get above ≈8 threads, but especially >18 threads), I run into a situation where the system is using up a large amount of my CPU instead of the actual code I want to run. On a 12-thread machine as I type this, the system is using ≈25–30% of my total CPU. If I try to run it on a 36-thread machine, the system takes >70% of the CPU which is simply unworkable (though running two instances of the code, limiting each to 18 threads, cuts this overhead back, oddly enough).

The Activity Monitor indicates that taskgated is using at least 10%, while notifyd, logd, and launchd are also using several percent, each (along with mds at 1% so I should turn off Spotlight, and sometimes lsd spikes to >40%, but that's more rare ... note that that process is another launch system daemon).

My older hypothesis was that it was a disk I/O issue, since the code was writing and reading many small files to try to keep track of certain things and recover if I needed to stop it.

My now-working hypothesis, based on what I could find online about the interactions between taskgated and launchd, is that this particular chunk of code is spawning a large number of processes and macOS's launching and security daemons are taking large amounts of CPU to make sure those processes are safe. These are things like calling "mv" and "rm" from the command line (os.system(...) in my Python code), and spawning other code that take a second or two to run (such as in a different conda environment when two have conflicting installs of necessary components). I count at least 40 potential spots where this chunk of code is possibly spawning child processes, and I thread it so that it's doing that concurrently however many threads there are (so, on a 12-thread machine, 12x40 over the course of ~10 seconds). It doesn't seem to me like this should be making my system take up so much CPU, but that's my best guess at the moment.

Possibly related, the taskgated is constantly spitting out to log files, "MacOS error: -67062", which again I've searched for and found no luck with diagnosing the issue. And, diskarbitrationd is spawning a lot of "<private>" messages in the Console, but its CPU% is around 0.3% so I'm less concerned about that.

My apologies for rambling a bit here, but I'm trying to provide the information I have, and hopefully someone here has an idea. If I can get rid of this 25% or larger issue, that can save months of time.

For what it's worth, I'm running macOS 10.14.5 and ..4 on two desktops, and 10.15.5 on a laptop. Same issue for all. Running in Linux on a nearly identical 36-thread system build does not have the overhead issue (but I really don't want to switch to Linux) which is another reason why I don't think it's a disk I/O problem.

  • I'm not aware of Docker, I could certainly try it if that's the only way this can be solved. The spawned processes range in longevity, it's anything from a simple "mv" terminal command to running another program that can take seconds to hours, though most are on the shorter side. I am certainly willing and able to change the code, I just don't want to spend a week doing it and this NOT end up being the problem that's causing these processes to go crazy. – Stuart Robbins Aug 12 at 7:50
0

I think the overhead you are getting with the daemons you are referring to are unavoidable on macOS. For example, launchd is the main process for launching applications and ensures that the processes it launched are kept alive if instructed so. Using more threads on macOS is a well known issue concerning a higher overhead for the kernel. This is why Apple documentation clearly states that you should use them judiciously and sparingly. Moreover, it seems as though macOS is not trusting your script, as is an unsigned “executable” and the error you are getting corresponds to:

security error -67062 Error: 0xFFFEFA0A -67062 code object is not signed at all

Thus, the extra overhead you are seeing is most probably due to Gatekeeper, which is constantly checking what your script is spawning and doing.

Possible (partial) solutions to your problem:

  1. Embed your python script in a signed app - this is what Apple recommends in the technical note TN2206.
  2. Use Linux instead
| improve this answer | |
  • Thanks for the info. Embedding in a signed app likely won't work because it must call other code from the US Geologic Survey which I'm sure is not signed. I thought perhaps disabling Gatekeeper might work (enabling the "sudo spctl --master-disable" option) but, no. It's going to take me awhile to think more about your response. I like macOS (been a fan since the early 1990s), I'll be annoyed if the easiest solution is to go back to Linux for this work. – Stuart Robbins Aug 13 at 21:26
0

Is your question to understand macOS better or have your Python code complete faster? I suspect the latter.

If so, have you profiled your Python code? What does the performance profile show?

macOS

taskgated and launchd are both involved in evaluating and launching processes.

Turn on macOS's server performance mode for raised resource limits.

If a lack of code signing is a suspected cause, you can ad-hoc code sign your – and others' binaries:

sudo codesign -f -s - <full path to bundle or executable>

External Processes

The spawned processes range in longevity, it's anything from a simple "mv" terminal command to running another program that can take seconds to hours, though most are on the shorter side.

On any operating system, avoid calling out to external processes when an in-language call exists. Launching a process and waiting for the process to terminate is expensive, compared to a system call.

See Python - How to move a file? to replace mv with os.rename(), shutil.move(), or os.replace().

Threads

Adding threads to your process will not guarantee that calls down into the operating system will not be queued and handled sequentially.

Threads in Python are posix threads and thus managed by the operating system. Adding threads gives the underlying operating system more work and more influence on how your process performs. In this regard, the difference between Linux and macOS is significant.

Use threads to handle data manipulation and, where possible, pass off file handling work to a dedicated file handling thread. Avoid touching the disk unless absolutely essential to continue the next task. Even then, try and pass data to and from other processes using pipes or inter-process communication (IPC).

Use a solid state drives (SSDs) instead of spinning hard disk drives (HDDs).

Linux

Given you appear to have proven Linux is faster than macOS, use Linux.

Docker

To save months, I would justify 1 - 2 days experimenting with Docker. This approach will allow you to run a light weight instance of Linux on your Mac and avoid the proven costs of macOS. This should ease the cost of spawning processes.

There will be an unfortunate learning curve to Docker but it will be time well spent.

Using Docker will give you a well defined working environment that can be started, stopped, and replicated without ties to the host operating system.

Unavoidable?

Be wary of assuming the 20-25% of system time is not useful, and is avoidable. macOS is a heavy operating system compared to Linux. High Performance Computing uses specific operating systems for a reason. If using Linux is easy and gets the results faster, sinking time into macOS appears unjustifiable.

| improve this answer | |
  • 1
    I'll try this, I will see if it actually works right without the performance hit. The Linux test was with a slightly older version of my code. – Stuart Robbins Aug 12 at 8:04
  • You've made many additions to your post, so I'll just quickly note: I'm running everything on NVMe, so disk I/O is very fast. In the last 10 minutes, I've changed all the mv/rm/cp calls with os.system to built-in Python calls, but that did not change the overhead. The other programs that it calls are not something I have any ability to modify. This chunk of code has already been updated to minimize disk I/O, I can't change it further (I can change other aspects, but those aren't bogged down in this overhead). I'll let you know if Docker works, BUT in principle, it doesn't answer my question. – Stuart Robbins Aug 12 at 8:31
  • Looks like you edited again, so to address your question at the top: My goal is both. Obviously, running the code faster is the practical goal, but if there are coding issues running afoul of the way macOS operates that causes this, then I should know about those so as I proceed, I don't re-introduce the issue. Running it virtually in a different OS is not ideal, at all, and seems to be a cheat factor for an underlying issue. – Stuart Robbins Aug 12 at 17:33
  • The -m cProfile tool is nice. I've spent the better part of the last 9 hours using it to optimize this code chunk, in a test region getting the time down from 14.1 seconds to 3.8 seconds. The HUGE time sink was that my code uses two different conda environments (they don't play well together), and using "conda run -n [other] [code]" was causing a ≈0.5–1-second lag to just start it up (x3500 times). So, I have used other tools to work around those required scripts. However, while this full test area now runs in 15 minutes instead of 25, I'm STILL getting a 20–25% System usage cut, same PIDs. – Stuart Robbins Aug 13 at 3:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .