Serious Performance Problems using Expose in Catalina

Question

Summary

I am having a problem where using Expose to drag a window from one Space to another causes an enormous slowdown that looks almost comical - like the machine turns into a ouija board and load spikes to easily over 200(!) during the time it is going through "fits"

Here is a video of the an incidence of this problem.

Conditions

1. Fresh Catalina install
2. MacBook 12,1 with dual core 3.1 GHz i7 CPU, 16 GB RAM and 1 TB SSD
3. Currently in the process of "building from scratch" and installing my prescribed software and tools

Walkthrough of the Problem

1. I invoke the Screens in the heading for each "Desktop" using Expose and attempt to move a window from one Screen to another

2. The behavior of moving a window as I drag has action where the lag of dragging movements enter a state of ghostly slow-motion. where the movement follows a sort of "possessed" pattern.

3. As my queue UI interactions play out, it looks like it has a mind of its own. (All further UI interaction appears to be ignored, but in actuality it is getting queued and will continue to run at a fraction of real-time speed.)
4. I need to stop and wait for all of those input actions to complete. This can take several minutes, sometimes over 15 if I try to do things like click on an icon in the dock, move a window within a specific Space, or other do what a frustrated user might do while his machine seems unresponsive.

Side Effects

Massive Load Spike

The immediate reason for this would appear because the system load which already was running between 3-4 (computer has 4 effective CPUs, so just under nominal efficiency when it is effectively idle from human input, it will shoot up to a very high number (well over 100 or 200 while going through these antics) and it will take several minutes for the load "settle down" after the slow-mo effects stop, and that's just for it to get from 200 down to about 25 where it will plateau for easily another another 10 minutes.

Queuing of System Processes

As I am still in the process of building my computer from scratch based on a build template document that I made to manually install my software tools setup on a fresh machine, I find this enormously disconcerting.

Build Details

1. Use updated to be Catalina compliant with things like DeviceKit
2. Some open source tools require explicit security authorization to even install, but the seem like innocuous exceptions which don't require additional kernel drivers and I hope they will be compliant by January.
3. I don't use "junk ware" that could easily border on Malware (such as "virus protection).
4. I do use brew cask install or any software that I can.
5. For anything else, I use App Store.

Initial Conclusions

I don't have any idea if these apps have any correlation to this behavior or not. When this does happen, I will often system processes come up and take more than 50% of the CPU, but I attribute this to the fact that they are queuing because of the absurdly high load and more a symptom than a cause of my problems. I have even given up Chrome for Safari in an attempt to better make use of shared resources and minimize deviating from the Apple nest. I have already turned on "Reduce Motion" and "Reduce Transparency" for Accessibility -> Display but it had no impact.

Possible Heat Problem?

One theory I had was that my fans are dirty and not cooling my CPU well. I did have performance problems when this machine was running Mojave with 30-90 second lock-ups, but since I had upgraded it through at least 3 OS versions, I attributed it to KEXT cruft or other junk software I used when I was less careful. The fans do spin up just fine as the CPUs heat up, but the CPU core temp wavers between 70°C and 80°C even when load is under 4. Therefore, I am wondering if all of this is simply CPU clipping from heat but I don't know where the temp will induce this fail-safe. If I simply cannot rule out blocked airflow I will crack this baby open and give it the canned air blast, but seeing as I replaced the fans and battery just 14 months ago that would be surprising as I don't work in a lint factory.

What I am Hoping to Achieve by Posting This

In any event, while I don't expect anyone to identify why Expose is behaving this way (though I would be grateful for some troubleshooting tips as I am only a few days into using Catalina.) What I was hoping for was if anyone was aware of how I might turn off any of the gratuitous animation features introduced by Catalina that might be exacerbating this problem.

Update 11/11/19 17:20

After running in Safe Mode and loading nearly every app I couldn't replicate the problem, even though things like scrolling were definitely affected. There didn't seem to be any temp problems and spikes to over 90° quickly subsided so unless this is a slow buildup heat issue, I don't think that is it.

One thing I failed to mention was that when this problem occurred, there was 3.7 GB in the 5 GB swap space. Starting up the machine normally and running the apps like I did in Safe Mode also didn't recreate those problems, but swap is still unused. I should have noted this, but in the past, I have correlated the sluggishness of the Expose animation with how much is in swap.

My new theory is that there is some memory leaking going on that uses up virtual memory and the slowness and load is a result of relentless paging. I think I will only be able to substantiate this after letting my machine have an uptime of more than a day or two with lots of apps open so I can see. Then I can identify which apps have the highest page rates.

Update 11/12/19 01:54

After opening it up and not seeing much dust at all I lifted the cooling plate to check the thermal paste which was dried and barely present. I added some fresh paste after cleaning both surfaces but I am not sure if I used too much or the surfaces weren't both clean enough but I am finding temps seeming to spike over 100°C occasionally and it overall running hotter, though the hot peaks are only instantaneous.

I ran GeekBench both before and after, both in Safe and regular mode and performance results were effectively the same. Results are actually slightly higher than the benchmarks established for my model. It seems awfully hot but it is not affecting performance any. Having read articles stating that the i7 can safely operate at 100°C (though some authorities say it is too hot for any duration).

I installed the Intel Power Gadget and it shows my overage clocking is about 2.45 GHz but that it does peak out at the stated 3.1 GHz so I don't think it is being clipped. Still, the performance mystery abounds, but one thing is for sure- it takes days if not weeks for all of the Catalina housekeeping on media and pictures to complete!

Answer 1

It sounds very likely that you have an hardware error such as fans not working, thermal paste not properly applied, vents blocked or similar.

A load average of 3-4 when the machine is idle is definitely not normal. Your machine is a dual-core machine - HyperThreading doesn’t really make it sensible to count that as 4 cores (it’s not nearly as good as separate cores). A load avg. of 3-4 means the computer is highly loaded.

The high load average could come from thermal throttling (either via kernel_task forcing the CPU to sleep, or simply frequency throttling). It could also come from other sources such as a malfunctioning disk - although then you would most likely be experiencing other problems as well.

I will recommend booting in Internet Recovery mode to check if you still have high load average and slow performance there. If you do, then it’s not a software problem.

Answer 2

Actions Taken

My solution involved the following:

1. Cleaning the fans and airways
2. Replacing the thermal paste
3. Adding monitoring for temp and fan speed
4. Increasing the overall fan speeds using custom rules over the system defaults

Cleaning the Hardware

I took my MacBookPro apart and found that there was a bit of dust accumulated in the fan itself and about 10% of the heat sink's fins were blocked with linty dust. There were also some dust-bunnies and random spots on the board and in pockets of the chassis. It didn't seem like a lot of dust, but I blasted it with canned air nonetheless.

Similarly, I was delighted that I still had a new tube of thermal paste in my computer toolkit. After unscrewing the heat-sink plate cinch from from the CPU core, I saw the old paste was all but dried up and very anemic. I used a dry cloth to wipe both surfaces clean and applied fresh pasted and cinched it back down. However, I may have used too much paste since there was some excess squeezed out the sides. The cinch was as tight as it would go without over-tightening and risking either stripping the heads, shearing the screws or ripping out the threading. The goal was to ensure there was no way for air bubbles trapped in the paste that could heat up and reduce the surface area where paste contacted both surfaces.

Software

Before I addressed the cooling, I added and configured three utilities:

Fan/Temp Readings and Controls: iStat Menus / Macs Fan Control

Both of these utilities have the ability to measure the fan speed and CPU temperature, both have a trial version, but iStat Menus reads about two dozen other temperature sensors throughout the system while MFC only reads the CPU core temp. While I first used MFS, I later found iSM to be the better overall choice as it has utility far beyond temp and fan management so I paid the $10 to use it as an overall metrics display.

MFC has limited fan control in trial mode while iStat Menus is fully functional where you can create custom fan speed rules so you can try it before you buy it to see if it is for you. Also, to be fair, iSM and MFC are not really comparable in scope since iSM allows you to make extensive customizable graphs for nearly every system metric imaginable.

Furthermore, iSM appears to measure stats which are already being reported much like /proc is in Linux. To test, running with and without iSM I saw little variance in performance (using Activity Monitor, which I don't recommend running indefinitely as it uses a lot of resources relatively speaking). Even the memory footprint of the UI elements of iSM is only 35 MB - less than 1/4 of Activity Monitor which can also spike CPU load making it inappropriate as a full-time monitor like iSM is.

While I still have MFS installed, I don't really use it in favor of iSM. I keep it in case I need to have a potentially more lightweight fan controller/speedometer than iSM, but I have not had the need for it yet.

CPU Clock Measurement: Intel Power Gadget

In addition to iSM, Intel Power Gadget provides visibility of the variable CPU speed which actually is constantly fluctuating depending on system demand. I am not certain that this tool would register "clipping" as the result of overheating, but I can't imagine why it wouldn't. Like iSM it also provides extensive graphing features. Furthermore, it provides a new metric of CPU speed as a datapoint that iSM now puts in the CPU metrics and it's listed right next to the CPU Core temp for easy tracking! The tool need not even be running to read this- it is added to the rest of the system metrics that iStat can read! This was a great find and added a crucial metrics that I lacked before.

Results

In order to establish a speed benchmark, I used Geekbench 5 to get a baseline before and after the cooling cleanup and tuning. I also ran it in Safe Mode as well as normal mode (with as little running as possible, though this was not a pure test since things like photoanalysisd were often chugging along in the background regardless). Still, what I found was very surprising: while performance was drastically improved when heat was tamed, measurements after cleaning actually showed the CPU hitting hotter temps at spikes. I have a hypothesis for why, however.

Performance Improvements

Before I cleaned the cooling system and added fresh paste, bench tests showed single/multi CPU ratings at around 700/1775 at start both before and after the cleaning. Running the fans at max at all times vs cleaning the cooling didn't change performance measurably. Also, that measure is actually just above the Geekbench stated average for my machine. (Safe Mode tests actually were about 5-10% slower.)

While I waited for my machine to grow sluggish from either heat or the bloating of swap usage and grinding from page ins and outs, the fact that I had imposed more aggressive fan speed rules with iSM seemed to keep heat spikes in check all its own. Furthermore, after cleaning - even if I return fan speed rules to system defaults as before - I couldn't recreate the lockup problems even though the CPU temp was reporting at being 10O°C for sustained high load. It would seem that the CPU can get hotter that previous thought in order for the CPU to get clipped because I could see that my CPU speed was in "turbo" mode at zesty 3.4 GHz when running a GB CPU test. Even under extreme duress with fan settings that let the CPU reach temps over 100°C for more than 30 seconds, the overall performance improvements were profound and the machine just in anecdotal usability. The nasty problems with Expose/Mission Control did not recur.

Apparent High Temperature Spikes

Admittedly there is one anomaly that initially made no sense. Before I cleaned it and just ran the fan at max speed, CPU temps had a high floor even at rest 65°-75°C but the maximum measured temp never seemed to get hotter than 90°C. After cleaning, measurement behavior was very different. While the floor temperature at rest was lower with fan defaults when idle (sometimes as low as 40°C), I noticed that both the CPU core temperature would fluctuate wildly with load (and CPU speeds) where before the measurements showed a much more gradual change even though the measurement polling and refresh was the same. Also, aside from faster heating and cooling of sensors, the max readings occasionally peaked over 100° when it never reported as being that hot before. With the fan rules set to more aggressive settings that respond to higher temps with faster speeds, the CPU floor temp was about 15°-20°C lower with a load between 3-4, often down in the 40's when load was around 1. (More on system load and temp volatility in a bit.)

Conclusions

1. It is clear that poor cooling was the primary factor in my machine's poor performance, though saying just that is a gross oversimplification of the results I saw. For one, I believe the measurements I took before the cleaning and pasting were not accurate nor precise because of dust buildup. It is possible that the wildly spiking temps that are commensurate with the changing CPU speeds that vary with the machines workload, the performance improvements don't track with this possibility. Since the temp sensors are what the machine uses to regulate CPU, it is possible that the disparity was causing some problems. Once the sensors could report accurately and precisely, all-around performance was closer to intended design.
2. In addition to the physical cleaning, running with faster fan speeds did wonders for improving cooling just as one might expect before cleaning, however with a cleaned cooling channel the effects were that much more evident. While it appears to keep heat-related problems at bay running at full speed nearly constantly before cleaning, after cleaning I could relax the rules substantially and taming the heat was accomplished much more easily knowing that actual vs measured temps were not in disparity.
3. Therefore, my running hypothesis for the temp volatility is that the CPU temp sensors' readings are outside the CPU code and had been muddied by dust accumulation. After being cleaned this may have allowed them to be more precise and accurate. It may be that higher temp spikes simply were not sustained long enough to to be read beforehand since the dust insulated the probes. Not knowing where the sensor probes that measure CPU temps physically are, I cannot say if this holds water or not.
4. Finally, after cleaning and tuning the fan rules, system load - while still often spiking as high as 30 or 40, was not so terrible that it ground the machine to a halt. High load spikes are just a fact of life on older Macs, but I now know they don't have to be crippling. Before I cleaned and cooled, it looked like Expose/Mission control animation problems kicked in when load was over 100. Now, loads can get as high as 40 if I rally push it and it's paging like crazy, but even when the machine is at 3.4 GHz and fans are maxed out under large CPU and I/O loads and where it is paging at a rate of 8 MB/second, performance impacts leave my machine still useable if but a little sluggish on the UI.

TL;DR: Dusting the cooling system and upping the fan speeds as specific temp thresholds is what it took to get my machine back to its healthy self. Also, the i7 CPU has a variable clock rate that is a power saving feature, not just a throttle for when the heat spikes - at least not under normal operating conditions. Adding some good metrics that don't tax the system are crucial in seeing something other than an anecdotally notable improvement and iState Menus seems like a great choice if you're data hungry. There are more lightweight open source and command line tools out there for purists.

I hope these lengthy breakdown was helpful. I found it was worth going into all of the details, even if there was some redundancy.