Activity monitor style way to see Neural Engine load?

Question

I am using an M1 mini with 8 GB of RAM, which is barely functional when I have voice control switched on, due to memory usage frequently going into the red zone in activity monitor.

It would be nice if Apple were able to provide guidance of system requirements and performance for various features within accessibility. However my message to [email protected] got the response that Apple do not provide this kind of information.

So I need to invest in a more powerful Mac (this one had a lifespan of about six months) because voice control is a must have for me. I'm not very keen on spending £1100 on another Mac mini (this would be simply be an 8 GB RAM upgrade, at approx 50 times the price of 8gb of ram), because that might turn out to be a short lived investment also, given apple's lack of information on the hardware which voice control really needs.

I noticed that the M1 ultra in the new Mac Studio has 32 NPU's. All of apples other devices, newer iphones, M1, M1 pro, M1 Max, have 16 NPUs.

The complete inadequacy of 8 GB of RAM is obvious, so I'm minded to get as much ram as I possibly can afford, but I'd love to know if there's a way to answer the question which Apple says it cannot answer - given my workflow of Head Pointer (an accessibility feature a few people are aware of - it uses the WebCam so that small movements of the head move the mouse pointer), plus full voice control (dictation plus commands), I would really like to know if when I am running both of these things, what is the load on the 16 core NPU. Hence, how much if at all is the Ultra going to help in this regard.

Without a way to see NPU load, knowing what hardware I actually need seems to require guesswork.

---

A few updates

• in activity monitor under the CPU tab, there's a GPU load column.
• speech recognition with voice control on on my M1 8 GB shows between 30 and 40% GPU load from speech recognition core.
• yet, usually after the Mac hasn't been rebooted for a couple of days, there's a voice recognition delay for both dictation and commands of up to 5 seconds (very slow compared to when it's working well).
• so it's not clear if this lag is due to ram being in the yellow zone, which it is, or something else.
• it seems pretty likely to me that this code is still being optimised for apple silicon and may improve in future os updates from Apple (without any fanfair).
• the actual dropout of dictation, seems not to occur with the latest update to Monterey

Further update

• it appears that at the current time, voice recognition isn't even using the NE. Evidence: gpu load in activity monitor - either NE load is shown as gpu load, or, seems more likely, the NE is a new thing and voice rec has been built to use gpu, and is still using it because the code hasn't been changed, as yet. If it will be or when, haven't found any information on this. Perhaps head pointer does use NE though.

Answer 1

You seem to consider that the money you've spent on the M1 Mac was lost. Remember that you got 6 months of service out of it - and it can be resold. You would likely be able to recoup a significant part of your investment there.

For the replacement Mac I would definitely go with one with more RAM. The number of NPUs is not important, as the accessibility features you have mentioned, do not benefit from extra NPUs. So I definitely wouldn't buy an M1 Ultra Mac Studio just for that purpose (note that I think the M1 Ultra Mac Studio is a really good computer in its right, so if you have other needs for it, definitely consider it).

Answer 2

There is no way to view what you ask since Apple didn’t ship such a tool. My guess is there’s no viable gateway to measure or metrics needed to fully diagnose performance issues so the existing tools were left in place by Apple Engineers since they didn’t need a new tool. Apple likely won’t build a test harness that they don’t see as needed.

What I do know is the workload you seek to measure isn’t confined to the “neural engine” part of the M1 you have.

Core ML then seamlessly blends CPU, GPU, and ANE (if available) to create the most effective hybrid execution plan exploiting all available engines on a given device.

https://machinelearning.apple.com/research/neural-engine-transformers

Apple then takes that plain hybrid execution model and complicates it with at least four optimizations that complicate any benchmark or trivial calculation of theoretical throughputs to deal with several constraints:

we can trade this flexibility off in favor of a particular and principled implementation that deliberately harnesses the ANE, resulting in significantly increased throughput and reduced memory consumption. Other benefits include mitigating the inter-engine context-transfer overhead and opening up the CPU and the GPU to execute non-ML workloads while ANE is executing the most demanding ML workloads.

So memory and bandwidth constraints are factored in for execution plans.

All is not lost since with the CoreML and other SDK anyone that is building models can dump timing and profiling code and measure exactly how long their repeatable tasks take on any Mac and log the results to the unified logging subsystem for easy analysis, storage, retrieval and collection.

Answer 3

The tool ASITOP attempts to estimate Apple Neural Engine load similar to top does for other CPU statistics that the OS reports directly.

---

It is a Python-based nvtop-inspired command line tool for Apple Silicon (aka M1) Macs.

Utilization info:

• CPU (E-cluster and P-cluster), GPU
• Frequency and utilization
• ANE utilization (measured by power)

Memory info:

• RAM and swap, size and usage
• (Apple removed memory bandwidth from powermetrics)

Power info:

• CPU power, GPU power (Apple removed package power from powermetrics)
• Chart for CPU/GPU power
• Peak power, rolling average display