How to diagnose too many kernel panics on a 2017 21" Retina iMac running Monterey

Question

This on a 2017 Retina 21" iMac Running MacOS 12.4

The day before I installed 12.4 I had a kernel panic. The day after I installed it I had four kernel panics in rapid succession in the space of a half hour. Since then I have had more. Sometimes a day goes by without one, sometimes there are several in a day. I have read and, for the most part, followed this:

If Your Mac Restarts and a Message Appears

The Kernel-YYYY-MM-DD-hhmmss.panic logs (/Library/Logs/DiagnosticReports/) offer no clues as to what software/hardware might be causing the panics

So far I have:

• Removed an external display and adapter • Removed a USB hub • Notremoved a non-Apple (Logitech) keyboard as the machine would be useless without it. • Not removed an external drives as the machine would be useless to me without them

I started the machine in Recovery Mode and:

• Ran Disk First aid on the boot volume: No problems found • Reinstalled MacOS

I started the machine in Diagnostics Mode and ran diagnostics: No problems found

The only extensions I have are those installed by:

• Graphic Converter • Bartender • Dropbox • Google Drive • Markup

The external drives that I have not removed have all been running fine on the machine for well over a year. I’ve had the keyboard for about 18 months.

I have attached the first few lines of the most recent panic report. This one happens to show that iStat menus was the panicked task, but I don’t think that means much. Every panic reports shows a different process as the panicked task.

Any ideas?

I have uploaded four .panic files. Complete links:

https://mgnewman.com/panic/Kernel-2022-05-27-160620.panic https://mgnewman.com/panic/Kernel-2022-05-27-160805.panic https://mgnewman.com/panic/Kernel-2022-05-27-163709.panic https://mgnewman.com/panic/Kernel-2022-05-27-163923.panic

The should be downloadable with curl, or wget or even a web browser.

They should be symbolicated:

    <key>boot-args</key>
    <string>keepsyms=1</string>

Should be symbolicated, but, AFAIK, they are not. At least on this machine, changing that one NVRAM setting did not have the desired effect on the kernel panic files.

MemTest86 Results

MemTest86 V9.4 Free (Build: 1000) Result summary
PassMark Software
www.passmark.com
Test Start Time: 2022-06-02 00:00:27
Elapsed Time: 3:22:42
CPUs Active: 4
CPU Temperature Min/Max/Ave: 67C/81C/72C
RAM Temperature Min/Max/Ave: 100C/142C/125C
# Tests Passed: 48/48 (100%)
Lowest Error Address: N/A
Highest Error Address: N/A
Bits in Error Mask: 0000000000000000
Bits in Error: Total: 0 Min: 0 Max: 0 Avg: 0
Max Contiguous Errors: 0

Safe Mode

I booted into Safe Mode; something I'd never done before, so I didn't know what to expect. What I really didn't expect was what a slug the machine was in Safe Mode. It was barely useable. Instead of moving across the screen smoothly, the pointer moved in fits and starts. In text fields the insertion point simply disappeared now and then. I tried to use the machine that way, but was actually grateful when a kernel panic rebooted the machine in regular mode and the machine returned running as smoothly as normal. So weird. Is Safe Mode always like that?

Answer 1

You've enabled kernel symbolication and we have found that it doesn't provided a human-readable backtrace. The possibilities for this are that (1) Apple has suddenly removed symbolication capability (unlikely), or (2) the addresses belong to a range that didn't map to a symbolicated region of the code. Neither of these would be expected behavior in this situation.

Furthermore, I'm not aware of any publicly available tools for decoding the macOSProcessedStackshotData field of the report, which might have turned up something actionable.

Nevertheless, we have some useful information even without symbols:

1. All four of your panic reports include the following identical signature:

   Machine-check capabilities: 0x0000000000000c0a
   family: 6 model: 158 stepping: 9 microcode: 240
   signature: 0x906e9
   Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz
   10 error-reporting banks
   Processor 0: IA32_MCG_STATUS: 0x0000000000000005
   IA32_MC0_STATUS(0x401): 0xb200000000030005
   

   This signature means that your problem belongs to a class of panic called a Machine Check Exception (MCE), which is initiated in x86 silicon rather than by xnu. So, in all likelihood, the displayed backtrace would just show the MCE AST panic handler and indicate nothing about the origin of the MCE. This is strongly suggested by the fact that subtracting the KASLR slide from each of the frame return addresses yields an identical inner stack:

   0xffffffxxxxxxxxxx : 0xffffffxxxxx81c8d
   0xffffffxxxxxxxxxx : 0xffffffxxxxxe1596
   0xffffffxxxxxxxxxx : 0xffffffxxxxxd0963
   0xffffffxxxxxxxxxx : 0xffffffxxxxx21a70
   0xffffffxxxxxxxxxx : 0xffffffxxxxx8205d
   0xffffffxxxxxxxxxx : 0xffffffxxxxx81816
   0xffffffxxxxxxxxxx : 0xffffffxxxxx15163
   0xffffffxxxxxxxxxx : 0xffffffxxxxxd19d7
   0xffffffxxxxxxxxxx : 0xffffffxxxxx1c94c
   0xffffffxxxxxxxxxx : 0xffffffxxxxx222cf
   

   That's the pattern in 3 of the 4 panics. The 4th one (Kernel-2022-05-27-160805.panic) appeared to have been in the middle of another operation when the AST occurred. It might have been marginally useful to know what it was doing but I'm guessing just machine_idle() or something like that.

   In other words: The backtraces aren't useful in this case.

2. Taking a look at Intel's Software Developer's Manual, Vol. 4, Ch 15-16, your report consistently shows that the first core-local MCA register (MC0) on your BSP (Processor 0) has an uncorrected internal parity error (0xb...05). What's more, the CPU wasn't able (or it didn't make sense) to identify an address that was associated with that error, so there is no IA32_MC0_ADDR register value to examine. The model-specific error code (0x...3...) is not decodable through public documentation so we don't know what it means for your specific CPU (Core i5-7500).

3. The fact that all four signatures are identical means that your issue is due to a specific, reproducible root cause.

And yet, despite these important clues, we simply don't have enough information or debugging tools available to get any more granular data on what the cause could be. MCE root causes span the entirety of hardware, firmware, and software – a bug in any of those can manifest as an MCE. Due to the low-level nature of MCEs and the fact that they halt CPU execution, the amount of useful diagnostic data that is able to be collected from a closed system at runtime from this generation Intel core is very limited (as compared to an "ordinary" kernel panic, when the CPU continues to fetch & execute instructions and xnu is often able to continue running to significant extent, allowing for interactive debugging).

What this means for you is that we're sort of stuck grasping at straws on our own. You indicate that you've done some hardware isolation experiments. There are only a couple of minor things I can think of there:

• What interface(s) (cable(s)) are you using to connect your external drive(s) to your Mac?
• Can you try running in Safe Mode (hold Shift during boot) for a while and see if the panic goes away completely or still recurs?
• Can you try running an extended memory test? (Download the free version and dd the memtest86-usb.img image onto a flash drive, then boot from it.) Apple Diagnostics often misses errors that MemTest86 catches so run that and see if you get any errors (it'll take a while!).

Answer 2

tl;dr - If you got here by searching on "kernel panics" the short answer is that there are many things you can do to suss out the cause of kernel panics; none of which worked for me.

If you really want to go through all this, then read the excellent and informative comments by pion as well as the extensive suggestions in his answer to the question.