How can I get macOS to tell me what chroma sampling or subsampling is being used on an external display on an M1 mac?

Question

After receiving my MacBookPro18,4 (i.e. MacBook Pro (14-inch, 2021)) with an M1 Max chip and seeing the ProMotion display, I was hooked on high (>60Hz) refresh rates, and treated myself to a LG 38WN95C-W for an external display.

Very pleased with 3840x1600@144Hz except that sometimes red text on a black background or white text on a red background would look eye-strainingly blurry and awful, with weird dark fringing around the edges. After playing with the settings, I discovered that the likely cause was Chroma subsampling; specifically that it was likely that my single Thunderbolt connection was not high-bandwidth enough to drive the display at 144Hz. I stepped it down to 120Hz (which is really all I was looking for anyway, although 60Hz→120Hz is very noticeable, 120Hz→144Hz is barely perceptible, at least to me) and now the fringing seems to be gone.

However, I would really like to know if any subsampling is still occurring, and in particular I'd like to be able to experiment with different configurations (is it my dock? my cable? the display interface; i.e. does DisplayPort work better? can I get 144Hz anywhere or is 120 the max without this?) without having to look at subpixels in the iOS magnifier app to determine if compression is still happening.

I've looked at the output of system_profiler SPDisplaysDataType and AGDCDiagnose, but the former doesn't show me a Y'CrCb value and the latter only works on Intel chips.

Update 2022-03-22: I got a new CalDigit TS4 Dock and, wondering if the higher bandwidth of Thunderbolt 4 would result in less chroma compression, I took a look at 144Hz running through it. No difference that I could see, based on this chroma sampling torture test image. But then I wanted to see if DisplayPort was any better or worse – it was hard to see, but it seemed like I was getting some level of subsampling at every refresh rate, so I plugged the thunderbolt cable back in, and now I don't get subsampling at 144Hz either. So… the entire saga was just a loose thunderbolt connection, maybe? Or some bad persistent state in the display, cleared by power cycling it when I was experimenting? I still really wish I could get a quick report on the display protocol so I didn't need to take multiple comparison pictures every time I made a change!

Update 2022-06-09: At some point the signal quality degraded again, and the torture-test image started showing subsampling reliably. So the connection at 144Hz is extremely prone to interference, possibly operating right at the top of the available thunderbolt bandwidth, or... something. Not clear. But I'm down to 120 again for good this time. I'd still love a clear answer as to how to diagnose this properly without guessing...

Update 2022-07-01: For unrelated reasons I tried to use the DisplayPort connection again, just to see. And now… it's working great, and even offers me variable refresh rates up to 144Hz, taking advantage of FreeSync. Upon first being plugged in, once again, I saw chroma subsampling. At every refresh rate. But then, cycling back and forth between thunderbolt inputs, I now see no subsampling, again, at every refresh rate. This whole process is crazy-making without any diagnostic tools! I have had some other issues with the dock, and have RMA'd it and will be replacing it this week, so maybe this has something to do with a defect in the dock where it's stepping down to lower display bandwidths because of a fault rather than because this is how it's supposed to work. I've already RMA'd it, and when I receive the replacement I'll conduct these tests again.

Update 2023-01-18: I seem to have settled into a pattern with this, where every time I unplug the computer from the dock and then plug it back in, I'm back to YPbPr 422 and I have to open Displays and toggle HDR on, then back off again, and it settles in to RGB and stays there. Very annoying but at least manageable.

Answer 1

Apparently the issue is that YPbPr is being used at all, and the monitor ought to be in RGB mode, and this is a well-known issue that has been much discussed elsewhere. I have now learned how to activate the service menu on my display, inspect the colorspace (currently, RGB 444, and no artifacts of chroma subsampling).

Since my display is working properly for the moment due to some random auto-detection going correctly for this particular configuration, if this issue were to recur, I'd use this script from Github to force the display into RGB mode again.

The aforementioned script does not actually work. What does seem to work reliably, at least for now, is to open Displays, switch to 120Hz, toggle HDR on, then toggle it off again, which reliably gets me an RGB mode. However, every time I unplug from the dock and plug back in, it's back to YUV 422 and I need to manually do this again. I have yet to figure out a way to automate it,

(Update 2023-1-25: I found a way to automate it. But at what cost…)

… or even to detect if it's necessary. The com.apple.windowserver.displays.plist preferences described elsewhere does not seem to reliably have a PixelEncoding key that accurately describes the current state of the external display; it's usually right but after having a gadget in my shell prompt that reads it every few minutes, I've noticed that it's just … randomly different sometimes, with no pattern I can detect. Editing it does not seem to have an effect, either; sometimes it just gets overwritten with a non-0 PixelEncoding.

I've filed a feedback with Apple and I'd encourage anyone else dealing with this issue do the same, since these janky hacks and workarounds should not be necessary. It can and should just autodetect the full bandwidth it should use.