I have an iMac with this spec:
I have two external monitors, both 4k. Recently, Spaces has become almost unusably slow for me. Switching between spaces on the main display (usually between full-screen IntelliJ projects) using the trackpad is the laggiest. It will scroll a bit, pause for a second, then jerkily scroll the rest of the way. I think I managed to speed it up a little by removing files from my desktop, but that certainly hasn't fixed the problem.
If I open Spaces, then dragging them to rearrange them takes several attempts, because the drag usually fails to start (or at least takes longer than the 30 seconds or so I've tried waiting), and even hovering over different windows takes a couple of seconds before moving the blue highlight.
For some reason, things seem smoother on the external monitors, which only have about two applications each on them.
Most things, like dragging windows around and opening menus, happen just as smoothly as you'd expect them to.