One partial solution is to use the Audio/MIDI setup utility to configure a "multi-output device" that only includes the audio devices you want to use, and setting that as your system audio device. However, using this method has the major disadvantage of not allowing any easy means of adjusting the volume on the underlying devices, even if there's only one device set up.
Another issue is that if any of the devices ever get temporarily disconnected or take a little while to appear at system startup or the like, you'll have to add the device back into the multi-output device.
Unfortunately, macOS doesn't appear to have any built-in means of disabling an audio interface entirely or setting the device priority, which is supremely annoying.
It must be possible somehow
. Mac misses a lot of obvious features. Very Apple-y IMO.