Linux has binfmt_misc which allows arbitrary executable file formats to be recognized and passed to certain user space applications. Does macOS have a similar feature?
No, macOS does not come with a similar feature.
If you're interested in adding it yourself, you can could take a look at this project:
It is a kernel extension that extends the kernel's
exec_shell_imgact() function to recognize other file types (namely Linux ELF binaries) and pass them to a user space application to be executed. The source code is available, so it could potentially be tweaked and extended to provide functionality similar to binfmt_misc.