I'd like to find a modern MKV video player app that supports macOS's new Picture in Picture feature.
Note: I'm not interested in replicating PiP functionality, I'm only interested in the actual Picture in Picture feature.
I found that IINA can use macOS's native Picture in Picture mode.
Picture-in-picture is a specific Safari browser and iTunes feature, and Safari is incompatible with MKV container file formats. At present there is no way to add macOS support for non-Apple approved container formats or codecs (the old Perian tool has not been updated for a very long time).
The way to achieve what you want would be to use an app like Adapter to convert the MKV file to MP4, which is natively understood by macOS apps, and then play it either in iTunes or QuickTime Player. In QuickTime Player you will need to select View -> Float on Top. This is not strictly picture-in-picture but has the exact same effect.
Try plex + pipifier.
Install the pipifier Safari extension as well as plex for MacOS. Start the plex server, then open Plex in Safari. Add your media folder into plex, then start watching a video. You can then use pipifier to watch almost any video in pip mode.