If you only really want/need "stay on top" functionality for your Terminal.app window (like myself), there is a Quake/Guake-style drop-down visor, SIMBL plug-in available that supports being pinned to the foreground; TotalTerminal.
So it's possible, despite the common misconception that it's not.
As of MacOS 10.11 (El Capitan), to install/run TotalTerminal, you might need to disable System Integrity Protection:
csrutil enable --without debug
It's no longer open source, but the last open source implementation is still available on GitHub. So if you're inclined to, you could investigate exactly how it's achieved. I emailed the developer to ask, but it's been so long since they worked on it, they couldn't (or wouldn't?) really tell me.