I have figured out a fix to this using Firefox 58.0.2. Firefox 59 slightly breaks this functionality - it opens as a new window in a new macOS space instead of a floating popup.
I don't have fixes for Chrome and Safari or Firefox 59, but I am certain they are possible.
(If you require Firefox 58.0.2, you can find it on Mozilla's FTP here. You will likely want to disable Firefox auto-updates, but this will compromise your security in the long run.)
In the address bar, type in about:config and hit enter.
On the warning page, click "I accept the risk". As long as you don't mess with other settings, your browser should remain safe.
In the search field, type link. - this will help us narrow down to the two settings we want to change.
Double click on browser.link.open_newwindow.disabled_in_fullscreen to set the boolean to false.
Double click on browser.link.open_newwindow.restriction and set 1 as the value and click OK.
This should force Firefox to open popups like the one you have as a new window that floats on top of the fullscreen window. Again, if you use Firefox 59, the popup will make a new window, but it will go to a new space instead of floating.
