As far as I can tell this is an issue that is provoked by something in Chrome.
I'm struggling with the same issue but while having multiple displays attached (behaves the same as spaces in most ways) where if I have Chrome on Display one and two.
If I now cmd+tab to an app that's only on display 2 it'll switch focus to that display but when I Cmd+tab back to Chrome it will focus back to display 1 in spite of having a Chrome window also on display 2.
If I now switch these two windows of Chrome around the focus priority doesn't change so it doesn't have anything to do with the creation order of the windows but soley with the Desktop which they're on.
For my 2-Display setup it prioritizes always the same one no matter where the Dock is located or anything else I could think of to try. I therefore think it's got something to do with the internal differentiation of macOS between the displays.
But now for the kicker: If I try to reproduce the same illogical behaviour with Safari it doesn't happen. Safari behaves exactly as one would expect regarding focusing order.
Thus I think it's a problem created by some interaction of Chrome and macOS because of the arbitrary lock to a Display.
I therefore think the easiest solution for your Problem is to move the Terminal Window(s) to the Desktop where Chrome focuses. Maybe reordering the Desktops might change something about the behaviour but I didn't check that.