With "Displays have separate spaces" unchecked, then it is behaving correctly.
Your apps are not on three spaces, they're on one, across three displays… so when you disconnect the others, the Mac correctly brings the apps back into the remaining screen. It cannot anticipate your ever adding those screens back again. You would lose the apps off-screen if that were the case.
The odd thing in your drawing is that your top bar of Mission Control ought to have it laid out that way - those apps should not all be next to each other as 3 Spaces visible next to each other on one display, they should each be on its own screen, all with the same numbered Desktop label.
This is what I see on mine, with two displays & 7 Spaces 'sets'.
Click for larger version
I've roughly Photoshopped one above the other rather than side by side, one on each screen. They're mainly empty right now, but if you look at the left-most pair, Desktop 1, that's the one I have most of my day-to-day oprganisation on. Safari to the left, then Finder; and on the right display, Calendar, TextEdit, RDC, Maps, Activity Monitor etc.
If I disconnect the right display, everything jumps over to the left display, so I don't lose it, & gets annoyingly intermingled. I try to never do that, of course.
With that setup, you're probably better off enabling "Displays have separate spaces" & then setting each of those apps in a different Space. You do need at least one more Space than you have displays to do that properly. Then, if you remove a display, the "Space" should be remembered.
If you use apps in fullscreen, then all the above is moot, because fullscreen apps break almost every Spaces protocol; cannot be numbered, will not stay where you put them & come up in random order each time they're invoked… [amongst the many reasons I never use fullscreen;)