Is there any way to prevent macOS from bringing all the app windows to front, or, to prevent the app windows shown on the other display from switching too?
No. This isn't the way app switching works. Irrespective of the number of connected displays, only app is the frontmost app at a time (is in focus and the Menu Bar shows the app's menu).
However, the approach mentioned below addresses exactly the issue that you face (I face it too) and in my experience is an easy approach to take too :)
I am assuming you use the Command + Tab keyboard shortcut to switch to Google Chrome app (and to switch to an app in general).
When using the Command + Tab keyboard shortcut to switch to an app, all the open windows are brought to front. This is the default macOS behaviour.
This may be sometimes annoying (as in your particular case), when you have a barrage of windows of an app open and you have meticulously set them up so that some of them are hidden behind window(s) of other apps.
To circumvent this behaviour, the easiest approach to follow is to use Mission Control instead of Command + Tab to pick up preciously the desired window.
First, invoke Mission Control. Now hover your mouse cursor over the Chrome window that you wish to bring to front. The window border is highlighted to indicate that it is under mouse focus.
Now if you are unsure if the window is the desired one (as it is hidden behind other chrome windows or if it too small to be legible), simply press Space key to sort of Quick Look it by enlarging it.
Once you are sure that you have correct window, simply click on it to bring only that window to front.
Note: I have enabled a setting called Group windows by application under System Preferences → Mission Control which makes the Mission Control display less cluttered by grouping app windows together.
This may restrict easily visualizing all open windows of an app if a lot them are open. It could be disabled, so as to view all open windows across all the app separately under Mission Control at the cost of having a cluttered view.
Another useful side-effect of this configuration is that hovering over a window shows the relevant title in the centre of the window, which further helps in easily identifying its contents.