While as mentioned in the question, it is not natively possible to show the count of open windows for an app in either the app switcher UI or the Dock, there are a couple easy methods (that I personally employ) to determine if an app has one or more window(s) open and easily switch to the the desired one. Setup and usage for both of them is described below.
Note: These approaches are applicable only for the apps that show a window and is not applicable for the apps which run in background and Menu Bar apps.
Method I : Using Mission Control
Mission Control gives you a birds eye view of all the running apps with open window in the system. To invoke Mission Control, press the Mission Control (F3) key on the keyboard.
You'll need to check Group windows by application under
System Preferences app → Mission Control to see windows grouped by application to easily serve your requirement by knowing at a glance if the target app has one or more windows open.
Mission control display both without window grouping by application and with is shown below for easy reference (Notice that the same set of apps and windows are currently open in both the cases). The former approach has the benefit that the app group also shows the app icon towards bottom centre of the group which helps in easy identification.
Additionally, you may have more than one Spaces active at the same time and the app windows may be scattered across different Spaces. Only the windows present on the current Space are shown at a time and you can easily switch between Spaces without leaving Mission Control view.
You now have an idea if the target app has one or more windows open. While in Mission Control view, you can simply click on the window group for the target app to switch to it (instead of doing Cmd + Tab). Once the target app is frontmost, you can perform Cmd + ` to switch to desired window of the app (if it has more than one window open).
This approach has a drawback that an app may additionally have one or more minimised windows and the Mission Control view doesn't show them, thereby defeating the purpose. That's where the second approach as discussed below helps.
Method II : Using Hot Corners
You'll need to configure Hot Corners for this approach.
Hot Corners is a feature of macOS that lets you assign instant actions for when you move the mouse pointer to any one of the four corners of your Desktop.
For more information on Hot Corners, read the discussion What are Hot corners in macOS?.
Configure any one of the four corners via Hot Corners configuration to show Application Windows.
Now switch to the target app using either Cmd + Tab or Mission Control (as discussed in Method I). Once the target app is frontmost, use Hot Corners by moving the mouse pointer to the configured corner. All the open windows for the app are shown.
The benefit of this approach is that it also shows the minimised window(s) (which are shown towards the bottom in smaller size). To switch to the one desired window among many open, simply click on it with your mouse pointer. If the desired window is minimised, it will be automatically maximised and brought to front.
I employ both the above discussed approaches as one of the powerful workflows in my day to day usage of macOS. (The screenshots may be a bit unintuitive, as I have set the Menu Bar to Auto Hide, have High Contrast display enabled and transparency effects turned-off in macOS).