Yes, if you know applescript.
For example, to switch to Space 1 on the Secondary Display. Note:
1) Primary/Secondary Display is defined by where the Menu Bar is (i.e. System Preference -> Display -> Arrangement), not by cursor focus.
2) This script switches to Space 1, whether it's a Desktop or fullscreen app. If you want to switch only to Desktop 1, it can be done, but not with this script as it is.
3) If you don't mind flashing, remove
delay 0.5 line.
4) The script cannot do without the animation/transition.
5) Enable Accessibility and all the standard applescript spiel.
6) Modify button number to switch to a different Space on that Display. Modify list number to switch a different Display.
7) Switching to a non-existent Space, e.g. Space 100, would leave the UI at mission control. Nothing bad is going in bad to your computer. It just stays there, and user will have to manually drop back to current Space.
8) No relative switching, i.e. move left or right a space. Just absolute switching.
9) Cursor focus doesn't switch display after running this script. That's a plus.
10) No simultaneously switching Spaces on both displays.
tell application "System Events"
do shell script "/Applications/Mission\\ Control.app/Contents/MacOS/Mission\\ Control"
tell process "Dock" to tell group 1 to tell list 2 to tell button 1 to click