In short: you can not force a frequency band in OS X 10.9 Mavericks. (On 10.5 you can...)
You want to connect to the device using Basic service set identification (BSSID) instead of regular Service set identification (SSID). Connecting to a BBSID will connect you to a specific device regardless of the connection strength. Connecting to SSID will connect you to a specific network name, if similar network names are available it will connect to the best signal/noise ratio. It must be noted that your OS X chooses the wlan, not your router, and OS X switches to the strongest signal available (2.4GHz or 5GHz).
To find a specific SSID and BSSID combination, you can run:
Before OSX 10.6 you could connect to a specific BSSID using:
/System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport --associate=XXX --bssid=YYY
Where XXX is the SSID/network name and YYY the MAC address of the base station you want to talk to.
On OS X 10.6 and beyond it is no longer possible to connect to BSSID directly. There is no known API for this and no third party Software exists that can do this. So you need to change your 5GHz SSID to a unique name or you need to go back to OS X 10.5, or you can change the 2.4GHz channel from within the router. I think these options are non-valid in your case.
UPDATE As maxim points out, you can force a frequency band and use that to fix to 5GHz.
On linux you can use
iwconfig, but this tool is not available for OS X.