open -jga TextEdit opens TextEdit hidden (
-j) and without making it frontmost (
-j option was added in 10.8 and it is only shown in the help message and not in the man page. If TextEdit is already running,
open -ja TextEdit (without
-g) would just make TextEdit frontmost and not hidden. For some applications like TextEdit and Mail,
open -jg opens a new visible window if the application is running but has no open windows, but you can use something like
pgrep -x TextEdit||open -jga TextEdit as a workaround.
A second option in 10.9 and later is
lsappinfo launch launchandhide=true nofront=true /Applications/TextEdit.app.
A third option is
osascript -e 'launch app "TextEdit"'. Unlike the
lsappinfo commands, it is blocked until the application finishes opening. For some applications like TextEdit and Mail, the
launch command opens a new visible window if the application is not already running.
Edit: none of the options above seem to work with VirtualBox, but you can hide VirtualBox after opening it:
open -jga VirtualBox;lsappinfo setinfo -app VirtualBox kLSApplicationIsHiddenKey=true
osascript -e 'launch app "VirtualBox"' -e 'delay 1' -e 'tell app "System Events" to set visible of process "VirtualBox" to false'