I am working on some OpenGL development and I get to the point where I can move the mouse but nothing responds (Sometimes I can wait for a really long time). The mouse still moves just none of the apps respond. This is especially strange since I have the sandybridge with 16gbs RAM. I would like to try to go and kill an emulator I am running to fix the issue, in Linux I would do this with ctrl+alt+fx to get to a terminal and type my kill command. This doesn't appear to work in OSX, is there another option?
There are no virtual consoles like in Linux; there is no way to interact with the system using the connected keyboard/mouse/display outside of the regular GUI which you have already found to be unresponsive.
To bail out with a minimum of needed interaction, there is a shortcut to kill the frontmost application directly: ⌥⇧⌘⎋, Command-Option-Shift-Escape. This is precisely the same action as the usual Force Quit dialog, except that it does not ask you to choose an application.
(Tip: If you press modifier keys while a menu is open, in this case press Shift while the Apple menu is open, shortcut variants like this one will be displayed.)
If you have another machine or even a phone around, another thing you can do is connect via SSH (turn on Remote Login in Sharing preferences) and use that to kill the runaway process. However, if your runaway program is hogging the RAM then I recommend you SSH in before you start work, as making a new SSH connection while the system is stuck in swapping will be just as sluggish, but using an existing connection is less problematic.
(Note that in the unfortunate case where your OpenGL program is hitting glitches in the graphics driver, as opposed to merely consuming resources itself, you may find that the system continues to be unusable even after your process exits, since the entire GUI rendering is based on hardware-accelerated graphics processing.)
i have never used Linux.. but alt + cmd + esc will open up force quit which will allow you to force quit an application. You can also do cmd+ [space] to open Spotlight and then you can open any app, including terminal.