On OS X 10.8 I'm trying to record screen using Quicktime. But for some reason the resulting video resolution is only 1280 by 800, even thought I'm on Retina display.
How can I record screen in Retina resolution?
Quicktime obeys the Displays preference pane like most other applications will.
The first step to increasing the resolution for recording will be to adjust the settings to have a larger equivalent screen size as I have done in this picture (and you can take things to even larger pixel counts by selecting More Space which in turn makes items like the menu bar physically smaller when rendered).
If that's not sufficient for your needs, you may need to select a tool like the Screeny app to record the "full" resolution.
In my case, I have the 13 inch MacBook Pro with Retina, so it will capture the screen at 2880x1800 no matter what setting you have on the Displays Preference when you choose "full screen" recording.