On Mountain Lion, I've got Gatekeeper enabled and set to allow Mac App Store apps only. Now: if I download any app from the internet (and I'm talking about apps I've never run before) it just runs, Gatekeeper seems to be not working at all. Does anybody know what's happening?
As bogdansrc mentioned, Gatekeeper only applies to application bundles that have a
com.apple.quarantine extended attribute. Some applications (like μTorrent) don't add them to the files they download, and some unarchiving programs (like
unzip) don't add them to files after extracting archives.
You can list extended attributes with
Gatekeeper is also disabled if you have disabled
LSQuarantine with for example
defaults write com.apple.LaunchServices LSQuarantine -bool false.
It should not run them. However, what browser are you using? Can you replicate this if you download the app with Safari?
In order for GateKeeper to work, the browser you downloaded the app with needs to set a "internet downloaded" flag on the file. This way, GateKeeper knows that you have downloaded the app from the Internet.
P.S. This should give you an idea on how "secure" GateKeeper is.