Occasionally, answers on this site include software recommendations. Let’s assume a "legitimate developer" pretends to have created a useful application that is (secretly) capable of doing harm. How can I be sure I do not install something that contains malicious code or a root kit, especially if it is closed-source software?
Sadly, malicious code can be so small and obfuscated in an infinite number of ways so you can't simply make a rule to test a package to tell if it's good or bad.
Statistically, people have established some guidelines like trying to know the source (typing in the web address directly and not trusting a link, not downloading it from an alternate site, insisting that it be code-signed and verifying the package check sum before installing executables), but these are indirect remedies.
Apple maintains a page with links to most security information here:
Additionally, you could insist that all software that you run is signed by a legitimate developer to reduce the likelihood that someone has added a key logger to the game you just downloaded.
Lastly, after several delays, Apple has released GateKeeper to effectively sandbox software so that it has to explicitly list the things that it does (like access the address book, or modify files without user interaction) so that you can only let the system run code that obeys the sandbox rules. From a consumer perspective, you can prevent all code from running while you do whatever checks you feel are appropriate before allowing that code to run on your Mac.
Run Sophos and keep it up to date. It's free and it doesn't slow your system down like we've probably all experienced on windows machines. Just sits up in the menu bar, doing it's thing, minding it's own business.
There is no surefire solution, but there are some simple steps you can take that in practice are effective:
Turn on Gatekeeper.
It should be on by default, but if you want to be sure: go to System Preferences, then Security and Privacy, and then click "Mac App Store and identified developers". Like this:
Try to get your software through the Mac App Store, whenever possible. This is much safer than getting software through the web.
When installing an app through the Mac App Store, take a quick look at the reviews, the number of people who have installed it, and the brand. If it has bad reviews or very few reviews, and if very few people have installed it previously, you might want to be cautious. If in addition you've never heard of the brand, you might be even more careful.
While there are no absolute guarantees in life, software that has good reviews and many installs is very unlikely to be malware. It is very rare for malware to make it onto the market, and when it has happened, it tends to get removed fairly quickly -- and I would expect this to be especially true for widely used software.
If you are forced to install software from outside the Mac App Store, be extra careful. This is your primary risk factor.
Only download software using https. Check that the browser address bar shows https (not http) and/or has a nice background glow.
Only install software from trusted sources: i.e., major brands that you recognize or who have been recommended to you by folks you trust to give you good computer advice.
Don't install sketchy stuff. Don't install pirated software. Don't install free file-sharing software. Don't download and install stuff off of bulletin boards and file-sharing sites. These applications are at much greater risk of containing malware, spyware, or other bad stuff.
If you are configuring a computer for someone who is not very software-literate, who you are worried might get fooled into installing bad stuff, and who probably won't need to install new software off the web for any legitimate purpose, you could configure Gatekeeper to only allow applications downloaded from the Mac App Store. This will provide extra protection against malware, at some cost in flexibility in where you can get your software from.
Make regular backups. Install and enable Time Machine. It is a simple and painless way to keep good backups. And good backups are great, last-ditch safeguard if you should suffer security problems: while you don't want to get infected by malware (it's a hassle and no fun), if it does happen, at least you should have backups of all your data.
Don't bother with anti-virus software. It doesn't add enough value. If you use Mac OS X's built-in protections, and exercise some good judgement when installing software, you are very likely to be fine.
And, don't get overwhelmed or too scared by the malware stuff. By selecting a Mac, you're already at significantly lower risk than most other users. And, by following the advice above, you've further reduced your risk. While nothing in life is risk-free, the risk is pretty low. Use good judgement, and you will almost surely be fine.