My research so far has found XProtect appears at first glance to be a basic AV engine that scans DMG & apps through Safari, Chrome and Firefox. I'm looking to understand what does it actually do and what does it exactly monitor outside of DMG and apps? I've read it doesn't provide protection from BitTorrent and removable USB thumb drives. What else?
The Xprotect "system" is not really an AV engine as such. It looks for a very limited number of text strings, or "signatures," in specific types of files. For this reason, it is far less useful than a real AV program (up to you if you think one is necessary). In addition, it only checks files downloaded with certain apps: Safari, Mail, Messages, and third-party apps that have activated a setting to use Xprotect to check their files as well. Also, it only sets a flag on files, and depends on user interaction for protection. So an un-savvy user may see the dialog warning that a file is dangerous, but still click the necessary button to open it.
As you say, it doesn't protect against BitTorrent downloads (though specific BitTorrent clients could use the feature), and doesn't protect from files copied from any kind of removable media or files copied over a network.
This is Apple's explanation:
About the "Are you sure you want to open it?" alert (File Quarantine / Known Malware Detection) in OS X
OS X improves download validation by providing file quarantine in applications that download files from the Internet. This means that downloads are checked for safety (known malware) when you try to open them.
File quarantine-aware applications that download files from the Internet, or receive files from external sources (such as email attachments), attach quarantine attributes.
- Quarantine-aware applications include Safari, Messages, iChat and Mail.
- These attributes include date, time, and a record of where the file was downloaded from.
When you open a file received through a quarantine-aware application, OS X warns you where the file came from. You receive an alert asking, "Are you sure you want to open it?" You should click Cancel if you have any doubts about its safety.
If you have multiple user accounts on your Mac, the user account that downloaded the file is the only user account that can remove the quarantine attribute on a file. All other user accounts can open a quarantined file, but they are still presented with an alert asking "Are you sure you want to open it?" every time they open the file.
Known malware check
Mac OS X Snow Leopard v10.6 and later also check for known instances of "malware", or malicious software. When you open a quarantined file, OS X checks to see if it includes known malware. If so, an alert message similar to the following appears:
If you see "(file name) will damage your computer." You should click Move to Trash.
If the file is a disk image, you should click Eject Disk Image and then delete the source file.
Tip: Click the Help icon in the lower left corner of the alert message for more information about malware.
Blocking web plug-ins
To help limit exposure to potential "zero day" exploits from web plug-in enabled content, OS X also blocks specific versions of web plug-ins from functioning – including Java web apps, or Adobe Flash content. Typically an update to the web plug-in is available on the same day, or shortly after OS X blocks the web plug-in. Install the new update to restore web plug-in function.
OS X Lion v10.7.5 and later include Gatekeeper, a technology that allows developers to sign applications. Signed applications normally don't present an alert when you download and open them. Internet files downloaded from other applications get file quarantine attributes but without date, time, and link of the file downloaded.
Advanced users only
You can toggle the ability of File Quarantine to receive updates from Apple about malware and web plug-ins.
Important: Deselecting this option disables the ability to identify new malware, and leaves your Mac vulnerable to new malware without notification.