UPDATE: Lion's FileVault 2 uses full-volume encryption rather than just home-directory, so it actually does make some of these issues go away and changes several others. I've added notes in [] below. Note that upgrading to Lion does not automatically convert to FV2; you must first turn off "Legacy FileVault" encryption for accounts using the old system, then turn on VF2 to reencrypt.
Speed: obviously, the computer has to decrypt the data to read it from disk, and encrypt it before reading from disk, which'll slow down disk access. This usually isn't a big deal unless you're doing something like video editing. There's also a delay at logout as it recovers space in the image (and maybe also backs it up; see below). [VF2 removes the logout delay, and can use hardware AES instructions (on CPUs that support them) to minimize the general slowdown.]
Too much security: the good thing about FileVault is that nobody can get at your data without your login password (of the computer's master password). The bad thing is that if you forget your password (and don't remember/haven't set a master password), you can't get at your data either. The recommended procedure is to set a master password (before enabling FileVault), write it down, and store the written copy someplace secure (e.g. sealed envelope in a safe deposit box). [VF2 creates a recovery key, which functions very similarly to the old master key. There's also an option to have Apple store the recovery key for you, although I'd recommend using that in addition to (not instead of) storing the key yourself.]
File non-sharing: FileVault security prevents you from sharing files out of your home folder. Your Public folder, normally accessible by other users on the computer (and if you have file sharing turned on, other computers on the network) is locked in the FileVault and available only to you. [FV2 removes this limitation.]
Risk of data corruption: normally, if your computer crashes there's a possibility that whatever files you had open could get corrupted; with FileVault, your entire home folder is contained in a disk image, and if that gets corrupted you could lose everything. IIRC both OS X v10.3.0 and 10.4.0 shipped with bugs that could corrupt the types of images used by FileVault (fixed in 10.3.1 and 10.4.1, but still...). So backup is very important if you have data you don't want to lose. [FV2 does not have the image corruption issue, but is still vulnerable to normal volume corruption; it will also limit your options for data recovery somewhat, as not all volume repair/recovery tools support encrypted volumes (yet).] Speaking of which:
Backup time: backing up the FileVault image file is generally only possible when you're logged out. External volumes (like backup drives) get dismounted when nobody's logged in, creating a bit of a catch-22 for many backup systems. Time Machine gets around this by doing a backup as you log out (after the image becomes accessible, but before the backup drive gets dismounted), so make sure you log out with the backup drive attached from time to time. Other backup systems... who knows. In any case, test to make sure you're actually getting a useable backup. [FV2 removes this limitation. Note that Time Machine under Lion supports encrypting the backup volume, in which case it can only back up when the backup volume is mounted and decrypted.]
Backup capacity: backup systems generally save space by only backing up files that've changed since the last backup. With your files hidden in a disk image, this isn't really possible. The image is stored as a series of "band" files (in "sparse bundle format"), so the backup system only needs to store the bands that changed, but since these don't neatly correspond to specific files in your home folder, it's still going to use more space than if it could back up the files directly. [FV2 removes this limitation.]
Backup recovery: since the files you're likely to want to recover from a backup are hidden inside the encrypted image, they're a lot less convenient to recover from backup. When using Time Machine, its slick view-through-time interface doesn't work with FileVault-protected files, you have to find the backup of the image, mount it, and recover the file(s) you want by hand. Similarly, if you want to recover a file from an online backup, you're likely to have to recover the entire image so you can mount it and pull out the file(s) you want. [FV2 removes this limitation.]
One more note: to keep people from bypassing FileVault, make sure you have secure (encrypted) virtual memory enabled (in System Preferences -> Security pane -> General tab), and either log out consistently (yeah, right) or require a password immediately after sleep or screen saver (same place) and set the screen saver to run automatically when the computer's been idle for a while. There is an option to automatically log out after a period of inactivity, but this doesn't always work (a program with unsaved documents will generally prevent logout). [FV2 encrypts the virtual memory swap along with everything else, although the advice about locking or logging off still applies. Also, 10.7.0-10.7.1 were vulnerable to FireWire DMA attacks in some states, but this appears to have been fixed in 10.7.2.]
