There are a number of possible pitfalls to having an encrypted home folder; most of them are more-or-less intrinsic to the idea, so they're not likely to go away... ever.
Speed: obviously, the computer has to decrypt the data to read it from disk, and encrypt it before reading from disk, which'll slow down disk access. This usually isn't a big deal unless you're doing something like video editing. There's also a delay at logout as it recovers space in the image (and maybe also backs it up; see below).
Too much security: the good thing about FileVault is that nobody can get at your data without your login password (of the computer's master password). The bad thing is that if you forget your password (and don't remember/haven't set a master password), you can't get at your data either. The recommended procedure is to set a master password (before enabling FileVault), write it down, and store the written copy someplace secure (e.g. sealed envelope in a safe deposit box).
File non-sharing: FileVault security prevents you from sharing files out of your home folder. Your Public folder, normally accessible by other users on the computer (and if you have file sharing turned on, other computers on the network) is locked in the FileVault and available only to you.
Risk of data corruption: normally, if your computer crashes there's a possibility that whatever files you had open could get corrupted; with FileVault, your entire home folder is contained in a disk image, and if that gets corrupted you could lose everything. IIRC both OS X v10.3.0 and 10.4.0 shipped with bugs that could corrupt the types of images used by FileVault (fixed in 10.3.1 and 10.4.1, but still...). So backup is very important if you have data you don't want to lose. Speaking of which:
Backup time: backing up the FileVault image file is generally only possible when you're logged out. External volumes (like backup drives) get dismounted when nobody's logged in, creating a bit of a catch-22 for many backup systems. Time Machine gets around this by doing a backup as you log out (after the image becomes accessible, but before the backup drive gets dismounted), so make sure you log out with the backup drive attached from time to time. Other backup systems... who knows. In any case, test to make sure you're actually getting a useable backup.
Backup capacity: backup systems generally save space by only backing up files that've changed since the last backup. With your files hidden in a disk image, this isn't really possible. The image is stored as a series of "band" files (in "sparse bundle format"), so the backup system only needs to store the bands that changed, but since these don't neatly correspond to specific files in your home folder, it's still going to use more space than if it could back up the files directly.
Backup recovery: since the files you're likely to want to recover from a backup are hidden inside the encrypted image, they're a lot less convenient to recover from backup. When using Time Machine, its slick view-through-time interface doesn't work with FileVault-protected files, you have to find the backup of the image, mount it, and recover the file(s) you want by hand. Similarly, if you want to recover a file from an online backup, you're likely to have to recover the entire image so you can mount it and pull out the file(s) you want.
One more note: to keep people from bypassing FileVault, make sure you have secure (encrypted) virtual memory enabled (in System Preferences -> Security pane -> General tab), and either log out consistently (yeah, right) or require a password immediately after sleep or screen saver (same place) and set the screen saver to run automatically when the computer's been idle for a while. There is an option to automatically log out after a period of inactivity, but this doesn't always work (a program with unsaved documents will generally prevent logout).
