There method isn't as automated as other options, but it has the advantage that it finds files associated with an app even if they aren't used during the monitoring session.
I use an app called Find Any File, which lets you search your entire hard drive easily using some complex queries.
This method relies on the assumption that all app files either have the name of the app in the filename, or are stored somewhere with the name of the app in an enclosing folder's name. This is almost always true; Application Support, login items/launchagents/launchdaemons, caches, the app bundle itself, etc all follow it.
I've yet to find an app with files in non-obvious places that don't follow this rule.
Here's how I would formulate my query for the app Balsamiq Mockups, which I installed a few days ago:

The reason I've got the two parts of the name separately is to cover cases where there is a space between the two words (like the app bundle) as well as where there isn't (like a com.developer.AppName
file).
The Modification Date is within the past ...
component is optional and only helpful when the app has been installed relatively recently. Generally, just the app name should be just fine.
The results show all the files that the app has created:

I've got the view in hierarchical mode to show the location of the files, but either view mode is fine.
If you want to delete the files, switch to the list view, select them all, right click and choose Move to Trash.

Bonus: If the app has a custom file extension, you can easily round up all of those files too: Name ends in .ext