For example, I would like for iTunes to play thru my iMac's speakers, but iChat's notifications thru my headphones.
If the software you're using doesn't support specifying an an audio output, you'll have to rely on a third-party utility. Someone else already mentioned Ambrosia's WireTap products, but there is one other I'm aware of: Audio Hijack Pro by Rogue Amoeba. At $32, it beats the pants off WireTap for price.
There's also a piece of software called SoundSiphon by Static Z Software that provides the same functionality as Soundflower, plus single-application audio capture. Priced at $29.
Unfortunately, there are no free utilities for this. Not even Soundflower can capture audio from individual pieces of software. I think Rogue Amoeba's relies on the same underlying magic that allows them to hijack audio from individual pieces of software for their AirPlay utility AirFoil (which allows you to route any application's audio to any AirPlay device, as well as any device running their AirFoil Speakers app).
The easiest way to do this depends on the application supporting it;
Some applications have a sound output (and input) picker that allows you to choose the output device, like the official Last.fm app.
(My apologies for the poor example considering I only have one output source.)
If an application is written to support it, you can simply change the device as shown above. Otherwise you'll have to rely on premium (paid) apps, such as one of Ambrosia SW's "Wiretap" series of apps. Though those are pricy, $69 for WireTap Studio and $129 for WireTap Anywhere.
For iTunes specifically, you can install
shairport-sync via macports.
If you run it as
shairport-sync -- -n 'your desired output device'
This will appear as a fake airplay audio server that you can connect to in iTunes (it's name will match your computer's hostname by default), which will output to whatever audio device you set it to. If you connect iTunes to the airplay server, iTunes is able to play to a different device then the rest of your computer.