Based upon the specs on Apple's website, a new iMac would provide a significant boost over your older hardware, however whether it's enough to justify it is a personal decision. The new iMac maxed out would have a 3.4 GHz quad-core i7 (turboboost to 3.9) compared to 2.8 quad-core, 32GB 1600MHz DDR3 ram vs 14GB 800 MHz DDR2 (yours also maxes at 32GB), SATA version 3(6GB/s) vs version 2(3GB/s), USB 3.0 vs 2.0, Thunderbolt vs. firewire 800, and increased (depends on your card) graphics performance from the NVIDIA GeForce GTX 680MX. So yes all that will outperform your current Mac Pro in its current arrangement and in a nice compact package with a fancy new 27" screen. But...
While all this sounds great, it's as far as you can go with it of course. You can upgrade no more past those specs on the iMac unlike the Mac Pro. I'm of the opinion that a refreshed or new Mac Pro is coming soon, so if it was me I would wait if possible. Even it was 6 months or so, I would wait. Only you know what you need however. If those specs seem good enough to you then do it. But if you need to be able to upgrade the graphics card, add a PCI-X card, want more than 32GB of ram in the future, need an optical drive, firewire(maybe), RAID...the list could go on... then waiting for the next Mac Pro is your best bet, but you probably already knew that.
Another option is going ahead and buying the iMac how you want it, use it until the next Mac Pro is released, then sell the iMac to recoup some cost after purchasing the new Mac Pro. It's more hassle, but possibly worth it if your current set up just isn't cutting it anymore. And since they both would likely have Thunderbolt, transferring data should be relatively quick.
Also...On the topic of upgrading, I'm not sure, but it may be possible that the motherboard of your Mac has an open slot for another Xeon processor (although you'd need a matched pair). So with some work you could potentially upgrade your existing hardware to have 8 cores. Might be worthing looking into... I don't know.
Mac Pro specs at Apple
2012 iMac Specs at Apple