You can remove it, but my experience has been that you'll eventually want it again, and even as an apple developer you may lose access to it in the future as they upgrade xCode.
It's not a big deal if you don't plan on maintaining your older apps, of you will always keep them updated and move them to new versions of xCode as they come out, but the transition from xCode 3 to 4 was hard for many developers, and even now some software is developed in xCode 3 to support kernel drive development for OS X 10.4, which is no longer fully supported in xCode 4.
Generally your best option is to store that installation app somewhere you can get to it later. It doesn't need to be on your mac all the time, but you may find yourself needing it down the road if you do develop anything in it you might have to update later.