In the user Library, there is a Developer folder with about 3.2 GB of space (2.8GB from DocSets and the rest mainly from Xcode/iOS DeviceSupport). I had Xcode installed, I presume these are from that? I no longer use Xcode, is there any downside to erasing these folders? If I install Xcode again wouldn't it restore what is needed?
If the only two directories you have in ~Library/Developer are Xcode and Docsets (or Shared/Docsets) then you should be safe to delete that folder. If you did re-install XCode you would get them (or updated versions) back.
No its not safe to delete everything in:
Doing so could cause you to loose backup snapshots, any generated Archives for applications and other customizations set in the XCode IDE.
Specifically within the Developer folder there should be two additional folders.
"Shared" and "Xcode"
Shared contains documentation that you have downloaded using Xcode, depending on how you work and whether you have internet all the time you might not want to trash that, but worst case you can just re download it, so if you need space, and are not concerned about offline documentation, here is where you can grab back some space.
The Xcode folder contain a lot, and potentially some things you might not want to loose.
If you use Xcodes Archives organizer feature, deleting this will wipe out any Archives you may have.
Contains information probably about using iTunes Connect in Xcode, I would guess its disposable, but you may have to reconfigure Xcode to talk to Apple for App Store distribution, etc.
This generally contains any changes you made to Xcode color schemes and font themes as well as any changes you may make regarding Key Bindings in Xcode, again this stuff should be pretty small and is probably something you don't want to throw away if you spent any amount of time tweaking those kinds of Xcode settings, you probably just don't want to trash those changes made.
Probably some information relating to iOS devices that you have attached to and tried to use in Xcode, should be safe to delete.
Stuff in the DerivedData folder is generally pretty safe to delete, it is all the stuff that gets generated every time you press the build button in a project. It includes the generated log and applications executables and libraries for all the products you build in a project.
If you use Xcodes built in snapshot tools as a means of version control, wiping out your snapshots will wipeout the possibility of reverting to older snapshots.
Safe to delete, unless you need your logs, you would know if you need to keep them.