From what is in /System/Library/User Template/Library, here is what ~/Library contains when you create a brand new user.
Almost all entries are empty directories.
Therefore, you can safely delete everything in ~/Library. Of course you'll lose all preferences you may have set, etc.
/Library
/Library/.localized
/Library/Application Support
/Library/Assistants
/Library/Audio
/Library/Audio/MIDI Drivers
/Library/Audio/Plug-Ins
/Library/Audio/Plug-Ins/Components
/Library/Audio/Plug-Ins/Digidesign
/Library/Audio/Plug-Ins/VST
/Library/Audio/Sounds
/Library/Audio/Sounds/Alerts
/Library/Audio/Sounds/Banks
/Library/ColorPickers
/Library/Compositions
/Library/Compositions/.localized
/Library/Favorites
/Library/Favorites/.localized
/Library/FontCollections
/Library/FontCollections/Fixed Width.collection
/Library/FontCollections/Fun.collection
/Library/FontCollections/Modern.collection
/Library/FontCollections/PDF.collection
/Library/FontCollections/Traditional.collection
/Library/FontCollections/Web.collection
/Library/Fonts
/Library/iMovie
/Library/iMovie/Plug-ins
/Library/iMovie/Sound Effects
/Library/Input Methods
/Library/Input Methods/.localized
/Library/Internet Plug-Ins
/Library/Keyboard Layouts
/Library/Preferences
/Library/Preferences/.GlobalPreferences.plist
/Library/Preferences/com.apple.symbolichotkeys.plist
/Library/Printers
/Library/Screen Savers
/Library/Sounds
/Library/Voices