From what is in `/System/Library/User Template`, here is what `~/Library` contains when you create a new user.
Almost all entries are empty directories.
Therefore, you can safely delete everything in `~/Library`. Of course you'll lose all preferences you may have set, etc.


    /Library
    /Library/.localized
    /Library/Application Support
    /Library/Assistants
    /Library/Audio
    /Library/Audio/MIDI Drivers
    /Library/Audio/Plug-Ins
    /Library/Audio/Plug-Ins/Components
    /Library/Audio/Plug-Ins/Digidesign
    /Library/Audio/Plug-Ins/VST
    /Library/Audio/Sounds
    /Library/Audio/Sounds/Alerts
    /Library/Audio/Sounds/Banks
    /Library/ColorPickers
    /Library/Compositions
    /Library/Compositions/.localized
    /Library/Favorites
    /Library/Favorites/.localized
    /Library/FontCollections
    /Library/FontCollections/Fixed Width.collection
    /Library/FontCollections/Fun.collection
    /Library/FontCollections/Modern.collection
    /Library/FontCollections/PDF.collection
    /Library/FontCollections/Traditional.collection
    /Library/FontCollections/Web.collection
    /Library/Fonts
    /Library/iMovie
    /Library/iMovie/Plug-ins
    /Library/iMovie/Sound Effects
    /Library/Input Methods
    /Library/Input Methods/.localized
    /Library/Internet Plug-Ins
    /Library/Keyboard Layouts
    /Library/Preferences
    /Library/Preferences/.GlobalPreferences.plist
    /Library/Preferences/com.apple.symbolichotkeys.plist
    /Library/Printers
    /Library/Screen Savers
    /Library/Sounds
    /Library/Voices