If the audio device you're using is always connected, please try these instructions (relevant portion below, although there are some other suggestions in that post that could be worth trying).
However, keep in mind that if the device is disconnected when your system starts, it will revert to the system default. (It is also possible, that if you are loading certain applications on startup, that they are resetting the preferred default, but we'll need more details on your system configuration/startup items in this case.)
You could also, depending on your situation, use an AppleScript method to create a one-click type approach to your audio selection.
Suggested first attempt to solve this issue:
Go to /Applications/Utilities and open Audio MIDI Setup. Select the device you want to use, click the gear, and then Use this device for sound output.