Definitely sounds like its fragmentation-related.
I found a few sources, take with a grain-of-salt but it seems to back up the hypothesis.
My earlier post:
Edit: Thank you for the info.
Some additional info could help with this:
- iPhone release (orig, 3g, 3gs etc...)
- codec of audio: (mp3? apple lossless?
Without the additional info, my first hypothesis would be that its a memory issue. You might try killing some other apps (yes, I know they shouldn't be running, but there may be some related processing happening in the background.)
To take it to another extreme for testing purposes, do you have video clips you could try? If so, test them out and come back if there's any useful information gathered.
One last thing, I doubt its related but, how full is the storage on the iPhone?