First of all let me tell you that there is definitely a way of removing the caps lock delay. I myself prefer to use caps lock over shift just from a comfort perspective, with caps lock being closer to my pinky finger than shift the majority of the time.
The solution is quite elusive, and you wouldn't really know you removed it unless you were specifically trying to get rid of it in the first place. The way I am going to show you is purely for the (current)latest version of OSX, Yosemite. However you can absolutely apply this method to previous and future versions.
The solution is simple. If you navigate to Keyboard in the System preferences via the apple logo in the top left hand corner you will reach this screen 
If you click the modifier keys button, you are able to change the function of each of the keys which are programmable. All you have to do is set the caps lock key to no action and press ok to bring you back to the keyboard menu. Once done, go back into the modifier keys and change the caps lock key back to caps lock and this will remove the delay! Do note that this fix remains in place until you sleep, restart or power down the device. At which point the delay is reinstated.
It's a strange fix, and it begs the question why do they not provide an option to remove the delay when this fix is purely software based. But hey, at least there is a way!
Happy capsing.