EDIT: This seems to be a popular landing question for users looking to remove the delay on the caps lock key in OS X. As of OS X Mojave,
Go to System Preferences; Accessibility; Keyboard; enable Slow Keys and go into options; turn the delay down to the minimum. The only unwanted side-effect I've noticed so far is slow backspace action when holding it down. Generally I use CMD+A / CMD+SHIFT+L/R / CMD+SHIFT+CTRL+L/R anyway so it's not a big issue.
As of El Capitan and earlier,
The solution is quite elusive, and you wouldn't really know you removed it unless you were specifically trying to get rid of it in the first place. The way I am going to show you is purely for the (current)latest version of OSX, Yosemite. However you can absolutely apply this method to previous and future versions.
The solution is simple. If you navigate to Keyboard in the System preferences via the apple logo in the top left hand corner you will reach this screen
If you click the modifier keys button, you are able to change the function of each of the keys which are programmable. All you have to do is set the caps lock key to no action and press ok to bring you back to the keyboard menu. Once done, go back into the modifier keys and change the caps lock key back to caps lock and this will remove the delay! Do note that this fix remains in place until you sleep, restart or power down the device. At which point the delay is reinstated.
They're strange fixes, and it begs the question why do they not provide an option to remove the delay when this fix is purely software based. But hey, at least there is a way!
Happy capsing.