I've only recently upgraded to Mavericks and hence I'm not entirely sure whether this is a Mavericks only problem or not, but here it is:
Security update 2006-007 apparently broke 8192-bit certificates on OS X and no one bothered to fix it (https://discussions.apple.com/message/3650856#3650856). However there is (or rather used to be) a solution for this bug (or feature?) and it is in https://discussions.apple.com/thread/2668985 You've just got to execute
sudo defaults write /Library/Preferences/com.apple.crypto RSAMaxKeySize -int 8192
in a Terminal prompt. Well I did, then I also restarted my macbook. However this had no effect other than setting the RSAMaxKeySize. Here is how I know the defaults command worked:
ingenious@book ~ $ sudo defaults read /Library/Preferences/com.apple.crypto RSAMaxKeySize 8192
However trying to import a (valid!) 8192-bit self-signed certificate into my system keychain (using Keychain Access) I keep getting the following error:
So, can anyone confirm this in 10.8.*? Is there any alternative way to make 8192-bit certificates work on OS X?
Edit: Could this thing actually be related to the new openssl in 10.9? (http://curl.haxx.se/mail/archive-2013-10/0036.html)