How to identify a missing datastore for com.apple.securityd?

Question

My fans are blowing constantly, and some investigation of my Console shows this error being spit out at a fantastic rate

CSSM Exception: -2147413737 CSSMERR_DL_DATASTORE_DOESNOT_EXIST

and

dbBlobVersion() failed for a non-existent database

The following applications are producing it, but I seem to remember a few more during different log watching sessions:

• Twitter
• com.apple.WebKit.Networking
• 1Password
• accountsd

I've tried the following things to work the problem:

1. Some digging showed this error might be related to a Keychain issue. I completely nuked my Keychain in an attempt to get it to rebuild whatever was missing to no avail.

   a. I have a crlcache.db reference that appears dead, but no number of deletion attempts makes it go away.

   b. Before nuking everything, I tried wiping away dead certificates without luck.

2. An Instruments run with File Activity watching the Twitter process in hopes of finding out which file it thinks it is looking for (too noisy and I'm not sure what I'm looking for).

3. A full over-install of a fresh 10.12.4 Sierra download burned 30 minutes but did nothing else to remedy the situation.

4. A Recovery-Mode Disk Repair First Aid session did find a catalog issue with my disk (late 2013 1tb SSD) and successfully repaired it. This is possibly related to whatever file is missing, but no log or notice tells me which file it is looking for.

Any tips for other things to try? I'm running Sierra 10.12.4 on a Late 2013 15" MBP with a 1tb SSD.

Answer 1

Resolution

a. I have a crlcache.db reference that appears dead, but no number of deletion attempts makes it go away.

This was the root problem, but it was difficult to make it go away with any of the existing tools. The crlcache.db entry was ghosted in my Keychain Access application, so an entry still existed. While I had reset all of my passwords, I hadn't fully killed the Keychain. All the applications I listed were using the Keychain to find their information, hitting crlcache.db and then either retrying or throwing I had to manually remove both of these files (essentially a hard-reset of the entire Keychain):

~/Library/Preferences/com.apple.security.plist
/Library/Preferences/com.apple.security.plist

Diagnosis

It was very difficult to diagnose the issue because nothing would tell me which file did not exist. This comment, with its command that gathered bug information for Apple, was the most helpful. This produced a giant tar.gz with lots of diagnostic goodies that told me much more about what was going on. Make sure to run along with whatever application(s) is misbehaving at the time.

sudo sysdiagnose securityd

Among the many plain text debug output files it produced, there was a large one called fs_usage.txt, and when I opened it up, I could see 1000s of familiar entries

08:01:11.999993  getattrlist                            /private/var                                                                                                                                                          0.000003   Twitter.3616
08:01:11.999996  getattrlist                            /private/var/db                                                                                                                                                       0.000003   Twitter.3616
08:01:11.999998  getattrlist                            /private/var/db/crls                                                                                                                                                  0.000003   Twitter.3616
08:01:12.000000  getattrlist            [  2]           /private/var/db/crls/crlcache.db                                                                                                                                      0.000002   Twitter.3616
08:01:12.000004  statfs64                               /private/var/db/crls      

Once I saw that, it was clear the Keychain was still the issue, and my ghosted entry had to go. Lacking the knowledge of how to perform laparoscopic surgery on the plist files, I simply amputated and started over.

Answer 2

In my case after removing Espionage it left a reference for its .keychains file in Keychain Access. Log entries were like these:

default 15:11:56.715938 +0200   com.apple.WebKit.Networking DbOpen of /Volumes/1Tb/Users/MyUser/Library/Application Support/Espionage/Espionage.keychain
default 15:11:56.716702 +0200   com.apple.WebKit.Networking open /Volumes/1Tb/Users/MyUser/Library/Application Support/Espionage/Espionage.keychain: No such file or directory
default 15:11:56.716765 +0200   com.apple.WebKit.Networking CSSM Exception: -2147413737 CSSMERR_DL_DATASTORE_DOESNOT_EXIST
default 15:11:56.716848 +0200   com.apple.WebKit.Networking CSSM Exception: -2147413737 CSSMERR_DL_DATASTORE_DOESNOT_EXIST
default 15:11:56.716910 +0200   com.apple.WebKit.Networking CSSM Exception: -2147413737 CSSMERR_DL_DATASTORE_DOESNOT_EXIST
default 15:11:56.716952 +0200   com.apple.WebKit.Networking dbBlobVersion() failed for a non-existent database

The resolution was rather simple: had to remove the database reference in Keychain Access using the menu entry in File called Delete Keychain Espionage. You should try to find the missing file reference in a DbOpen line before the CSSM Exception lines to see what's missing.