After Security Update 2019-003, OpenLDAP is corrupted. How can I recover?

Question

I installed the Security Update 2019-003 this morning. The update went awry, and produced on (postgres) induced panic on the host, a second attempt succeeded. After the update, slapd will not launch anymore.

I've tried restoring /private/var/db/{krb5kdc,openldap,auth} from a Time Machine backup (through Cmd-R reboot, including temporarily turning SIP off) but whatever I do, I can't get it running again. db_recover won't help either. slapd testing produces:

bash-3.2# /usr/libexec/slapd -T test
5cdfeed8 bdb_monitor_db_open: monitoring disabled; configure monitor database to enable
5cdfeed8 bdb_db_open: database "cn=authdata": unclean shutdown detected; attempting recovery.
5cdfeed8 bdb_db_open: database "cn=authdata": recovery skipped in read-only mode. Run manual recovery if errors are encountered.
config file testing succeeded
bash-3.2# slaptest -v
5cdfef28 bdb_monitor_db_open: monitoring disabled; configure monitor database to enable
5cdfef28 bdb_db_open: database "cn=authdata": unclean shutdown detected; attempting recovery.
5cdfef28 bdb_db_open: database "cn=authdata": recovery skipped in read-only mode. Run manual recovery if errors are encountered.
config file testing succeeded

How can I recover this from my Time Machine backup without completely restoring the server and overwriting everything else?

Apparently the restore I tried (copy openldap over from a Time Machine backup) did not work as Postgres DB-recovery balks at what is in authdata:

bash-3.2# db_recover -cv -h authdata/
Finding last valid log LSN: file: 31 offset 174121
Recovery starting from [30][28]
db_recover: Log sequence error: page LSN 23 1709981; previous LSN 30 28
db_recover: Recovery function for LSN 30 8130 failed on forward pass
db_recover: PANIC: Invalid argument
db_recover: process-private: unable to find environment
db_recover: DB_ENV->open: DB_RUNRECOVERY: Fatal error, run database recovery
bash-3.2# db_recover -cv -h openldap-data/
Finding last valid log LSN: file: 1 offset 1754044
Recovery starting from [1][28]
Recovery complete at Sat May 18 13:45:46 2019
Maximum transaction ID 80000cad Recovery checkpoint [1][1754044]

Answer 1

I have been able to repair this. It may have been luck. What I did:

1. Copy all private/var/db/openldap/authdata from all Time Machine backups to a working directory (for i in 2019*; do ditto $i/DumbledoreRoot/private/var/db/openldap/authdata/ /tmp/$i/authdata; done). My boot volume is called DumbledoreRoot.
2. Run crash recovery on all the authdata copies, to see which one are recoverable (for i in 2019-0*; do echo $i; db_recover -cv -h $i/authdata/; done)
3. Most would report an error. This is not strange, Time Machine cannot really backup this data as backing up a live database mostly gets incorrect results in the backup. I do wonder how restore from a Time Machine does this, though. There must be a better way.
4. Replace /private/var/db/authdata with what is in that backup. I probably should have copied openldap-data as well, given what happened next
5. Ran slapd -T test and noticed there still was some corruption (though different):

Output:

bash-3.2# /usr/libexec/slapd -T test
5cdff90d bdb_db_open: database "dc=Dumbledore,dc=local": unclean shutdown detected; attempting recovery.
5cdff90d bdb_db_open: database "dc=Dumbledore,dc=local": recovery skipped in read-only mode. Run manual recovery if errors are encountered.
5cdff90d bdb_monitor_db_open: monitoring disabled; configure monitor database to enable
5cdff90d bdb_db_open: database "cn=authdata": unclean shutdown detected; attempting recovery.
5cdff90d bdb_db_open: database "cn=authdata": recovery skipped in read-only mode. Run manual recovery if errors are encountered.
config file testing succeeded

Despaired. Then on a whim, started Open Directory from Server.app and to my surprise, it started and my network user accounts were back.

So, I'm probably just lucky and better answers may help future problems.