Found it, and it's fairly easy (as one would expect from macOS). I'll explain step by step.
How to store S/MIME certificates in Open Directory
- Launch the Directory Utility app, either directly from Spotlight, or (that's how I found it) by going through System Preferences > Users & Groups > (authenticate if necessary) Network Account Server - Edit (or Connect if you haven't yet) > Open Directory Utility
- Select the Directory Editor on the top
- Select Viewing: Users, if not selected by default, then click on a user, and the user's set attributes will be shown
- To add an attribute, by clicking on the lock above, authenticate to get into edit mode - note that, this time, you must authenticate with the directory admin credentials, because now we want to edit the directory entries
- Click on the + sign below the attribute list to add a new attribute
- As you will see, the userCertificate and userSMIMECertificate attributes are already available from the dropdown list
- Select one of them, then press OK
- When you click on the attribute, its data are shown below the attribute list
- You can toggle between Text and Data view – to add an S/MIME certificate, either copy paste into the text view or just drop the file in the Data view
- That's all!
I believe, the S/MIME certificate data should in PEM format, but I am not sure about it.
How to retrieve S/MIME certificates from Open Directory
In order to enable clients to retrieve S/MIME certificates from Open Directory (or any LDAP actually), here's how to do this on a Mac client:
Use the Directory Utility app, as explained above, to add the LDAP server to the search list on the client Mac (also explained here). Make sure that the LDAP server is added in the Search Policy list under Contacts. The Contacts server in the Server app need not be enabled.
After you have successfully added the LDAP server, open your Contacts app on the client. There should be a new entry Directory Services on the left sidebar. Click on it, then, in the Search field, enter the name of a user in your LDAP directory. The directory entry should then be displayed, and left of the email address, you should see a small certificate symbol similar to the symbol in Mail app that is displayed for signed emails. When you click on it, the S/MIME certificate retrieved for this user will be displayed.