It seems that the only way to modify the trust settings via the security command is by the trust-settings-import option. The xml file output by security trust-settings-export would be a good place to start; but, it looks a little tricky.
Probably your root CA certificate is malformed, as your method works for me. Maybe your CA root certificate has not been generated with the expected properties. Try with -r trustAsRoot in your case, but I recommend to check your certificate generation.