I set up a server running Postfix + Dovecot w/ SASL auth and SSL everywhere (actually, STARTTLS). It's about as "bog-standard" as you can get (meaning I followed one of the innumerable cookbooks on the topic).
My outbound SMTP uses port 587 and IMAP is on 993. The auth mechanism is PLAIN. Like I said, STARTTLS everywhere.
Connections work fine - sending and receving - and as expected in: Postbox, Thunderbird, Mail Pilot, Mailmate, Airmail (all on Mavericks), iOS 7, whatever the recent Android candy flavor is, and Thunderbird on Windows 7.
Mail.app will not connect. It throws the warning triangle up immediately when you start Mail. Connection Doctor shows 'INITIATING CONNECTION' and the host IP with some hex strings but no other information.
Now here's the rub: before starting any mail client, I started a tcpdump filtering on the destination address. If I start any other mail client, or just do
openssl s_client -connect to the server address, I see packets flowing from here to there.
Mail.app never does anything to cause a packet to flow. No TCP packets ever leave my ethernet port bound for the destination server.
In short: what the hell is going on here?