I have been struggling for hours to make sense of a specific failure from connecting to an SSH server from one recent macOS system.
ssh user@machine -p 22
I have added all keys to ~/.ssh/authorized_keys
on the target machine. The keys have been generated with default settings for ssh-keygen
.
The server is a freshly provisioned Ubuntu 18.04 instance on Azure, we also tried a different cloud provider.
I can successfully connect with public/private key authentication (PKA) from an Ubuntu machine. I can't connect to port 22 with public/private key authentication (PKA) from the macOS machine in question.
What am I missing?
Symptoms
For connecting from the macOS machine we observe the following symptoms:
- Connection to the server works with password authentication
- Connection to the server does not work with PKA
- When I start a second SSH server using
/usr/sbin/sshd -d -p 2222
, connection to the server works with PKA (usingssh -p 2222
on the client) - Same for ports 23, 80
- Same behavior using
- the built-in SSH client
- a freshly brewed one (version 8.1p1 with
brew install openssh
) - cyberduck
Client logs
I checked the logs of ssh -vvv
for both ports. The only difference (apart from host key differences) are the following lines which are missing when using port 22 (this makes it very hard to search for the problem):
debug3: put_host_port: [machine]:2222\
...
debug3: put_host_port: [ip-address]:2222\
debug3: put_host_port: [machine]:2222\
...
debug3: receive packet: type 7\
debug1: SSH2_MSG_EXT_INFO received\
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>\
It appears that, when connecting to port 22, the server fails to send (or the client fails to receive) a "packet of type 7" with SSH2_MSG_EXT_INFO
and kex_input_ext_info
.
Client config
The output of ssh -G
(which shows the effective configuration) is identical for port 22 and for other ports.
Server logs
When looking at the output of /usr/sbin/sshd -ddd
, the following entry is missing for port 22:
debug3: send packet: type 7 [preauth]
The server doesn't send a "packet type 7"; but perhaps this is just an unrelated symptom?
I also see the following differences:
Port 22:
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1 [preauth] debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa [preauth]
Port 80:
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c [preauth] debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa [preauth]
Extra entries (compared to port 22):
KEX algorithms: ext-c-info
host key algorithms: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com
Different order:
ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,
Workaround
For machines that are under my control, I can change the default port of the SSH daemon.
Question
What else can I try to narrow down the problem?