2

I have been struggling for hours to make sense of a specific failure from connecting to an SSH server from one recent macOS system.

ssh user@machine -p 22

I have added all keys to ~/.ssh/authorized_keys on the target machine. The keys have been generated with default settings for ssh-keygen.

The server is a freshly provisioned Ubuntu 18.04 instance on Azure, we also tried a different cloud provider.

I can successfully connect with public/private key authentication (PKA) from an Ubuntu machine. I can't connect to port 22 with public/private key authentication (PKA) from the macOS machine in question.

What am I missing?

Symptoms

For connecting from the macOS machine we observe the following symptoms:

  • Connection to the server works with password authentication
  • Connection to the server does not work with PKA
  • When I start a second SSH server using /usr/sbin/sshd -d -p 2222, connection to the server works with PKA (using ssh -p 2222 on the client)
  • Same for ports 23, 80
  • Same behavior using
    • the built-in SSH client
    • a freshly brewed one (version 8.1p1 with brew install openssh)
    • cyberduck

Client logs

I checked the logs of ssh -vvv for both ports. The only difference (apart from host key differences) are the following lines which are missing when using port 22 (this makes it very hard to search for the problem):

debug3: put_host_port: [machine]:2222\
...
debug3: put_host_port: [ip-address]:2222\
debug3: put_host_port: [machine]:2222\
...
debug3: receive packet: type 7\
debug1: SSH2_MSG_EXT_INFO received\
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>\

It appears that, when connecting to port 22, the server fails to send (or the client fails to receive) a "packet of type 7" with SSH2_MSG_EXT_INFO and kex_input_ext_info.

Client config

The output of ssh -G (which shows the effective configuration) is identical for port 22 and for other ports.

Server logs

When looking at the output of /usr/sbin/sshd -ddd, the following entry is missing for port 22:

debug3: send packet: type 7 [preauth]

The server doesn't send a "packet type 7"; but perhaps this is just an unrelated symptom?

I also see the following differences:

  • Port 22:

    debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1 [preauth]
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa [preauth]

  • Port 80:

    debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c [preauth]
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa [preauth]

    Extra entries (compared to port 22):

    • KEX algorithms: ext-c-info
    • host key algorithms: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com

    Different order:

    • ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,

Workaround

For machines that are under my control, I can change the default port of the SSH daemon.

Question

What else can I try to narrow down the problem?

Improve this question
7
  • Are the Ubuntu machine and the Mac on the same network? (i.e. could anything on your network, internet connection, or similar be interfering with your traffic)
    – jksoegaard
    Nov 22, 2019 at 11:12
  • Thanks. I don't think this is the problem, because the connection works from the Mac with password authentication.
    – krlmlr
    Nov 22, 2019 at 11:20
  • 1
    @krlmlr Setting this up in my env (Catalina ssh-client > El Capitan ssh-server) this works w/o a problem. Interestingly I don't see any receive packet: type 7 in my ssh -vvv output....
    – klanomath
    Nov 22, 2019 at 11:27
  • 1
    @krlmlr I found a related question w/o real answer with a similar problem: ...it turn out that there is a firewall/gateway enforcing the particular policy which only accept Password authentication for Port 22 in the front of the network. It has been applied on the company network which affect easier to reproduce here. It actually work as soon as the policy is taken out and by using the existing configuration. This doesn't really fit with your finding ...I can successfully connect with public/private key authentication (PKA) from an Ubuntu machine... but maybe they are in diff. networks
    – klanomath
    Nov 22, 2019 at 11:54
  • 1
    @krlmlr [char limit had struck] but maybe they are in diff. networks I meant: the Mac ssh client (no successful PKA) and the Ubuntu ssh client (successful PKA) are in different networks (or different policies apply) and trying to access the remote Ubuntu ssh server (via PKA) they behave differently.
    – klanomath
    Nov 22, 2019 at 12:06

1 Answer 1

Reset to default
2

The problem here seems to be that you have "something" interfering with your network traffic that is specifically blocking out your attempt to authenticate via keys on port 22.

This "something" could be so called "endpoint protection" software (software firewall), it could be a network device on your network (i.e. router with deep packet inspection, firewall, etc.) or it could be a network device on the network path or at the server end.

This seems to be the most logical conclusion as this can explain why it doesn't work on port 22, but do work on other ports.

I.e. if you hadn't included the port information, I would have looked at the authorized_keys file on the server. It is possible to specify that a specific key can only be used for authentication when connecting from a specific host - or that it must not be used when connecting from a specific host. If you had set that up to allow the Ubuntu host and not the Mac host - that could explain parts of what you're seeing. However, this method does not include matching again the port number - so it cannot explain that.

This is all assuming that you're running the exact same configuration for port 22 and port 2222 (which you stated you are), and that you're running standard OpenSSH software on the server. If you're not, that software might include configuration to discriminate based on ports - but I don't think it is likely.

Improve this answer
2
  • Thanks a lot. I did try a different network the other day, but it turned out that the connection was still routed via the evil network with deep packet inspection. Also, I wasn't aware that SSH could be DPI-ed, but at the connection stage this does seem feasible. The other question I have is why someone would implement such a device.
    – krlmlr
    Nov 22, 2019 at 12:53
  • I don't know who runs your "evil network", so you haven't given any information we can work with... but one guess is that it is a company network where they check user/password for SSH in a central database/LDAP/RADIUS/whatever at the "firewall" (i.e. network device) in addition to at the actual SSH server. As no passwords are used with key based authenticaction, they could have blocked that to ensure that passwords always are used. Some people do the weirdest things.
    – jksoegaard
    Nov 22, 2019 at 13:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .