Too Many SSH Keys

Working with development boards in my local network, ssh started to act up on me when I try to log into one of them:

This blog post explains why this is caused by seemingly unrelated actions and how to fix it. Before going into the details I will examine the context leading up to this problem.

If you have a GNU/Linux system connected to the internet with an open SSH port, you will have seen the constant stream of breakin attempts that clog up bandwidth today. For example on one of my systems, the log shows 1257 failed login attempts from 92 different IP addresses for yesterday alone. And this is on a system running fail2ban to block connection attempts after a few failed attempts. So very likely the numbers will be even higher without this basic precaution.

So even though brute force attacks are difficult with such a login rate limiter, it should be obvious that using passwords alone is not a good idea nowadays. So I started using ssh keys for remote logins many years ago. But just as I don’t use a single password for all web sites, I have multiple ssh keys for individual uses. By default they will all be placed into the ~/.ssh directory. Of course I could place them somewhere else, but I like the fact of having a central place for the ssh configuration that I can keep an eye on. But suddenly I started running into the error shown at the beginning. Even though the target system allows password authentication, ssh simply disconnects before I even have a chance of entering the password:

Adding the option -vv to the ssh command gives us more details of what is happening and we see that ssh offers all the public keys it can find below ~/.ssh to the remote system. It seems surprising at first, but of course every single public key unknown to the target system will be counted as an authentication failure. With too many such attempts, the remote sshd will simply shut down the connection. Note that it does not matter if the key in question has already been unlocked through ssh-agent or not. All potential public keys will be offered and if a match is found ssh-agent will only prompt for the passphrase if it hasn’t been entered already.

With this in mind, let’s check the target systems configuration. After reading the man page for sshd_config we know that the relevant parameter is MaxAuthTries:

The comment shows that the actual line is not effective but in this case the config file lists all the defaults as comments so it is easy to understand and adjust if so required. So with this default we will run into this exact problem once we have six or more ssh keys below ~/.ssh.

Reading the documentation of ssh, ssh-agent, etc. it seems like only keys with the default names of id_rsa, id_dsa, etc. should be offered by default, but I found that naming makes no difference. All key files with the correct permission (600) on the private key will be considered. It’s not straightforward to check, but ssh-add -l will list the fingerprints of all the candidates. I have not found a faster way to match them with file names then doing it by hand.

To fix this situation, we can either increase this server side limit on every system we want to login, or we have to prevent ssh from offering all the keys it can find. As the former increases the attack surface, I want to do the latter.

Checking the man page of ssh_config we learn that there is an IdentitiesOnly option that seems to do what we want. If we turn on this option, we should specify all the keys we want ssh to use by default in separate IdentityFile directions. Below is an excerpt from my configuration file which also shows how to specify keys by target system. The last section instructs ssh to pick a specific key and user-id when talking to the github servers.

Following this example it should be easy to align individual keys with their respective intended service like github, gitlab, codeberg, etc.