//// .. title: Too Many SSH Keys .. slug: ssh-too-many-keys .. date: 2021-03-12 12:24:28 CET .. tags: linux, ssh .. category: .. link: .. description: .. type: text //// :source-highlighter: pygments Working with development boards in my local network, 'ssh' started to act up on me when I try to log into one of them: [source,console] ---- dzu@krikkit:~$ ssh user@ls1012afrwy Received disconnect from 192.168.44.102 port 22:2: Too many authentication failures Disconnected from 192.168.44.102 port 22 dzu@krikkit:~$ ---- This blog post explains why this is caused by seemingly unrelated actions and how to fix it. Before going into the details I will examine the context leading up to this problem. pass:[] If you have a GNU/Linux system connected to the internet with an open SSH port, you will have seen the constant stream of breakin attempts that clog up bandwidth today. For example on one of my systems, the log shows 1257 failed login attempts from 92 different IP addresses for *yesterday* alone. And this is on a system running http://www.fail2ban.org/wiki/index.php/Main_Page[fail2ban] to block connection attempts after a few failed attempts. So very likely the numbers will be even higher without this basic precaution. .Photo by https://unsplash.com/@contradirony?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText[Samantha Lam] on Unsplash image::/images/many-keys.jpg["Many keys", align="center", width="400"] So even though brute force attacks are difficult with such a login rate limiter, it should be obvious that using passwords alone is not a good idea nowadays. So I started using ssh keys for remote logins many years ago. But just as I don't use a single password for all web sites, I have multiple ssh keys for individual uses. By default they will all be placed into the '~/.ssh' directory. Of course I could place them somewhere else, but I like the fact of having a central place for the ssh configuration that I can keep an eye on. But suddenly I started running into the error shown at the beginning. Even though the target system allows password authentication, 'ssh' simply disconnects before I even have a chance of entering the password: [source,console] ---- dzu@krikkit:~$ ssh user@ls1012afrwy Received disconnect from 192.168.44.102 port 22:2: Too many authentication failures Disconnected from 192.168.44.102 port 22 dzu@krikkit:~$ ---- Adding the option '-vv' to the 'ssh' command gives us more details of what is happening and we see that 'ssh' offers all the public keys it can find below '~/.ssh' to the remote system. It seems surprising at first, but of course every single public key unknown to the target system will be counted as an authentication failure. With too many such attempts, the remote 'sshd' will simply shut down the connection. Note that it does not matter if the key in question has already been unlocked through 'ssh-agent' or not. All potential public keys will be offered and if a match is found 'ssh-agent' will only prompt for the passphrase if it hasn't been entered already. With this in mind, let's check the target systems configuration. After reading the https://linux.die.net/man/5/sshd_config[man page for 'sshd_config'] we know that the relevant parameter is 'MaxAuthTries': [source,console] ---- user@ls1012afrwy:~$ grep MaxAuth /etc/ssh/sshd_config #MaxAuthTries 6 user@ls1012afrwy:~$ ---- The comment shows that the actual line is not effective but in this case the config file lists all the defaults as comments so it is easy to understand and adjust if so required. So with this default we will run into this exact problem once we have six or more ssh keys below '~/.ssh'. Reading the documentation of ssh, ssh-agent, etc. it seems like only keys with the default names of 'id_rsa', 'id_dsa', etc. should be offered by default, but I found that naming makes no difference. All key files with the correct permission (600) on the private key will be considered. It's not straightforward to check, but 'ssh-add -l' will list the fingerprints of all the candidates. I have not found a faster way to match them with file names then doing it by hand. To fix this situation, we can either increase this server side limit on every system we want to login, or we have to prevent ssh from offering all the keys it can find. As the former increases the attack surface, I want to do the latter. Checking the https://linux.die.net/man/5/ssh_config[man page of 'ssh_config'] we learn that there is an 'IdentitiesOnly' option that seems to do what we want. If we turn on this option, we should specify all the keys we want 'ssh' to use by default in separate 'IdentityFile' directions. Below is an excerpt from my configuration file which also shows how to specify keys by target system. The last section instructs ssh to pick a specific key and user-id when talking to the github servers. [source,console] ---- dzu@krikkit:~$ cat ~/.ssh/config # ssh(1) config-file IdentitiesOnly yes IdentityFile ~/.ssh/id_dsa [....] Host github Hostname github.com User git IdentityFile ~/.ssh/id_github dzu@krikkit:~$ ---- Following this example it should be easy to align individual keys with their respective intended service like github, gitlab, codeberg, etc.