Broken Recovery Mode in Ubuntu 16.04 LTS

Ubuntu 16.04 LTS is the first long term Ubuntu version using systemd as the init process. Unfortunately the switch from upstart to systemd broke the recovery mode that comes in handy in a number of situations.

Just as many other distributions, Ubuntu offers a GRUB option to directly enter rescue mode on boot up. Having used this facility for a while in Ubuntu 14.04, I did not expect any problem after my recent upgrade to Ubuntu 16.04 LTS (Xenial Xerus). Well, actually I use Ubuntu GNOME, but this should not make any difference here.

Entering recovery mode in Ubuntu leads to a whiptail menu of several options. Usually I only care for the "root - Drop to root shell" choice and take it from there. So I did a few days ago and was severely surprised when the system started outputting boot progress messages after a while and seconds later again presented the whiptail menu.

It became quickly apparent that the root shell was still there but was now competing for input and output with the whiptail process. The screenshot shows the situation when I tried to enter "ls" to the shell. The "l" ended up in the shell but the "s" was interpreted by whiptail as the initial character of the "system-summary" item.

Continuing to do potential dangerous (recovery) tasks in such a situation is a good example of how to shoot your own foot, so I stopped the session and planned to return to it later with some free time on my hands.

For this specific use case, I wanted recovery mode because of the root file system mounted in read-only mode. Of course it should also be possible to transition a running system into this state. Years ago, "init S" would have done that. Consulting the systemd documentation shows that "systemctl rescue" should have the same effect nowadays. Unfortunately the system seems completely dead after this output:

This must be a Ubuntu specific problem as a quick test on a Debian 9 (Stretch) system works exactly as intended. The root password is needed there, but once entered the rescue mode just works. Obviously it was time to report this bug: Ubuntu bug 1661851.

Before returning to the initial scenario, I prepared by consulting documentation about systemd debugging. Booting into the rescue mode again, I immediately checked with "systemctl list-jobs" and sure enough there were quite a few jobs (104 to be precise) waiting and three jobs running. Of course "friendly-recovery" was running, but the other two were not expected. They turn out to be dynamically generated units from "systemd-fstab-generator" to ensure the presence of drives listed in /etc/fstab.

Why those units ultimately run into a timeout seems to be the real problem. Using "systemd-analyze blame" within the regularly running system shows that those units do not normally contribute any significant time to the system startup.

Although I would have liked to solve the problem, this was good enough for another bug report: Ubuntu bug 1662137