Over the years, I have made many poor experiences with software as a user. Such experiences can often give hints to developers about what to do better.
Below, some examples are discussed. (For the time being, just one. The list will grow in due time.)
In 2024, I upgraded a remote Debian installation from “bullseye” to “bookworm”. With past version upgrades,
including to “bullseye”, there had been very few problems. Off the top of my head, I can only recall one or two
cases of reconnection problems
with tmux; however, these were not that hard to solve after an Internet search.
(Unfortunately, I do not remember the details.) Instead, my complaints have
arisen after the upgrade and centered on disputable choices by the Debian
team, e.g. an increasing reliance on the ill-conceived systemd, the counterproductive
“alternatives” system, and similar.
This time, everything seemed to work perfectly, including tmux. In a very last step, however, I
manually restarted the openVPN daemon that I had running on the server. (Apparently, this had not been done
automatically, unlike the openSSH daemon, but this might well have been the right decision.)
Suddenly, nothing worked—the VPN was gone. Even a remotely triggered reboot brought no improvement, and I found myself in a potential pickle, as I had configured openSSH not to allow connections from outside the VPN.
Fortunately, many years ago, for just such an eventuality, I had created an ad-hoc user that I only ever used to log in while circumventing the VPN—and had, in fact, only ever used once, those many years ago, in order to test that everything worked. In a second and third “fortunately”, the authentication data were easy to find and nothing else had gone wrong with the log ins.
Once logged in and looking for errors, I found that openVPN objected to the certificates (SHA1) used, which, apparently, were considered unsafe. This, in and by it self, might have been fair, as there, again, had been many years since I set up the VPN. However, I had seen no word of warning on this topic in advance; the old certificates were no less secure than earlier on the same day, before the update; and a more gracious approach would have been to allow connections while warning that the certificates were insecure and that an admin should upgrade them as soon as possible. There also did not seem to be a fallback mode, where I could configure openVPN to accept the old certificates for the immediate future, go back to work in the VPN, and generate new certificates with the additional protection of the VPN. And, of course, if working with insecure certificates is less safe than working with secure certificates, then working without a VPN at all is even worse...
A few remarks:
Who exactly is to blame for what, above, is somewhat unclear, as several parties are involved, including the Debian and openVPN teams. Moreover, openVPN draws on openSSL in this corner and it is conceivable that a change in openSSL, outside the control of openVPN, contributed to the problem. As for me, I had in part read, in part skimmed, a lengthy Debian document with various instructions, caveats, lists of changes, etc., that were relevant to the upgrade. I have no recollection of any caution about such issues or any mention of a temporary workaround.
Having a sufficient heads up to resolve the issue, by generating new and more secure certificates, before the upgrade would have been highly beneficial. Likewise, having the ability to temporarily reactivate the old VPN-setup in order to resolve the problem without resorting to even less secure connections.
How much additional security the VPN brings is unclear, but, at a minimum, it has advantages like the ability
for openSSH to immediately decline most hostile log-in attempts as coming from outside the VPN and the ability
to make some file transfers through tools like socat, in a very convenient manner, without having to
pay additional attention to matters like encryption. It should also reduce the risk of man-in-the-middle attacks
and the damage from loss of an authentication secret on the level of openSSH.
(This provided that the VPN is sufficiently strong, which need not be the case with the old SHA1 certificates.)
Generating new (SHA512) certificates proved easier said than done. Not only were my memories of the original generation quite vague, but the provided tools for easy generation (easyRSA) had changed between Debian versions, Internet sources seemed geared at the old tools, and the instructions and/or the code delivered with the new tools seemed incorrect—at any rate, my first generation attempt seemed to work but ultimately did not help when I restarted the VPN with the new keys, certificates, and whatnots distributed at the right places.
I installed an older version of easyRSA from an older repository, proceeded with older instructions, and the VPN now worked.
However, now entered the idiocy that is systemd. While the VPN was running perfectly, systemd
arbitrarily and incorrectly concluded that the VPN was malfunctioning,
restarted it, and caused a temporary-but-unnecessary loss of connection. Once the VPN was back
up again, systemd soon repeated its mistake and restarted the
VPN—lather, rinse, repeat. I never timed the events, but I would guesstimate that the VPN was gone
around a quarter off the time through this nonsense.
Worse, on occasion, systemd failed to properly terminate the VPN, causing the restart to fail,
as the port to use was already in use. The almost-terminated VPN was in a useless state and did not allow connections;
the almost-started VPN had already exited and did not allow connections either. Sigh.
In a next step, I proceeded to remove openVPN from the control of systemd. This eventually succeeded,
but was easier said than done. A particular complication is that the daemon user did not have enough rights...
Apparently, then, systemd starts the VPN as root, instead of using the
daemon user, which is utterly inexcusable. Here, as is increasingly the case with Linux, especially
where Lennart Poettering, the greatest enemy that Linux has ever known, has had his fingers, cheap convenience is chosen
over security, correct functioning, and whatnot. (Here, the convenience of root access and not having
to bother with setting correct rights on various files and directories.)
The problem is lessened by openVPN relinquishing rights once started and running as the user nobody.
However, this still leaves an unnecessary period of exposure compared to a correct setup where the VPN is
started using the
daemon user (openvpn on Debian; vpn on my Gentoo-based home system), never has root access,
and never can read/write/access/whatnot anything that the daemon user has not explicitly been granted access to.
I am not aware how this was handled in the previous version of Debian, but I strongly suspect that the VPN was,
at that time, outside the ever-growing influence of systemd as (a) such problems, to my knowledge,
never occurred, (b) the upgrade came with various other systemd changes, including relating to logging.
Based on my experiences here and on the ever-lowered quality of Debian over my years as a user, I will likely
refrain from any further Debian upgrades (as opposed to updates) and switch to some other distribution
in due time. Gentoo is a strong candidate, but I will make the exact choice at a later
date—I am in no hurry. (Gentoo, among other advantages, gets by without systemd, which is one
central problem less.)
The following is an automatically generated list of other pages linking to this one. These may or may not contain further content relevant to this topic.