(there's probably better write ups on this argument, Phoronix was just the first link that came up when Googling for the /proc/cmdline Linus systemd rant)
That is interesting... so basically the systemd developers wanted to turn the generic parameter kernel command line parameter "debug" into something that could be easily used by admins to find out why their system won't boot (whether the problem is in the kernel or in userspace), and got flamed to a crisp by kernel developers who want to keep using "debug" to debug their kernel code because they've always done it that way, and couldn't care less about helping actual end users.
The kernel devs are mad that flipping a kernel debug switch (on the kernel command-line, after all!), causes a userspace program to change its behavior, to the point that "the system becomes undebuggable on a dracut/systemd system" (kernel dev's words; apparently systemd spews a bunch of debug info into dmesg.)
If you're going to say it's the kernel's fault, it's ultimately because they ever exposed the kernel cmdline to userspace, as something like this was inevitable.
> got flamed to a crisp by kernel developers who want to keep using "debug" to debug their kernel code because they've always done it that way
If their reports can be believed, they literally can't get work done (if it involves turning on debug in the kernel of a systemd installation), so it's far from some petty turf war or irrational hatred for change. Even if they should change, you seem to take it for granted, that they should have to change their kernel workflow, based on what something in userspace does -- why?
The kernel command line has the "quiet" option, which has for many years been used by user-space to display a fancy graphical progress bar instead of 100s of lines of text;
"debug" is just the inverse of that.
There was also some bug in systemd that caused it to log an excessive amount of assertion failures into some kernel buffer, which was in fact a really bad bug and was probably the reason for the whole flame-fest; naturally this bug has been fixed.
But think about it: is it likely that the whole distracting turf war over the "debug" flag helped in getting the actual bug fixed as fast as it could have been?
The reason why I would prefer to have systemd respect the "debug" flag, even if it means that kernel developers have to learn a new workflow, is that there are a lot more system administrators than kernel developers out there, and (on average) the kernel hackers have lots more expertise than the admins.
So if a system has problems booting there should be a simple, short, generic parameter to get some verbose logging enabled, that can be used to track down the problem whether it's in the kernel or in user space, which you can't know beforehand of course.
The kernel developers may be understandably unhappy about the change, but once they get over their initial reaction using a different parameter for kernel-only debugging is really quite trivial at their level of competence.
The reason why I would prefer to have systemd respect the "debug" flag, even if it means that kernel developers have to learn a new workflow, is that there are a lot more system administrators than kernel developers out there, and (on average) the kernel hackers have lots more expertise than the admins.
By this logic, there are a lot more servers than there are desktop Linux users, and more than likely a lot more system administrators than desktop Linux users (I believe I'm in the minority as I'm a system admin, who uses Linux daily on all his computers, desktops, laptops, and servers), so the things systemd does that are intended to make the desktop linux experience better, like faster boot (which was never strictly a problem on servers, except in the time it usually takes a server BIOS to get to the grub prompt, which systemd can't address), "better" hot-plugging of devices, socket activation (which already had a solution and is something you don't want on a server for the example use-case of mysql), the FSS logging that journald does (because sysadmins have already sufficiently solved that at scale) so systemd shouldn't be encroaching on server linux use. And there are a lot more system administrators out there than desktop environment developers, and (on average) have more system administration experience than the desktop environment developers.
This also assumes that sysadmins don't use the kernel debug flag, and only kernel developers do. I've had to use it on the rare occasion, and I've been glad that messages from the kernel are easily selectable via a specific kernel command line option as distinct from userspace messages.
> The reason why I would prefer to have systemd respect the "debug" flag, even if it means that kernel developers have to learn a new workflow
Again, you have things backwards. It was the systemd developers that DIDN'T want "to respect the debug flag". The systemd developers tried to prevent the debug flag from getting past the kernel. It was the kernel developers that forced the systemd developers "to respect the debug flag"
All the credit that you're attributing to the systemd developers was actually down to Linus Torvalds drawing a line in the sand.
You have things backwards. It was the systemd developers that didn't want the debug command line to be used against systemd. They were the ones who filed the patch to prevent the debug command line from reaching user space (/proc/cmdline) and it was the kernel developers (well, Linus specifically) who refused to patch and demanded that systemd cleaned up their debugging output so that 1) it didn't crash Linux (which it was doing previously), and 2) the output was more useful to end users for debugging.
Yeah, that was an odd fix. Then /proc/cmdline doesn't reflect what the actual kernel command line was, thereby making debugging more opaque. The sane option, which the kernel developers suggested, was that systemd namespace pre-boot options it takes from the kernel command line, just like every kernel subsystem already does.
(there's probably better write ups on this argument, Phoronix was just the first link that came up when Googling for the /proc/cmdline Linus systemd rant)