I've been avidly reading all I can about it. I understand how it came about, and my sympathy is firmly with the maintainer. I know research is on going as I type this, but is anyone with the expertise I don't have able to explain in simple terms how bad it is, or could have been?
I could have been much worse, but was identified last minute. Debian, Ubuntu and Fedora had the code only in their testing channels, and overall only .deb or .rpm based distros with systemd have been affected (see https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78b... ). We have to make sure open source maintainers are not underpaid, and stop the madness of global commons exploitation.
> It baffles me how such an important package that so many Linux servers use every day is unmaintained by the original author due to insufficient funds. Something gotta change in OSS. I think one solution could be in licenses that force companies/business of certain sizes to pay maintenance fees. One idea from the top of my head.
The problem is once you change the license and start charging, someone forks it and just offers a less funded, but free version of essentially the same, and the ecosystem fractures. See Redis, Terraform for recent examples. Not exactly the same, but in this context it’s funny how much ire they have drawn when really the root of it is “how do we ensure continuing to develop this software is financially supported by users”.
> I think one solution could be in licenses that force companies/business of certain sizes to pay maintenance fees. One idea from the top of my head.
This just doesn't work. Fully open source software (as opposed to source available) is so much more useful than the alternative that there's always going to be an OSS fork for any sufficiently useful project. AFAICT Elasticsearch and Redis have not really "won" by their respective license changes but rather have just fragmented their own market and sown the eventual seeds of their destruction.
The real question should be how many more similar things there are in the entire
Linux codebase
I would bet good money there are more that haven't been detected.
Every intelligence service in the world wants one or more.
Many can hire truly bright programmers who will contribute to Linux
over years, doing nothing nefarious at all, to build a trustworthy profile.
and even contribute meaningful and positive features.
Then, maybe 2 of 3 trusted profiles make tiny changes as part of a much larger valid contribution. each one a tiny adjustment, each one not particularly interesting from a security point of view, unless you look at all 3 at the
same time, and of course they were not added at the same time.
I am quite certain they have people who are working at Apple and Microsoft
as well with the same mission.
It’s outrageous because it’s basically unprovable, though we have at least one data point the MO exists to some degree. It would be naive to think it’s not attempted at least.
How does one go about becoming a corporate plant? If the CIA offered me $1M to inject a back door into some shenanigans corporate product and told me it was being used to target Bashar al-Assad or cripple Russia’s war effort, would I do it? I don’t know honestly. Principles say it’s not right, but I wouldn’t blame someone (or myself) for saying yes.
Of course, being a data guy I’m probably not at risk of such a decision, since no one is injecting backdoors into corporate analytics. Dare to dream of a shadowy government payout I suppose.
> How does one go about becoming a corporate plant? If the CIA offered me $1M to inject a back door into some shenanigans corporate product and told me it was being used to target Bashar al-Assad or cripple Russia’s war effort, would I do it? I don’t know honestly. Principles say it’s not right, but I wouldn’t blame someone (or myself) for saying yes.
Does anyone know if distros using the systemd-notify patch are considering not linking to libsystemd and instead implementing the notify-protocol themselves?
As this would only be like 10 lines of code, instead of linking a huge library that will in turn pull in another bunch of libs, it seems like it should reduce attack surface.
Great, I was trying to figure this out for Debian but I get PTSD every time I end up on any of the sites related to a package because they all look different and I can never figure out where I'd even find what I'm looking for (i.e. issue tracker, mailing list). :-)
I'm not so sure what to think about the dlopen patch though. It obviously fixes this attack vector, but in general it makes it harder to discover dependencies in a simple way. I wish PE-style late binding would exist for ELF.
I think the really scary part is the particular modus operandi that this demonstrates, that is easily repeatable. A malicious actor builds up a reputation in the open source space and eventually takes ownership of the release process at which point a backdoor is inserted.
There is absolutely nothing that would suggest this hasn't happened before or will not happen again.
Imagine a state actor offering retirement money and a plane ride to a non-extradition island nation for a few commits. Assuming the contributor is using a VPN already, no one would know they were adding surreptitious backdoors in their code.
The only solution is to scope security relevant functions and add a multi stage formal verification process before freezing the commits. This assumes no collusion.
What I find curious is, everybody is acting like, yeah, we found that one attempt! But there gotta be more. And this is only open source. At Google, Meta, Microsoft there gotta be the same forces at play.
So yeah, congrats to everyone involved discovering this, excellent work, really! But how sure can we be, that there was no other successful attempt?
We can never be sure about that. One’s threat model has to assume that Mallory can find a way in. Backdoor’s are not super common, but there will always be 0-days. Security operations need to be setup with this in mind. Design your infra for defence-in-depth. Logging and access control is very important. Log everything that happens in your systems. Don’t use long-lived credentials. Don’t give people more access than they need. Audit users’ privileges, remove privileges that are no longer required. Try to make your logs tamper-proof. (Perhaps keep copies of the logs offline?)
Yep. Assume one or more layers are pwned and rootkitted, especially if it runs Linux because the attack surface is huge in most distros. Never fall for the braggadocio of arrogant, lazy MAANG employees who blindly declare the infra to be "secure" without evidence. Where possible, run a transparent proxy/scrubbing/NIDS/NIPS network layer running OpenBSD or similar. Log setup using WORM media, log structured fs, and/or blockchain would be ideal.
Someone else posted this in another thread and I found it a really interesting read. It doesn’t really answer your question directly, but it is very related.
That is definitely creepy. Having an age of AI in mind I wonder how much more sophisticated will viruses become and what makes me believe my machines are not compromised already? I hope to never get an answer, or even better, get a negative one.
Makes me wonder whether it's time to reduce complexity and diversity in technology? Wouldn't it be beneficial to recreate everything from scratch(except maybe for machines producing hardware)? Leave one or two compression standards that are easy to understand and implement once and for ever. Maybe even aim to make them "finished" since the very beginning? So that they won't require changes after some time. I believe that recreating that and foundational things: compiler(of a language that is reinvented and standardized), OS components, web protocols and so on would be beneficial. There are so many of each and each can be vulnerable now or get vulnerable in the future. Many pitfalls aren't mentioned like why would we believe existing hardware producing new components to be trustworthy or how to make such a revolution without loosing so much progress, who will found it and how can it found commercial interest.
These are, of course, just murky dreams that linked book gave birth to, but there is no reason not to be paranoid.
In practice, it's a non-issue for anyone using a stable version of a major Linux distro, such as Ubuntu 22.04 LTS or RHEL/Alma/Rocky 9. The liblzma in those distros were cut and frozen long before our suspect began to have serious involvement in its development. (Not sure about Amazon Linux. Arch doesn't seem to be affected.)
The more worrying question is, what else was compromised by these kinds of people, and for how long?
I too have been trying to find out about the hack.
There has been a lot of interesting analysis of the code, which has a feeling of the awe and sophistication of stuxnet yet on the scale of “this could be the work of a single motivated coder … like one of us”.
But what I haven’t found yet, and would like leads on, is what ther thinking is about the attack in a wider context…
What is the current thinking on:
1. Was the malicious code inserted by the new contributor, or was this likely a compromised account?
2. If the new contributor did it, did they become an xz maintainer with the aim of acting maliciously?
3. If the new contributor was malicious, why take so long to attack and why do it in xz? Or are they also attacking lots of other systems, perhaps under other names?
4. And what can we do about these attacks? How can we build a system that isn’t ultimately brought crashing down by one or two bad actors?
5. Small technical detail I haven’t spotted in writeups: was the attack commands signed in some way so only the attacker can use it and the world can go hunting for a smoking gun cert that matches the attacker?
First they need to gain some credibility. And they also need to obfuscate what they were trying to do. The payoff if they were to succeed is potentially enormous, so it makes sense to do things slowly.
> and why do it in xz?
xz is almost a perfect target- undermaintained, complicated code that isn't well understood by many people, deployed and used widely.
> Or are they also attacking lots of other systems, perhaps under other names?
The answer probably depends who the attacker(s) are, and what sort of funding or other motifivations they have. We can only guess.
> 4. And what can we do about these attacks?
This is a big unanswered question. I haven't seen any great suggestions yet. We were very lucky with this particular attack.
> How can we build a system that isn’t ultimately brought crashing down by one or two bad actors?
The complexity involved in this attack suggests that it was probably a group, IMO.
> 5. Small technical detail I haven’t spotted in writeups: was the attack commands signed in some way so only the attacker can use it and the world can go hunting for a smoking gun cert that matches the attacker?
I think it is just as plausible that a state actor casts about for an account of a committer they can trivially compromise who is on one of the thousands of tired projects that can slip this kind of attack in…
I feel total sympathy for the maintainer if they 100% didn’t know what was going on.
It’s also unfortunate that the maintainer will be associated with the lack of judgement. But they will get through it. Probably time to pass the torch fully now though.
This is just a small number of committers and one vuln but by its execution suggests there’s more to be discovered and what’s now on those machines this vuln could execute upon and what’s it done to machines on those networks.
Not as bad as the fallout is going to be. Already some people are calling for mandatory identification when contributing to open source projects. This will, of course, have a chilling effect on FOSS contributions, while it seems unlikely (to me) that this would improve security in any meaningful manner.
Who's going to be on the hook for ID verification? Ah, right...
Just a thought, making open source maintainers do more unpaid work is probably not the answer to the problem of overworked open source maintainers.
It was caught just in time so the “badness” in terms of “what do I need to do to mitigate this right now” is thankfully pretty low.
But IMO the real “bad” here is that it calls into question the entire model of volunteer open source contribution, which underpins a crazy amount of the tech world. No easy answers there.
It is still pretty early but yeah, I suppose you could look at it as “-1 day” vulnerability… or a back door which goes back to my comment about it being still pretty early. Who added this code? And who is auditing it?
It was somehow astonishing to me that the code which is actually shipped (tarball) is not the very same code as the one in the repo. Somehow renders code reviews useless.
Seems to be a common practice anyway.
There are no known backdoors in the /usr/bin/xz utility. The backdoor we’re talking about here was in the library. (In Ubuntu I think the package is called liblzma5.)
First: Apply all official patches via APT. That’s the most important thing. The easiest way to avoid getting pwned is to apply security patches without too long delay.
Then consider defense-in-depth. If you have sshd exposed to the Internet, you could put it behind a VPN such as Tailscale (or pure WireGuard).
And FWIW: With our current knowledge, any LTS version of Ubuntu should not have been affected by this backdoor.
perfect 10 on severity, probably close 10 on impact. Caught before it was widely exploited (it got noticed because it just rolled out, in an attempt to get it into the release window for 24.04 etc) but it’s essentially RCE on all Debian, Ubuntu, and fedora openssh servers.
Brew was installing the affected version 5.6.1 of `xz`. They have in meantime rolled back the version to 5.4.6
There's not enough information yet to know for sure what the effect is / was on systems running the backdoored version.
