They noticed that some ssh sessions took 0.5 seconds too long under certain circumstances. 😲
Holy hell that’s good QA.
Microsoft employee.
Definitely not from the team working on search on Windows then.
Don’t see why you’re being downvoted, the person in question who discovered this is a postgres maintainer employed by Microsoft.
Probably people think this is a troll or something.
I wrote it because I was surprised, especially since I’m not a fan of microsoft and their policies. Lately, I have the feeling Microsoft is better than Google (relative terms) when it comes to oss.
What is additionally surprising is the breaches of Microsoft services in the last year. There is one every few weeks or so… And then they pick up a backdoor because login took 0.5 instead of 0.1s.
Anyway, his findings are amazing.
This isn’t the same thing, but I’m reminded of Minecraft.
Minecraft is a massively popular game. Notch once said he planned to make it open source when its popularity died down. But now Microsoft owns it.
Not only that, but Mojang accounts don’t work anymore. You have to have a Microsoft account to play it now. Even trying to download and play an older version of the game offline requires Microsoft to approve it. Microsoft is actively tightening the leash on the game because it makes them money. Open sourcing the game will likely never happen now. The best we can hope for it for versions to fall into public domain after 70-ish years.
That’s how I see Microsoft. They only care about what its beneficial for them to drive profits. Working on open source projects, and open sourcing a few of their tools to get the benefits of community adoption and code review is great, sure. But they’d sooner try to incorporate Linux into Windows to keep people in their surveillance ecosystem, than to open source Windows.
Remember when Windows 10 was the last version, until they changed their minds? Remember when they floated the idea of charging a recurring subscription to use Windows, before they silently dropped the idea? Remember when there was credible talk about the next version of Windows being cloud-based where they controlled all your data and you had no privacy? Hell, you have basically no privacy on Windows 10. Trying to reclaim some involves registry edits, special third party tools, and a constant battle with automatic updates reverting your changes.
I’ll say it again. Microsoft doesn’t care about OSS. It’s just currently beneficial for them to pretend they do.
Goggle seemed to care a lot about OSS, then started making everything in Android depend on their proprietary ecosystem to function. Now Google is using the dominant position they got by taking advantage of OSS adoption, and have been pushing privacy-invading standards and trying to get rid of ad blockers online, among many other things.
For these huge companies, OSS is just a tool to get more control and power. The moment it’s no longer useful, they’ll find ways to work around the license and enshitify everything again.
It keeps happening. I refuse to keep trusting bad actors every time they dangle a shiny trinket over our heads.
I do appreciate the work this person did in finding the bug. It’s not all doom and gloom.
I agree with you sentiment here. That’s why I wrote ‘relative terms’ in my comment.
Since Nadela took over, Microsoft did some open thing which benefited community. So, Microsoft opened somewhat.
During the same time, under Pichai, google went the other way: they focus more on monetization and try to control stuff the apple way. Manifest v3? Google also didn’t do anything really worth mentioning in the last 10y in terms of products. Well, except ‘attention’ article. And even this they didn’t believe in and they cannot deliver a decent product. I just tried google advanced Gemini and it’s, to put it politely, shit. Google also had some positive actions like mainlining a lot of stuff in Linux Kernel to more easily upgrade android.
So, while google is closing down and making mistakes, Microsoft is opening a bit up.
If you look the state from the last year and the state now. Microsoft improved. Google went the other way.
Microsoft doesn’t care about open source, they care about the money Cloud Services using open source bring them. I don’t think google cares as well. For reason read this: https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/
Well half a second delay is pretty noticeable when you ssh into a machine sitting right next to you. It should be instant. And if it isn’t something’s off.
Damn, it is actually scary that they managed to pull this off. The backdoor came from the second-largest contributor to xz too, not some random drive-by.
They’ve been contributing to xz for two years, and commited various “test” binary files.
It’s looking more like a long game to compromise an upstream.
Either that or the attacker was very good at choosing their puppet…
Well the account is focused on one particular project which makes sense if you expect to get burned at some point and don’t want all your other exploits to be detected. It looks like there was a second sock puppet account involved in the original attack vector support code.
We should certainly audit other projects for similar changes from other psudoanonymous accounts.
Yeah, and the 700 commits should be reverted… just in case we missed something.
That is a should be standard procedure
Time to audit all their contributions although it looks like they mostly contribute to xz. I guess we’ll have to wait for comments from the rest of the team or if the whole org needs to be considered comprimised.
deleted by creator
It would be nice if we could press formal charges
Assuming that it’s just that person, that it’s their actual name and that they’re in the US…
there will be federal investigationjust speculation if the culprit is a foreign actorDo you have a source for this?
Source: they made it up
I don’t have a source but I think it is safe to say given the large corporations and government institutions that rely on XZ utils. I’m sure Microsoft, Amazon, redhat ect are in talks with the federal government about this
updated my post, it was just some speculation i misread
This is the best post I’ve read about it so far: https://boehs.org/node/everything-i-know-about-the-xz-backdoor
In the fallout, we learn a little bit about mental health in open source.
Reminded me of this, relevant as always, xkcd:
Yes, exactly.
And looking at you npm : npm
That whole timeline is insane, and the fact that anyone even found this in the totally coincidental way they did is very lucky for the rest of us.
If you’re using
xz
version 5.6.0 or 5.6.1, please upgrade asap, especially if you’re using a rolling-release distro like Arch or its derivatives. Arch has rolled out the patched version a few hours ago.Gentoo just reverted back to the last tar signed by another author than the one seeming responsible for the backdoor. The person has been on the project for years, so one should keep up to date and possibly revert even further back than just from 5.6.*. Gentoo just reverted to 5.4.2.
Just updated on void and saw the same thing
Dang, Arch never sleeps, does it? That’s a 24/7 incident response squad level of support.
Backdoor only gets inserted when building RPM or DEB. So while updating frequently is a good idea, it won’t change anything for Arch users today.
No, read the link you posted:
Arch does not directly link openssh to liblzma, and thus this attack vector is not possible. You can confirm this by issuing the following command:
ldd "$(command -v sshd)"
However, out of an abundance of caution, we advise users to remove the malicious code from their system by upgrading either way.
I think that was a precaution. The malicious build script ran during the build, but the backdoor itself was most likely not included in the resuling package as it checked for specific packaging systems.
when building RPM or DEB.
Which ones? Everything I run seems to be clear.
https://access.redhat.com/security/cve/CVE-2024-3094
Products / Services Components State Enterprise Linux 6 xz Not affected Enterprise Linux 7 xz Not affected Enterprise Linux 8 xz Not affected Enterprise Linux 9 xz Not affected (and thus all the bug-for-bug clones)
Those getting the most recent software versions, so nothing that should be running in a server.
Fedora 41, Fedora Rawhide, Debian Sid are the currently known affected ones AFAIK.
I think it needs to be
- rolling release (because it was caught so quickly that it hasn’t made its way into any cadence based distro yet)
- using the upstream Makefile task to build a RPM or DEB (because the compromised build script directly checks for that and therefore doesn’t trigger for a destdir build like Gentoo’s or Arch’s)
- using the upstream provided tarball as opposed to the one GitHub provides, or a git clone (because only that contains the compromised Makefile, running autotools yourself is safe)
Points 1 and 2 mean that only rolling release RPM and DEB distros like Debian Sid and Fedora are candidates. I didn’t check if they use the Makefile and the compromised tarballs.
The person who found the backdoor : https://mastodon.social/@AndresFreundTec/112180083704606941
Time to bring back the reproducible build hype
Why didn’t this become a thing? Surely in 2024, we should be able to build packages from source and sign releases with a private key.
It’s becoming more of a thing but a lot of projects are so old that they haven’t been able to fix their entire build process yet
Probally more SLSA which reproducible builds is apart of
Reading the comments here https://news.ycombinator.com/item?id=39865810 it appears that libarchive may be tainted as well.
This is pretty insane. Can’t wait for the Darknet Diaries on this one.
ELI5 what does this mean for the average Linux user? I run a few Ubuntu 22.04 systems (yeah yeah, I know, canonical schmanonical) - but they aren’t bleeding edge, so they shouldn’t exhibit this vulnerability, right?
The average user? Nothing. Mostly it just affects those who get the newest versions of everything.
In this case I think that’s just Fedora and Debian Sid users or so.
The backdoor only activates during DEB or RPM builds, and was quickly discovered so only rolling release distros using either package format were affected.
Not regular Fedora, though, it was only in Fedora Rawhide and Fedora 41, so very very early, bleeding edge distributions. Nothing that a regular Fedora user would be using.
https://access.redhat.com/security/cve/cve-2024-3094
E: and Fedora 40 beta which some regular users could conceivably be using
https://www.redhat.com/en/blog/urgent-security-alert-fedora-41-and-rawhide-users
It mostly affects/targets the build systems of binary distros - infecting their build machines with this would result in complete compromise of released distro down the line.
apt info xz-utils
Your version is old as balls. Even if you were on Mantic, it would still be old as balls.
Security through antiquity
deleted by creator
Lzma balls
Here is the official statement from OpenSUSE: https://news.opensuse.org/2024/03/29/xz-backdoor/
t y for sharing.
#showerthoughts The problem is in upstream and has only entered Debian Sid/unstable. Does this mean that for example bleeding edge Arch (btw) sshd users are compromised already ?
Looks like the 5.6.1-2 release on Arch moved from using the published GitHub releases to just using the git repository directly, which as I understand avoids the exploit (because the obfuscated script to inject the exploit is only present in the packaged tarballs and not the git repo itself)
They also believe we (Arch users) are unaffected because this backdoor targeted Debian and Redhat type packaging specifically and also relied on a certain SSH configuration Arch doesn’t use. To be honest while it’s nice to know we’re unaffected, it’s not at all comforting that had the exploiter targeted Arch they would have succeeded. Just yesterday I was talking to someone about how much I love rolling release distros and now I’m feeling insecure about it.
More details here: https://gitlab.archlinux.org/archlinux/packaging/packages/xz/-/issues/2
Someone always has to be the guinea pig.
That being said, maybe there’s an argument for distros that do rolling releases to have an “intentionally delayed rolling release” that just trails the regular rolling release by a fixed amount of time to provide more time for guinea pigs to run into things. If you want rolling, but can live with the delay, just use that.
OpenSuse Slowroll does pretty much that, a slightly delayed rolling release.
Arch is on 5.6.1 as of now: https://archlinux.org/packages/core/x86_64/xz/
We at Nixpkgs have barely evaded having it go to a channel used by users and we don’t seem to be affected by the backdoor.
Arch had a patch rolled out yesterday [1][2][3] that switches to the git repo. On top of that the logic in the runtime shim and build script modifier was orchestrated to target Debian and RPM build systems and environments [4].
[2] https://gitlab.archlinux.org/archlinux/packaging/packages/xz/-/issues/2
[3] https://security.archlinux.org/CVE-2024-3094
[4] https://www.openwall.com/lists/oss-security/2024/03/29/4
The link mentions that it is only ran as part of a debian or RPM package build. Not to mention that on Arch sshd is not linked against liblzma anyways.
Arch has pushed the patched
xz
just a few hours ago: https://archlinux.org/news/the-xz-package-has-been-backdoored/Thanks a bunch.
It was also on Gentoo. I had this version installed for a day or two.
Since you didn’t build a RPM or DEB package however, your didn’t compile in the backdoor.
Yeah, it’s probably fine. I also don’t use systemd. I was just pointing out that another rolling release distribution had the affected version.
Homebrew rolled back the release after finding out
Here’s a link to the PR for anyone who’s interested
This also affects Fedora 40 and Fedora Rawhide - https://discussion.fedoraproject.org/t/warning-malicious-code-in-current-pre-release-testing-versions-variants-f40-and-rawhide-affected-users-of-f40-rawhide-need-to-respond/110683
Damn fine work all around.
I know this is an issue fraught with potential legal and political BS, and it’s impossible to check everything without automation these days, but is there an organization that trains and pays people to work as security researchers or QA for open source projects?
Basically, a watchdog group that finds exploitable security vulnerabilities, and works with individuals or vendors to patch them? Maybe make it a publicly owned and operated group with mandatory reporting of some kind. An international project funded by multiple governments, where it’s harder for a single point of influence to hide exploits, abuse secrets, or interfere with the researchers? They don’t own or control any code, just find security issues and advise.
I don’t know.
Just thinking that modern security is getting pretty complicated, with so many moving parts and all.