* On 2024 31 Mar 20:46 -0500, Andy Smith wrote: > In the xz case the further you go looking for a root cause the wider > the implications are: > > Q: Why was there a back door in sshd? > A: Because some malicious code was linked to it. > > Q: How did malicious code get linked to it? > A: Its lzma dependency was compromised. From what I have read, lzma is not a direct dependency of openssh. It turns out that it lzma is a dependency of libsystemd and that relationship affected openssh. Jacob Bachmeyer in analysis (https://lists.gnu.org/archive/html/automake/2024-04/msg00000.html) says: Lastly on this topic, some of the blame for this needs to fall on the systemd maintainers and their "katamari" architecture. There is no good reason for notifications of daemon startup to pull in liblzma, but using libsystemd for that purpose does exactly that, and ended up getting xz-utils targeted as a means of getting to sshd without the OpenSSH maintainers noticing. End quote. > Q: Who compromised the lzma dependency? > A: One of the developers of that project who had full rights to > commit code to it. > > Q: Why did a persona that no one knows anything about get full > access rights to a code repository that is linked to openssh? > A: Because they did some work over a period of years that looked > genuine and the single other developer who was overwhelmed with work > decided to give them access based on that > > Q: Why did lzma, a dependency of openssh, have a single overwhelmed > developer? > A: Because no one felt the need to pay a team of developers to work > on it or audit work on it. As Jacob notes, lzma is apparently not a direct dependency of openssh which makes this story even more incredible. Now we have to consider the implications of patches to software that are related in non-obvious ways. Our systems are currently very complex and that complexity leads to vulnerabilities in software that is not obviously linked. Pretty scary stuff, actually. > Society demands that open source developers work, often for free, > and that they merge contributions. If they push back and say they > are unable to due to workload then they are encouraged to seek help > by adding more committers. That's what apparently happened here: the > attacker(s) seemingly counting on the pressure that would exist to > give them rights within the project. It is hammered in to open > source developers over and over: > > Allow others to contribute, or even to take over, if you are too > busy. It's the right thing to do. > > We have now seen proof of what has long been theorised: that the > above way of working is very vulnerable to attackers who are willing > to put in some effort, and that "enough eyes make all bugs shallow" > doesn't hold true unless the process is actually providing those > eyes. "Jai Tan" was somewhat patient at playing the long game it would appear. The developer spent a lot of time and effort twisting the Autotools build system into something that could hide the needed code. You're absolutely correct, Andy, working its way deep into the project while simultaneously planning to compromise it enabled the developer to build a near disastrous back door. It was also a very clever effort as apparently the interaction of openssh, systemd and lzma were studied carefully and understood to the level that choosing lzma as an attack vector would likely go unnoticed as the relationship is apparently non-obvious (though I've skimmed through the analysis my hobbyist coding skills aren't up to the challenge of understanding this in its entirety). > I have no answers on how to fix such a deep-rooted societal problem > but I am not going to start yelling obscenities at people on public > mailing lists because they are wanting to discuss a CVE or whatever. At the end of the day this is about trust. The compromising developer apparently set out from the start to exploit the trust of the main maintainer and the overall community. The sad part is, to me at least, that the effort to analyze and understand the non-obvious relationship of openssh, systemd, and lzma and then use that to create a back door into openssh could have been used to improve security and create better software. Another hard lesson is that contributions are not always based on the ideal of altruism that we in the "western world" hold dear. BTW, I don't want to start a systemd bashing subthread, but I think it bears some scrutiny give this latest event (disclaimer, yes I use systemd as PID 1 on Debian Stable). Finally, I am still involved with a project (hamlib) that is packaged in many distributions. I had to pull back several years ago and turn over the day to day operations to a developer I came to trust. For my own part I became the administrator of the project after the original developers had moved on and I was almost the most senior active developer left. Patches were backing up and someone needed to take action so I did. Your comments hit home as while I am content to continue to participate in the Hamlib project on the limited basis that I do, at some point it will need to be handed off to someone willing to take the reins. This recent event makes me just a little bit more skeptical about anyone requesting to take over the project unless that person is a known member of an already long established project. Not an easy situation for project maintainers and the community to fix. - Nate -- "The optimist proclaims that we live in the best of all possible worlds. The pessimist fears this is true." Web: https://www.n0nb.us Projects: https://github.com/N0NB GPG fingerprint: 82D6 4F6B 0E67 CD41 F689 BBA6 FB2C 5130 D55A 8819
Attachment:
signature.asc
Description: PGP signature