Re: Processing times for the NEW queue (was Re: Bits from DPL)
On Thu, Mar 06, 2025 at 11:19:52AM +0100, Timo Röhling wrote:
> I'm not an FTP Team member, but I happen to have analyzed exactly this
> question in detail [1]. The FTP team is very transparent in this regard and
> provides all processing logs, so any DD can verify this for themselves.
Thank you for this analysis and for providing both data as well as the
code to produce it, that's super helpful!
> In summary, the median wait time in NEW is currently less than 48 hours, and
> in the last 10 years it was seldomly longer than a week. 90 percent of all
> packages going through NEW are processed within a few weeks. Only 2 percent
> of all packages going through NEW are held up for several months or longer.
This analysis does not/cannot account for the distinction of the truly
"new to the archive" packages, vs. package renames/SONAME bumps etc. It
is also my hypothesis, as you also identified with your later point (2),
that there are significant differences between the time it takes to
complete the two different types of reviews.
While it's totally understandable why the data is presented this way,
it's worth perhaps keeping this limitation in mind and applying some
caution when trying to interpret the data, as median/90p/98p may not
tell the right story in what what may well be a bimodal distribution.
> That means that in an average month, more than 200 packages pass through NEW
> within two days, and only about 20 packages get stuck for more than three or
> four weeks.
>
> So why do people feel NEW processing is generally slow? I have a few ideas:
While I agree with basically all your hypotheses (and thank you for so
eloquently making them), I think there is an additional reason, that
challenges the framing of the question itself. Your question perhaps
incorporates an underlying assumption that "two days", or "[less] than
three or four weeks" is not slow.
I'd like to challenge this assumption.
I can ship code from a VCS host, for free, in a few seconds. Heck, I can
even ship code from a debian.org domain, from a shared "debian"
namespace, in the same amount of time. Salsa admins are not approving
every new repository by hand, and it would be preposterous for anyone to
even suggest doing that.
So why is this different?
One could of course say that we, as a project, offer our main archive as
a more trustworthy and curated set of software than say, e.g.a random
personal GitHub repository, or even the Salsa "debian" namespace. But,
in turn, this makes the assumption that the only way to legally
distribute software of a certain quality and trustworthiness is by
having a team of 5-10 people review it all by hand. This is, IMHO, a
flawed and outdated view. I believe this practice has not historically
scaled alongside the growth of the project, but also _cannot
fundamentally scale_ at the pace of modern software development and the
expectations that many of us have.
The way I see it, gatekeeping in the way the NEW system works was very
common in the 90s and early 2000s, with software development patterns
where development teams authored software, and another set of "more
privileged" teams (whether these were "operations", "release
engineering", "delivery", a "CAB" or "change control board" etc.), would
manually review, approve and ship said software. A certain amount of lag
was expected, and often built into the system. When the next release was
going to be shipped in CDs three years from the time the code was
written, waiting a couple of weeks was kind of OK?
In my experience, these methodologies have fallen out of practice,
through various movements (code review culture, DevOps, "lean" and
"agile" software development, "shift left"), associated tooling (the
rise of VCS & DVCS like git, code review & CI/CD platforms like GitHub
and GitLab, ...) or... even the availability of internet itself.
Basically, collaborative software development has come a long way in the
past ten or twenty years.
That's not to say that these practices do not still exist in certain
sectors/industries, nor that there isn't a lot of grey in between these
two worlds. But I hope we can all agree that the expectations a modern
software developer has for how quickly it will take for their code to
reach its intended audience, has *radically* changed over the past two
decades.
"A few weeks" simply does not cut it anymore -- the average DD would
likely revolt if they had to wait for a manual review of a few days or
weeks for every Debian upload of theirs, or for every testing migration.
We only tolerate it for NEW because most of us have to rarely go through
it.
With all that said, I'd like to say that I am immensely thankful to
these 5-10 people for their work and what is legitimately a lot of work
we all benefit from. I also understand that it's hard to contemplate
*any* change when you are overworked to the point of burning out, and
especially when you feel your work is thankless and unappreciated.
But, many of us desire more _fundamental_ changes in this space and have
been raising this point for years. I personally have felt like stuck
between a rock (status quo) and a hard place (sounding thankless to an
overworked team of volunteers) for more than a decade. So while the
reasons may be understandable, it's especially saddening to hear that
the team does not even want to contemplate adding new members to its
ranks.
So I'd like to ask the ftp-master team in particular: what would you
suggest is the best way to approach your team in collaboratively
evolving and improving the way NEW works? How can the project, either
through its DPL, or as individual members desiring such larger systemic
changes, convince you for the necessity of making said changes, and
ultimately help you in implementing them?
With gratitude,
Faidon
Reply to: