[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Auto reject if autopkgtest of reverse dependencies fail or cause FTBFS

On Jan 16, 2017, at 10:52 AM, Scott Kitterman wrote:

>This is going to take a lot of work.  I see random failures routinely block
>migrations in Ubuntu (postfix is a current example - there's two alleged
>regressions that to the extent they are valid are completely unrelated to
>anything that changed in the package).
>The question is who's going to do the work?  I don't see the release team
>having tons of spare cycles to dive into the details of individual test
>results and package failures and decide what's RC and what's not.  The only
>thing that scales to something the size of the Debian archive is to let the
>maintainers decide.

Speaking only about our experiences in Ubuntu, I can anecdotally but
emphatically claim that gating promotions on passing autopkgtests has
dramatically improved the quality of the running end user systems.

It used to be that you had to be very careful about running and especially
dist-upgrading the current devel series.  You never knew if something major
like the kernel or X would break, or even when minor breakages would be highly
inconvenient.  It just wasn't safe without a lot of precaution.

But now I don't hesitate to run devel almost as soon as the new series opens.
That's not to say that serious breakage never happens; not everything is
tested of course, and stuff happens.  But it's rare, maybe once or twice a
cycle for boot-to-desktop to break, or a package regression sneaks through.  I
just have way way more confidence in the distro now that these tests block

Yes, it can be more work at times, and it's not always easy to diagnose or
reproduce promotion problems.  (I'm currently flummoxed by a systemd
regression triggered by a network-manager fix.)  But I'd much rather have the
luxury of debugging these problems on a still-functioning system and without
also-frustrated users hammering me on IRC, with deadlines looming.

I think it does mean that maintainers will have to step up and take more
responsibility for nursing their packages through to promotion, but I also
think they are in a much better position to do so than J Random User who runs
an upgrade only to be left with a broken system or application.

One other point.  I don't know how many folks run unstable (or in Ubuntu's
case, devel), but for most software I work on, few users really test
pre-release versions.  As much as you plead with them, "hey, beta 3 is out,
please test!" they just won't for totally understandable reasons.  So problems
arise *after* the final release because that's when people start to really
hammer on it, and integrate it with their own software, environments, and
workflows.  That means day-to-day user testing just can't be all that reliable
because there are so few data points, and it's another reason why I think
automated testing/CI is so important.  (It's also an investment over time; you
don't have to have 100% coverage from day one, but every new test can improve
overall quality just a little bit.)  It's also why I feel it's important for
*me* to run unstable/devel.  True, it's my day job, but I also feel a
responsibility to help ensure the little corner of stuff I use, care about,
and know about is in as good a shape as possible before it gets into the hands
of our users.  I *want* to feel the pain before they do.


Attachment: pgpUtzCAlAur0.pgp
Description: OpenPGP digital signature

Reply to: