Re: maintainer-built binaries
Hi,
On 12/3/19 3:54 PM, Ansgar wrote:
Paul Wise writes:
I was looking at the TODO item about maintainer built binaries.
I saw Paul mentioning this on irc, which made me clean up an old branch
that tries to do something like this:
https://salsa.debian.org/ivodd/dak/tree/new-throwaway-binary-v10
This branch adds new options ThrowAwayNewBinarySuites and
ThrowAwayNewBinaryComponents to specify for which suites and components
the binaries from a NEW source+binary upload should be discarded when
the upload is accepted from NEW. The old binaries are moved to morgue.
The branch also adds a test to check this.
Some notes:
This can only work if the name of the changes file for the source+binary
NEW upload is different from the buildd binary upload. That currently
isn't the case. An option to add a suffix to the relevant filenames was
merged in sbuild here (not uploaded yet):
https://salsa.debian.org/debian/sbuild/commit/98fccdfb5eaa07c7bb5a4df0bdd4f154b7e4d069
The idea is to get this deployed on the buildds eventually.
I didn't test the interaction with policy queues, so it shouldn't be
used there. Currently, NEW doesn't interact very well with policy queues
anyway. And on the security archive, throwing away the binaries will
break the subsequent upload to the ftp-master archive (when the security
update is released). This option is mainly targeted at unstable (and
maybe experimental) and only for main (not for contrib and non-free).
The simplest way to discard maintainer built binaries appears to be
that any sourceful uploads should get any associated binaries ignored.
This should make NEW work the same as it does now, make almost all
binary uploads come from buildds but not block maintainers from doing
binary-only uploads where maintainer-built binaries are needed, such as
for language bootstrap after NEW processing or later where needed.
If you discard binaries from sourceful uploads, they won't be available
for NEW processing.
AS noted above, my branch doesn't do this.
If you discard binaries after they were in NEW, you create race
conditions as a file with the same name will be uploaded by buildds.
In what situations would there be race conditions?
When do_new runs, the upload is accepted in
dak process-policy new
and the old files are removed in
dak clean-suites -a new
If the packages files for the buildd suites are built before do_new, the
source wouldn't be there yet. If they are built afterward, the old
binaries aren't there anymore.
So by the time the buildd sees the source, the original binaries are no
longer in the archive (outside of morgue).
Note that this assumes that do_new and building the buildd suites are
mutually exclusive. If that isn't the case, locking would be needed to
avoid this race condition (my impression is that the locking is already
there in the debian config, but it would be good for someone else to
confirm this).
(Also I suspect it would make version tracking to enforce newer uploads
always have a higher version than previous uploads, even when binaries
were removed in between, more complicated.)
That depends on the implementation of this kind of tracking (which
doesn't exist yet). Note that this issue currently also exist with
NEW-uploads that are rejected.
It also means maintainers would have to upload binaries twice some
times (before and after NEW processing).
Yes. However this would only be the case if there is a build-dependency
loop between the source and binaries built from it. That's not that
common. It happens in the bootstrapping case Paul mentioned.
I think the need for multiple uploads in this rare case is better than
the much more common case where people need to do a new source-only
upload to get the binaries built on the buildd after they want through NEW.
I'm not sure if it isn't better to eventually switch to source-only
uploads for NEW instead.
That would be better, but would require quite some work. I don't think
this will happen any time soon. For that reason, I think throwing away
the binaries for NEW uploads would be an improvement for now.
Please keep in mind that some packages will not get built by buildds
anyway, such as some non-free stuff (not building them on buildds is the
default for non-free).
I don't think we should throw away NEW binaries for contrib and
non-free, because there's no (easy) to discover if they can/will be
built on the buildds or not.
Cheers,
Ivo
Reply to: