Re: Serializing transitions
[ I find the tone of your mail needlessly aggressive, we're just
discussing an idea at this point and seeing if it's worth investing more
time into it ]
On Fri, 26 Mar 2010, Marc 'HE' Brockschmidt wrote:
> No, this can't be defined later. It's a central part. Would any new
> binary lead to a dedicated overlay? How do you determine when this is
> needed and when not?
We have several types of transitions and it will need different kind of
checks. Let me give some examples.
A direct upload to sid:
- must not drop a binary package that has reverse dependencies
(ex: drop libfoo4 when new source package provides libfoo5)
- must not change the SONAME of a public library without changing
the package name
- must not increase the automatic dep via shlibs/symbols files (we first
want to make sure that such an updated library is built/available on all
arches before it hits unstable)
- must not remove a Provides where it was the only package providing it
(ex: perlapi-5.8.0, phpapi-*) if there are reverse dependencies
on that provides
I'm not sure we can catch all transitions, I have no suggestion for
python-defaults or gcc-defaults transitions for example. But we are
generally aware of those transitions and they rarely happen out of the
blue.
I did not want to go into details because I believe that this part
is doable and it doesn't bring much to the initial goal of the discussion:
determining if such a system could work to better manage the transitions.
> > Instead transitions should be managed in dedicated repositories
> > that are overlays over sid. We would have tools (command-line, web
> > interface, etc.) to manage the transition in that repository.
>
> What would these tools do? Why don't they exist currently?
See below. They would be used to tell the system which packages are affected
and should be included in the transition. (And they do not exist becuse
nobody wrote them yet, although some release people have scripts to identify
set of packages concerned by transitions)
> > We could tell which packages need to be rebuilt (bin-NMU, sourceful upload)
> > and the system would automatically rebuild the package (and it
> > would also be automatically rebuilt every time that the package is updated
> > in sid).
>
> How do you determine the set of packages that need to be transitioned?
> Are these provided by the uploader? Can they be computed?
A mix of both, the uploader should be able to provide a raw list and/or tell
the system "all source packages whose binaries are depending on libfoo4".
The release team would verify that the set of packages includes all
required packages but some edos-debcheck based tools could also help to
try to ensure that automatically.
> > Some web interface would display the status of the transition,
> > displaying which package/arch have been built or not, and which one failed
> > to build from source.
>
> You mean like the existing pages on buildd.debian.org? You just need to
> feed them the list of affected packages to get that.
Good if it can be done with a simple link to buildd.debian.org! It would
still be nice to be able to attach some information like bug numbers near
FTBFS so that people managing the transitions know whether all failures
have been reported.
> > Multiple transitions will still end up mixed in sid if you push them
> > before packages have migrated to testing but they are all already
> > completed and you only have to deal with RC bugs and delays to ensure
> > package can migrate to testing.
>
> "Only" deal with RC bugs? This touches an interesting point you didn't
> cover at all: How are bugs reported? How can these bugs be tracked if
> the same source version is fine in sid, but breaks in an overlay - or,
> worse, breaks in one overlay, but not in another?
Bugs are reported like usual. However you are right that we need to extend
our bin-nmus versioning scheme to avoid collisions in package versions
between various overlays. And/or we need to extend our tools to be able to
explicitly give the source version (in fact there's such a request already
in the dpkg-dev BTS:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=440094)
In sid:
Package: bar
Version: 1.0-1
Source: barsrc
In python2.6 overlay:
Package: bar
Version: 1.0-1+python2.6+b1
Source: barsrc (1.0-1)
In libfoo5 overlay:
Package: bar
Version: 1.0-1+libfoo5+b1
Source: barsrc (1.0-1)
It might also mean some changes to the version-tracking code.
> > Dealing with transitions that way make it clear that the maintainer has to
> > take the responsibility to prepare/complete the transition if he wants to
> > see his package in sid and in the next release.
>
> I assumed that this was already the case. I also don't see how enforcing
> such a technical system would solve this problem?
Currently a maintainer uploading new version of the library and having
given a list of packages to bin-nmu can disappear, the transition will
complete a some point because otherwise it blocks you as release managers
to deal with other transitions. So you ensure relevant RC bugs and FTBTS
are fixed so that you can migrate packages. But it should not be your
responsibility.
With the proposed system you would no longer have to finish any current
transition by yourself (or even prod people to to their work). If it's not
yet finished, you can simply pick another one that is more advanced and do
it first. If the maintainer never finishes the transition work, his
updaded package will never reach sid.
Transitions are prepared in parallel and are competing to be ready first.
Your job as release managers is to decide of the order based on the
advancement of currently pending transitions while still trying to limit
disruption.
> > Preparing the transition in experimental is not always done and takes
> > much more energy than such a system would take.
>
> Why, actually?
I don't an exhaustive answer but here are some points:
1/ you can't request bin-nmus of reverse-dependencies in experimental
(to verify that all packages build fine with the updated package, and
that's one of the main task in preparing the transition)
2/ you have to manually reupload a new source package to unstable with all
the delay it induces for getting the package built on all arches
3/ some maintainers are too confident that nothing is going to break
> Trying to improve the handling of transitions is a good thing. I don't
> consider your mail to be a proposal, as it doesn't cover any of the
> actual technical problems.
It's still an idea, it's not yet a proposal with a spec but before going
further and maybe trying to draft a DEP I want to know if you believe that
such a system could improve the handling of transitions.
I hope I responded to some of your concerns by showing that it's possible
to find solutions if we know where we'd like to go.
BTW, I got pointed to
http://wiki.debian.org/SummerOfCode2010/DeveloperPackageRepositories and
having this implemented would mean that it should not be too difficult to
implement those overlays (by reusing the infrastructure).
> Another interesting question: Do you plan to implement this? Do you
> expect someone else to do it?
I don't plan to this implement this alone. But if we decide it's something
that we want to try, I will help (at least to drive the DEP, I can't
promise to work on something we have not yet thought out).
> And, seeing the not ending discussions about your handling of the new
> dpkg source formats: Do you think that such a change could be
> implemented without pissing off so many people?
Sure. That's why I suggest to draft a DEP.
Cheers,
--
Raphaël Hertzog
Like what I do? Sponsor me: http://ouaza.com/wp/2010/01/05/5-years-of-freexian/
My Debian goals: http://ouaza.com/wp/2010/01/09/debian-related-goals-for-2010/
Reply to: