summarising answers to Vancouver critique (was: Re: Bits (Nybbles?) from the Vancouver release team meeting)
Hi Martin, *!
I spent a lot of my time reading the list in the last few days. The following
is a short summary of the the answers I observed on d-devel in the last days.
I'll amend that with observation I made in the last years from the sidelines
as a interested non-DD.
Thank you for providing a good collection of the questions the proposal has
collected.
The words MEMBER, REGULAR, IRREGULAR, WIDESPREAD and NARROWSPREAD shall be
used as defined by Henning in
http://lists.debian.org/debian-devel/2005/03/msg01495.html
extending the usage of the words REGULAR and IRREGULAR also to "releases".
On Wednesday 16 March 2005 12:50, Martin Schulze wrote:
> Steve Langasek wrote:
> > The much larger consequence of this meeting, however, has been the
> > crafting of a prospective release plan for etch. The release team and
> > the ftpmasters are mutually agreed that it is not sustainable to
> > continue making coordinated releases for as many architectures as sarge
> > currently contains, let alone for as many new proposed architectures as
> > are waiting in the wings. The reality is that keeping eleven
> > architectures in a releasable state has been a major source of work for
>
> Up to this mail, this wasn't a problem that was publically discussed.
> Why? Was it not a problem? Or was our communication so bad that we
> were unable to even discuss problems?
My recollections are, that there were technical problems with the scripts
which had to be solved before the additional w-b queues
See http://lists.debian.org/debian-devel-announce/2005/01/msg00011.html for
details.
> Up to Steve's mail, the reason why we couldn't release were d-i wasn't
> ready, testing-proposed-updates wasn't in place, testing-security
> wasn't in place and the like. Keeping architectures in sync was never
> addressed as a problem. But suddenly it is?
Reading the release updates from the past there were always arch-sync problems
but they were never the current release-blockers and thus not the focus of
widespread discussion. woody too was in the end delayed because of additional
w-b queues. But this in turn was needed because of the arch-count. Even then
the focus was "why does it take 3 months to add new queues?" and not "would
we be able to release without w-b if we only had three arches?"
The post that is most often cited in this context is Joey Hess' mail from Feb
22: http://lists.debian.org/debian-devel/2005/02/msg01155.html, detailing the
debian-installer guru view of the problem.
> It is one of Debian's strengths that we do our releases synchronous
> through all architectures and have security support for all of them in
> a proper and sycnchronous fashion. If we drop this feature, Debian
> will lose a lot.
>
> Granted, there were problems every once in a while concerning one
> transition or another. However, if these transitions that had to be
> forced into sarge were so problematic that they will be used as reason
> by the release and ftpmaster team to drop architectures, I would have
> expected this problem to be communicated a lot earlier.
The people behind this proposal do not deny their support of the sarge
release. It is _now_ that those problems have to be tackled for etch to solve
them within the next 6-12 months, so etch can release within the next two
years. And some (like my #1 customer) believe this already a tight schedule.
> I'd rather like to implement whatever techniques could be helpful in
> order to help mitigate the problem instead of dropping architectures.
Until now the "burden of implementation" rested on the core teams and the
cooperation between the porters and the infrastructure teams was suboptimal.
While there is much deserved critique on the form and presentation of the
issue, the proposal should be read exactly as that: a proposed framework of
conditions under which arches can be supported by Debians core teams and thus
giving the porters a chance to realize what act they need to get together to
become a REGULAR arch. It also leaves the door open for IRREGULAR arches
which could cater to the specialised needs of embedded/pet hardware. This
last part is still under discussion what is required and possible for the
various arches.
> > the release team, the d-i team, and the kernel team over the past year;
>
> Also, why is it such a problem to keep d-i usable for all
> architectures? We were able to manage boot-floppies for 11
> architectures and d-i was supposed to be a lot easier.
>
> Due to it's much better modularisation many architectural problems
> need to be and can be addressed by the porters of that particular
> architecture. If the porters cannot keep up with the installer, than
> *this* would be a good reason to remove this particular architecture
> from the list of architectures to be released. At least this is what
> I would have expected.
>
> The same goes for the kernel team. Now that we have a kernel team in
> which porters for all architectures are supposed to participate it's
> clearly the porters obligation to ensure that the kernel is fine on
> their architecture, which will lead to being able to release this
> architecture or not.
Partly this is answered in Joey Hess' mail I cited above, and judging from
there and other mails I read about this topic this just means that currently
much porter-work is done by those who also do the general infrastructure
work.
This said, I don't see anyone denying access to porters in any of the mainly
technical projects like d-i, kernel, X and other arch-sensitive stuff.
You are completely right: iff "the porters cannot keep up with the installer,
than *this* would be a good reason to remove this particular architecture
from the list of architectures to be released." Joey Hess explained multiple
times, that he won't drop support for arches which support sarge now. See
http://lists.debian.org/debian-devel/2005/03/msg01530.html for example.
The remaining question is how this will be handled for etch. I see few
incentives for Joey Hess to struggle with arches which cannot keep up with
other requirements for a REGULAR arch. Yes, that would mean that porters of
IRREGULAR arches would have to take up the slack to test and work on d-i, but
on the other hand it would probably lead to more development for both REGULAR
(by current people) and IRREGULAR (by porters wanting to support their arch
as REGULAR) arches.
> > not to mention the time spent by the DSA/buildd admins and the security
>
> Thanks for asking the security team how much work it is to support 11
> architectures. Err... Huh? I wasn't asked? Uh?
>
> However, to clarify this for the readers who aren't involved in
> security: Thanks to the awesome buildd network and a stable Debian
> release supporting 2 architecturs is as easy or difficult as
> supporting 20. The difference is just waiting for one buildd reply or
> waiting for 19.
>
> In most of the times the package will compile fine on all
> architectures since it's a stable Debian release and hence its
> packages are supposed to be rebuildable on a stable Debian machine.
>
> There are exceptions to the rule, but they're not that common. A new
> S/390 kernel and a new MIPSel kernel caused some configure scripts to
> fail, but this can be solved quite easily (once debugged and
> documented) by patching the configure script, so it's just a rebuild
> that is required.
>
> More problematic are cases when software doesn't compile anymore on a
> certain architecture due to compiler problems or kernel limitations.
> However, these are very rare and I only remember two such cases: chbg
> doesn't compile on HPPA anymore and xemacs21 cannot be compiled by a
> buildd anymore.
>
> It causes the security team a lot more work to support more than one
> distribution and/or to support older software for which current
> patches don't work and where the code is too different to easily find
> out whether it is vulnerable as well or not.
>
> Supporting stable and oldstable for a year until we drop support for
> oldstable always causes a lot of extra work for us. Supporting more
> than one architecture for a given distribution is not.
>
> This, however, refers to a working buildd network. However, every
> once in a while one architecture fails because only one buildd is
> configured for stable-security and this buildd is defunct. For
> example, there was no working i386 buildd for a long time. Currently,
> there is no arm buildd (for whatever reason, I dunno).
>
> The hate architecture for many developers however, m68k, has worked
> quite well normally. There are about a dozen buildd machines and
> several of them are configured to compile stable-security. If one
> machine is busy or down, another one picks up the package.
> Additionally, this architecture has quite responsive buildd admins so
> problems can be discussed and resolved via irc or mail quite easily.
>
> These are features I would wish for all architectures: more than one
> buildd configured for stable-security and responsive buildd admins.
Thank you for this clear and in-depth summary of securities' take on the
problem.
> > team. It's also not clear how much benefit there is from doing stable
> > releases for all of these architectures, because they aren't necessarily
> > useful to the communities surrounding those ports.
>
> Could you elaborate on this?
>
> Why is a release, especially for these architectures that you want to
> drop, not useful for their respective user communities?
One example given were embedded systems manufacturers which anyways have to
add heavy modifications after a release to create a value-added product
(similar to what Ubuntu tries).
The answer depends heavily on the arch in question and I have not yet seen
conclusive statements from stakeholders although server arches like sparc
would surely benefit from REGULAR releases iff the porters can support it.
This is a point were the first proposal just falls short of the expectations,
detailed elaborations of the rest raised. The proposal specifies to a high
degree what the people at the Vancouver meeting think is required for a
REGULAR arch, but they failed to propose or even sketch working models for
the IRREGULAR arches. In the ensuing thread-mess several interesting
proposals were made by stakeholders in the potential etch-IRREGULAR arches:
1) Recompile the sources after a REGULAR release.
While this is very easy to do, high risks of needing extra patches which would
be hard to integrate into the archive make this approach less attractive.
2) Setting up a irregular testing structure, tracking those packages which
pass REGULAR testing.
This could be a important intermediate step for demonstrating the ablity of a
porter team to support a REGULAR release.
3) Concentrate on a core set of packages.
Following Progenies componentized approach (http://componentizedlinux.org/),
IRREGULAR arches could try to archive almost-REGULAR releases for a
(arch-specific) useful subset of the pool as determined by arch-stakeholders
(e.g. base+opie for handheld arches or base+net for routers based on embedded
arches).
4) Concentrate on a particular version of packages
Following Ubuntus fork-freeze-release cycle
(http://www.ubuntulinux.org/ubuntu/) IRREGULAR arches could try to stablise a
given snapshot of unstable as basis for e.g. a commercial spin-off tailored
to arch-specific needs. Like 1) this carries risks of additional work as
patches from the snapshot stream back into unstable.
5) Source only distribution
John Goerzen has some code in
http://lists.debian.org/debian-devel/2005/03/msg01387.html for a simple
implementation of 3) with a source-base autobuilder for packages outside the
core set. In the following discussion I got the impression that the approach
is nifty, but especially for the embedded arches fails to address the
pertinent issues (build speed, availability of security updates) as well as
delaying build-failures to a point where recovery is very hard.
6) Work hard to archive REGULAR status
Since all arches are sarge-REGULAR it should not be impossible - given
adequate commitment from porters - to stay that way for etch. The proposal
was just not the best way to tell that it will not be possible with only the
people currently doing the work.
7) Use your imagination
This list is of course neither athoritative nor complete. The discussion has
only started and important stakeholders have not yet voiced their opinion nor
had time to form one. See next comment.
> > Therefore, we're planning on not releasing most of the minor
> > architectures starting with etch. They will be released with sarge, with
> > all that implies (including security support until sarge is archived),
> > but they would no longer be included in testing.
>
> This would be the death of such ports. People using such
> architectures would then have to find out whether Debian is still
> suitable for them, but I doubt that it would. Maybe forking Debian
> into the universal operating system that Debian (then) once was or
> getting some kind of compiled Gentoo or Rocklinux would be the proper
> way/distribution for our porters.
They will have two years time to see whether they can accomodate their needs
within the possibilities the IRREGULAR infrastructure can provide them.
There was doubt expressed whether all arches _need_ REGULAR releases at all.
It would be good to have a place where those potential etch-IRREGULAR arch
stakeholders could collect their take on what they expect from Debian. But
this can and should be organised by those stakeholders and not by someone who
has only second-hand knowledge of the deeper problems involved.
Some of this probably just vanished in the incredible amounts of the preceding
threads and will have to be re-iterated after the smoke clears.
> > This change has superseded the previous SCC (second-class citizen
> > architecture) plan that had already been proposed to reduce the amount of
> > data Debian mirrors are required to carry; prior to the release of sarge,
> > the ftpmasters plan to bring scc.debian.org on-line and begin making
> > non-release-candidate architectures available from scc.debian.org for
> > unstable.
>
> If you need to split the package archive don't call non-i386 second
> class, but ports or tomething. Otherwise you could also name it crap
> or shit because that's what the name implies.
The working title was indeed very badly chosen. Henning has a very good
proposal for the actual naming in the mail I refer to in the beginning of
this mail.
> If the mirrors complain about the sheer archive size of Debian why
> wouldn't it work out if we would only require tier-1 mirrors to mirror
> the entire archive, leave this decision to tier->=2 mirrors and offer
> better mirrroring software that can be configured much easier to
> mirror only a subset of
> {stable, testing, unstable, experimental} x {source,all architecures}
>
> To me it is not clear why we need to go from "you have to mirror
> everythying" to "we will drop architectures" instead of loosen the
> requirements for mirrors and offer a way for them to decide which
> parts to mirror.
As explained further down, REGULAR and WIDESPREAD are orthogonal categories.
> > Note that this plan makes no changes to the set of supported release
> > architectures for sarge, but will take effect for testing and unstable
> > immediately after sarge's release with the result that testing will
> > contain a greatly reduced set of architectures, according to the
> > following objective criteria:
>
> For many people this would imply that Debian sarge was the last real
> Debian release.
>
> > - the release architecture must have N+1 buildds where N is the number
> > required to keep up with the volume of uploaded packages
>
> - and all of them must be configured to build stable-security *and*
> testing-security in addition to stable-proposed-updates and
> testing-proposed-updates.
>
> This would actually help mitigate the problem of broken/down buildds
> with unresponsive admins.i
>
> > - the value of N above must not be > 2
>
> This number seems to be arbitrarily choosen.
One of the reasons is stated by Anthony in
http://lists.debian.org/debian-devel/2005/03/msg01547.html
"The reason for the N = {1,2} requirement is so that the buildds can be
maintained by Debian [...]"
I also gathered, that security updates and announcements should not be delayed
for a REGULAR arch.
> > - the release architecture must have successfully compiled 98% of the
> > archive's source (excluding architecture-specific packages)
>
> This is only achieved for the i386 architecture.
This heavily depends on the definition of architecture-specific packages and
is still und discussion. Points I have seen include:
* relaxing "arch-specific" to also be able to exclude KDE/GNOME from mips
(until someone commits to properly support it for whatever reason he has)
* relaxing the 98%
* porters "getting their act together": REGULAR status demands prompt
compilation of new sources so as not to delay testing propagation by
unnecessary shlibs-skew as well as prompt and reliable security building,
both being indicated by easily reaching the 98%
Finally reaching 98% in itself might not even be the real reason for this rule
but to have an exit ready should an arch fall behind, starting to delay other
REGULAR arches.
> > - the release architecture must have a working, tested installer
>
> Just as usual...
>
> > - the Security Team must be willing to provide long-term support for
> > the architecture
>
> We can only support an architecture if there is buildd support and we
> can only support stable distributions (and to a limited extend
> wannabe-stable distributions aka testing).
> > - the Debian System Administrators (DSA) must be willing to support
> > debian.org machine(s) of that architecture
>
> If they aren't, new people should be added - which is a long-standing
> goal already...
>
> > - there must be a developer-accessible debian.org machine for the
> > architecture.
>
> Just as usual.
These three points are indeed a bootstrapping problem. Though there were
statements from Steve and/or Anthony to the effect that if a potential
REGULAR arch (like amd64) demonstrates commitment and ability to hold REGULAR
status by providing the services REGULAR demands there would be low overhead
to officialy stamp the arch REGULAR since the actual work of supporting it is
already done by the responsible porters.
Contrasted to the current situation without guidelines what a potential new
arch has to be added to Debian this should be a important step forward.
> > We project that applying these rules for etch will reduce the set of
> > candidate architectures from 11 to approximately 4 (i386, powerpc, ia64
> > and amd64 -- which will be added after sarge's release when mirror space
> > is freed up by moving the other architectures to scc.debian.org).
> > This will drastically reduce the architecture coordination required in
> > testing, giving us a more limber release process and (it is hoped) a
> > much shorter release cycle on the order of 12-18 months.
>
> Since this is the first time I hear that so much of "architecture
> coordination" is required for testing, could you elaborate how much
> time this is referring to and which problems the release team has
> faced in particular?
In one case I remember, a delayed build on one arch lead to having binaries
with the same Source-Version on both sides of a library transition. Up to and
including sarge the maintainer had to notice that his package has not yet
propagated to testing (waiting on the new version of some library) and
probably get help from d-release why his package is stuck. Then he has to
realize that there is nothing he can do to get his package into testing
before the new library can be propagated to testing.
To archive REGULAR status after sarge, arches will have to demonstrate that
his won't happen by fulfilling strict performance criteria while on IRREGULAR
arches, this problem doesn't impede REGULAR arches and can be worked around
by e.g accepting version skew or rebuilding against testing.
> I'd be glad if somebody could propose an
> alternative solution, as dropping most architectures would be a large
> regression for Debian and its community.
This thread contains interesting starting points. I hope to add my share in
collecting them in this mail.
> > Architectures that are no longer being considered for stable releases
> > are not going to be left out in the cold. The SCC infrastructure is
> > intended as a long-term option for these other architectures, and the
> > ftpmasters also intend to provide porter teams with the option of
> > releasing periodic (or not-so-periodic) per-architecture snapshots of
> > unstable.
>
> Looking at how "easy" it is to roll out an update to stable with
> regards to the responsiveness of ftpmasters, I find the above option
> quite interesting. FWIW, I'm trying to discuss the next update to
> woody for one month again already. Don't think I've gotton a response
> yet. Doing a "periodic per-architecture snapshot" is probably much
> easier, right?
>
> This would be the death of many ports since it is not possible to
> support those snapshots security-wise by the security team and they
> would be locked out of a stable release as well. We cannot support an
> arbitrary number of source packages. This would get the Debian
> archive totally out of sync.
One of the underlying assumtions - as far as I discerned it - was exactly that
the security team already cannot support those architectures which won't
fulfil REGULAR requirements because of other reasons than unstable
snapshotting.
Also as I tried to explain above, this very much will depend on porters actual
needs for releasing and their willingness to put their money where their
mouth is.
> It's also an interesting thought how many architectures will suffer
> from the _all dependency trap when maintainers don't care about crap
> architectures anymore that won't be released at all.
Hmm, this is a point I have not seen considered until now, people should keep
an eye on this. Though I remember discussion how the _all (= $Source-Version)
can be remedied also in the light of bin-NMUs.
> Done.
AOL!
Thank you all for your patience, reading until the end ;)
Regards, David
--
- hallo... wie gehts heute?
- *hust* gut *rotz* *keuch*
- gott sei dank kommunizieren wir über ein septisches medium ;)
-- Matthias Leeb, Uni f. angewandte Kunst, 2005-02-15
Reply to: