Re: Bits from ARM porters

[Aurelien Jarno <aurelien@aurel32.net>]

Hi,

On Tue, Dec 03, 2013 at 01:18:33PM +0100, Hector Oron wrote:
> Hello,
> 
> 2013/12/3 Aurelien Jarno <aurelien@aurel32.net>:
> 
> > On Tue, Dec 03, 2013 at 12:43:36AM +0100, Hector Oron wrote:
> >> 5.2 Setup arm64 debian-ports
> >> ────────────────────────────
> >>
> >>   ⁃ arm64 setup as new bootstrapping port
> >>   ⁃ manual builds could be uploaded but possible lack of space
> >>   ⁃ 9 more packages needed for a minimal bootstrap
> 
> > I have been contacted by Wookey earlier this year about adding arm64
> > port to debian-ports, and everything is now ready. I am still waiting
> > for the buildds email addresses and ssh key though.
> >
> > It is already possible to upload packages, as space is not an issue on
> > debian-ports since we moved to the machine offered by DSA. That said
> > adding a new architecture is a problem (I had to add mips64el on the
> > waiting list recently), as we are lacking CPU and disk I/O. Remember we
> > have about 2/3 of the architectures in the official Debian archive, on
> > a single virtual machine.
> 
> Wookey, any ETA for providing buildds email addresses and ssh key?
> 
> Aurelien, currently debian-ports is a libvirt VM all by itself on
> DL850 hardware, which it is undesired, so debian-ports cannot have
> more resources on that machine until it moves away (preferably as
> Debian service if that's desired).

Yes, I understand this point.

> >> 7 Debian-Ports integration in Debian
> >> ════════════════════════════════════
> 
> > I find strange that it has been discussed and actions have been taken
> > during an ARM bof, without having been contacted. Anyway let's see the
> > various points:
> 
> Aurelien, this is chicken-egg problem, I added that point for
> discussion for the meeting (which was published when announced the
> discussion agenda, few months away). I was wanting to discuss with
> you, once there is some material for discussion. Actions have not yet
> been taken, as it needs to be discussed with you. Let's have a look to
> the issues more deeply:

Ok.

> >>   debian-ports needs a user mailing list.
> >
> > That could be a good idea. Note however that there is currently a
> > buildd-maintainers@debian-ports.org contacting the buildd maintainers of
> > all architectures there.
> 
> There are no public forums for debian-ports discussions. Some
> debian-ports buildd maintainers would benefit from that so share
> experiences and share common problems. It would also be a common
> ground to know which ports are being added/removed, etc... Instead of
> handling that in private. I think we all are in the same thinking that
> a mailing list would be good. So, for fixing the problem:
> 
> >>   Which mailing list should be used for debian-ports discussion?
> >
> > I am opened to suggestions that do not involve the debian-ports machine,
> > as the goal is to reduce the things hosted there.
> 
> While the ARM sprint, Manuel Montecelo suggested to create one in
> alioth or we can either replace debian-ports@lists.debian.org alias by
> proper mailing list, as that list currently spams all porters lists,
> and some people is annoyed with it.
> Options:
>  a. create new mailing list on alioth
>  b. re-purpose debian-ports@l.d.o
>  c. other

I would go to option a, opening a debian-ports group on alioth, and also
hosting the git there (which means one less service to move).

> >> 7.1 Hand machine over to DSA
> >> ────────────────────────────
> >>
> >>   All that needs to be done to handover machine to DSA:
> >>   ⁃ Identify services running on d-ports
> >>   ⁃ Transfer services to DSA machine
> >>   ⁃ Transfer domain names to DSA
> >
> > As already said earlier, I am fine doing that as long as we do not loose
> > features or contributors. What is clearly missing in the list above, is
> > the manpower to do the transfer and the maintenance once the transfer is
> > done (unless DSA is planning to do the full administration, including
> > wanna-build, archive, ...).
> 
> No DSA should not do the service administration. I would expect the
> manpower to do the transfer and maintenance would be done by
> debian-ports team (which includes you, me and other people behind the
> curtains if there are some).

I am personally can provide some manpower from time to time to do the
transfer, but clearly can't be involved that much more. It will mostly
depends how many software has to be rewritten to match the new machine
or DSA requirements. I remember having spent hours rewriting parts of
mini-dak when moving debian-ports to the DSA provided machine to
minimize the effects of a very low fork rate due to virtualization,
while people where sending mails to complain the archive was slow or
broken. I don't want that again.

> > Here is the list of services:
> > - mini-dak for running the archive
> > - FTP server for uploading packages and serving the archive and CD images
> > - web server for serving the archive and CD images
> > - wanna-build
> > - postgresql for wanna-build
> > - web server for wanna-build frontend (pgstatus)
> > - mail server + wbpy to store the build logs
> > - rsync server for serving the archive, currently restricted to mirrors
> >   due to I/O issues
> > - git server and web server for the code and data used on debian-ports
> > - script to create an incoming directory
> > - script for transitions tracking (ben)
> > - POP3S server for buildds behind NAT
> > - DNS server for debian-ports.org
> > - web server for the public website
> > - IPv6: not really a service, but used for buildds without public IPv4
> 
> I opened RT#4808, attaching that list, in case, we decide to take action on it.

Ok. If we want to be able to accept new architectures ASAP, I would
suggest to serve the archive from another host. This is especially 
true for the rsync service used by mirrors, it's really killing the
whole machine with I/O.

Otherwise there are some services that can be easily moved like git
server (see alioth comment above), main web server for the public
website and the DNS server.


> >> 7.2 Enable unreleased suite handling in archive tools
> >> ─────────────────────────────────────────────────────
> >>
> >>   Aparently, keeping separated archive for debian-ports would be good,
> >>   so we can still have waky-hacks in -ports, while do clean bootstrap in
> >>   Debian archives.
> >
> > The unreleased suite is a very important feature that should not be
> > lost, unless we allow porters to NMU packages in a short timeframe and
> > even during freeze. Another feature of the archive is to be able to
> > upload packages versions newer than the current one, but older than in
> > the sources. This allow things to progress even if the current package
> > in unstable is broken and the maintainer doesn't cares about ports.
> >
> > People proposed to add theses features to dak, but nobody actually did
> > the job so far.
> 
> In general, people was not kind on merging debian-ports archives with
> Debian ones, the main points I got:
> - People doing nasty stuff to get ports bootstrapped
> - Wanted a clean bootstrap in official Debian archives
> - ftp-master tools do not support unreleased suite (which maybe with
> multi-archive support could be fixed in the long run)
> 
> In particular, I asked directly to Mark Hymers (ftp-master) and he
> liked more the option of debian-ports having its own separated
> archive.
> 
> In conclusion, I believe debian-ports could and should be running the
> software is currently running with no changes.
> 
> Do you agree?

I agree with that, but in the long term we should either switch to dak,
or spend time to improve mini-dak. In the former case it means to add
features like unreleased or the various synchronisation with the main 
archive (like autoremoval of a package when it is removed from the
Debian archive). I personally don't have time for that.


> >> 7.3 Merge wanna-build DB into official one
> >> ──────────────────────────────────────────
> >>
> >>   ⁃ We want to be able to keep same architecture in both Debian and
> >>     Debian-ports (Note: Debian-ports packages carry scary hacks, and
> >>     Debian bootstrap should start from clean start)
> >
> > Note that we are running the same software than in Debian, even if it
> > is sometimes lagging a bit. Thanks to Philip Kern for his work on that.
> >
> > Remember that it means wanna-build should look for unreleased. Also
> > remember it means that non-DD should be able to access wanna-build. If
> > possible the same persons should also have a shell with access to the
> > packages files and wanna-build to be able to handle transitions and
> > schedule NMUs, like the release team is doing.
> 
> This point ought to be discussed with Philip Kern, I have not yet got
> around to that.
> Philip, would you mind to let us know your thoughts on the matter? Do
> you prefer to keep two separated databases for wanna-build or merge
> them? Please, also consider that ftp-master is working on
> multi-archive support and that might need a database re-design.

In addition to that, you also have to see if it is desirable for the
users to have a single or separated interfaces.


> >> 7.4 Enable non-DD uploaders for d-ports
> >> ───────────────────────────────────────
> >>
> >>   ⁃ Recognise porting work in the NM process independently of whether
> >>     individual packages are listed as being maintained by that
> >>     person. Needs some tools or existing tools adapting to ports
> >>     structure.
> >
> > I don't really see the point, as it is already the case. Actually most
> > of the uploaders in debian-ports are non-DD, and it is something that
> > should not be lost in the transfer either.
> 
> If debian-ports is converted to official Debian service, then all the
> user database is extracted from LDAP, which allows access to DD, but
> non-DD have a problem (similar with alioth guest accounts). Somehow
> that needs to be solved. While in the sprint, few DAM members were
> around and we were able to discuss it a little bit. Apparently, the
> consensus was to make those people Debian contributors or the such
> early in the process, if I understood correctly.
> 
> To be clear, we all want those non-DD to continue to do their work
> transparently, but there are some technical/structural challenges to
> solve before making it a reality. Maybe some DAM member can throw more
> light on how can we enabling those people, maybe is not that much of a
> problem if we keep a separated archive for debian-ports.

We should clearly keep the process very simple here, as otherwise most
of the people would not have contributed. This is especially the case
for ports retired from the archive, for which *I think* people would
have contributed to the port before if it was easier (according to their
view) to contribute to Debian. The fact it's simple to do things on
debian-ports helps here.

On the other hand we don't give them full power either. If we restrict
the upload and wanna-build access to a specific architecture, then they
can only harm their own port and it's something that they have to deal
within porters. That's probably the point to improve, to get a real team
of porters for each architecture, and not a set of individuals without
coordination, sometimes sending contradictory requests, as it has been
the case for some architectures in debian-ports.


> Aurelien, here I am trying to solve a long standing problem, not
> making it worse, which involves several teams and I would like to be
> carefully planned before any action is taken. Somehow, I assume the
> move of debian-ports into Debian infrastructure is wanted, if not,
> please say so and I'll stop here. Thanks.

I am pretty fine with the move of debian-ports into Debian
infrastructure, I just want to make it right, and to not loose features
or people in the move, that will lead us to create debian-ports-2 in a
few years because the thing has become to complex.

Aurelien

-- 
Aurelien Jarno                          GPG: 1024D/F1BCDB73
aurelien@aurel32.net                 http://www.aurel32.net