Hello world,
as previously announced[1] we had a FTPMaster meeting in the
LinuxHotel[2] in Essen during the week from March 21st till 27th. While
there have been[3] quite[4] a number[5] of blog[6] posts[7] about this
meeting[8], a more formal set of minutes seems to be called for, so here
we go.
Anyways, Mark Hymers, Torsten Werner and myself met on Sunday evening
whilst on Wednesday evening Ansgar Burchardt and, for the
wanna-build/buildd people, Phil Kern joined in. We took the opportunity
of a mostly undisturbed and quite nice work environment to enhance
various aspects of our setup and code, something that would have taken
considerably more time if done outside such a meeting.
During this week we did discuss a whole lot of different topics and did
a great set of changes to our codebase. More than all of us ever
expected in our most optimistic dreams. There have been nearly 300
commits with 94 files changed having 5175 insertions and 1453 deletions
including 16 changes to our database structure. Much of this is not of a
great interest to most of our readers, would bore the hell out of you
and massively bloat this mail. Which is already long. So please bear
with us if we sometimes give a summary only or fail to mention the more
embarassing bugfixes. If you do want more details to one point or topic,
don't hesistate to mail us (or check our git repository if it's code
related)[9].
And before we finally bore you with the details of our work, let us tell
you that there is again a good reason to send condolences to one of our
team members. Similar to our last meeting where we took the opportunity
to welcome a new FTPMaster, we are now torturing Ansgar Burchardt,
having made him FTP Assistant. And like our last promotion - he also
didn't run away screaming.
The main points of the meeting:
- DSA upgraded our main archive machine to Squeeze and started a move
from PostgreSQL 8.4 over to the new 9.0. We had to make some changes
to our codebase and database to get this working, but to our delight
these changes were minimal. Taking it one step further, DSA has
provided a replication of the database from the ftpmaster host over to
the DD accessible copy of our archive on ries, a major reason for
switching to PostgreSQL 9.0.
As this worked out quite nicely we took the opportunity of this meeting
and also asked DSA to upgrade the backports and security machines to
Squeeze and PostgreSQL 9.0. Which means that all our archive hosts are
now the "latest and greatest" stable release and we can use features
from there which we didn't yet have on Lenny.
- The DD accessible copy of the dak installation was moved from
merkel.debian.org (which will be turned off by the end of this month)
to the former ftp-master machine, ries.debian.org[10].
- The update frequency of the DD accessible copy changed. Database
changes are reflected as immediately as the replication allows, while
filesystem changes of all directories except ftp/ are transferred once
every hour. The ftp/ directory is synced once from within dinstall.
If you want to access the database on ries, use psql service=projectb,
it will do the right thing for you. Should you use projectb access for
one of your own projects, to gather whatever data it provides, keep in
mind that we do not guarantee any kind of schema stability there. But if
you can specify your needs we can always think about creating a view for
you - or exporting the data in a defined format like
RFC822/Yaml/whatever if it makes more sense or a direct projectb access
is impractical. Just talk to us.
- We have imported all the metadata that is needed to generate Packages
and Sources files as well as a full set of source contents into our
database. We are already providing the source contents files right
beside all the other Contents files. They are in a similar format to
the binary contents files, one per suite/component pair
(eg. dists/sid/main/Contents-Source.gz). This means it is now easy to
look for similar files (name based) when searching other peoples
code.
- A new generator for Packages and Sources files is also ready. We
(will) put its output into http://ftp-master.debian.org/newdists/ for
the next two to three weeks, to enable other people to check its
output against the old way. We did diff the files and didn't find
anything broken, besides a slightly different sorting, but it won't
hurt to do the extra step.
- We moved the Contents files so their place makes more sense now and
also split them by component. They are now inside
main/contrib/non-free and the debian-installer ones have been renamed
to Contents-udeb-$arch.gz. To keep those tools that are using the
current location working, we have put symlinks into the old places,
pointing at the Contents files of main. We intend to remove the last
compatibility symlinks at the time Squeeze is moved to
archive.debian.org. We probably remove them from experimental, sid,
wheezy, and testing-proposed-updates earlier. None of these changes
are done to the Contents files of Lenny and Squeeze (or their point
releases), they will stay as they are.
- A new field, Built-Using, got implemented in dak and a patch for dpkg
prepared. This will help us to be able to correctly follow
requirements licenses like the GPL have, as a binary that is built
using source from a different package can now declare this
relationship. Our main customers for this will be cross-compilers and
the d-i people. This enables us to get rid of a pretty ugly kludge we
currently have behind the curtain, so-called $release-r0 suites, which
have up until now been our way of dealing with this.
- Work on optimizing the NEW process for packages building differently
named packages on different architectures started. In the future dpkg
will add one extra entry to the .dsc file of every package containing
a list of all "overrides" this source package can generate,
independent of the architecture the packages are build one. This gives
the archive a full list of needed override entries allowing us to
process them once in NEW. This way we save multiple NEW iterations for
additional binary uploads.
Once more the main customer of this will be the linux kernel packages,
but there are also some library packages who will benefit from this
change.
There is a small, but deliberately choosen, disadvantage in the
archive side implementation: unused overrides added this way will be
deleted after two weeks time. In case a binary upload providing the
packages for those overrides comes in later, another run of NEW will
be needed. But as those uploads should be from a buildd we do not
expect much trouble here, especially not as it just means one more
round in NEW. Without this workaround, dak would have cleaned the
overrides out immediately as it checks for unused overrides at every
dinstall. A two week period ought to be enough for everybody...
- In a discussion with the Debian Hurd porters it was decided that the
Hurd port stays on FTPMaster until Wheezy is released. Should they
have managed to get the port into a state that it is released together
with all the others (probably as a technology preview), it is kept in
the archive. Should they not manage this the port will be removed from
the main archive and move fully to debian-ports.org. It may then
reenter the main archive whenever it is ready to get released with the
next release. (Obviously when we say "move to debian-ports" this does
not mean we expect the debian-ports people to "just eat it". They are
running their archive and may have their own needs and pre-conditions
prior to accepting a port, like getting help with the work that needs
to be done or with the hardware for it, so any port who has to look
for new place should ensure to coordinate with the involved people.)
In case it does not work out with debian-ports.org, the removal from
main will still be done, but we are confident that the teams can work
out something acceptable.
- Further on it was decided that the alpha port will be completely
removed from Debian, while the hppa port will move over to
debian-ports, at least for the lifetime of oldstable, which they want
to continue to support. Of course, should anyone really want to
support the alpha port, we will help migrate it elsewhere.
Additionally we will support new architectures like armhf, sparc64,
powerpc64, sh4 and s390x in case someone does the neccessary
groundwork needed prior to an inclusion, and gets all the needed
preconditions settled. If porters want to discuss inclusion, they
should contact us.
- We managed to speed up the creation of the Maintainers and Uploaders
files. Instead of roughly 15 minutes to do that, it is now done within
about a minute, additionally having the side effect that they are now
even correct, the old code included outdated data in these files.
- The long standing project of enabling autosigning for the buildds also
got finished. That means that packages successfully built can now be
uploaded automatically, without waiting for the buildd admin to wake
up/come back from holiday/whatever. The rules for this are simple
* the buildd host must be maintained by DSA and have restricted access
* the key must have a size of 4096 or higher and the private part
must never leave the buildd
* the key expires within 120 days
* there are not more than 2 keys per buildd (so they can do a key
rollover)
* no network access during build time/for the build part
We maintain a set of keyrings, one per architecture, together with a
shell script which allows the wanna-build admins to add and remove
keys as necessary. Keys are restricted to single architecture uploads
(and .all debs for future use). This should speed up availability of
binary packages for architectures not provided by the maintainer and
also ease the load on the buildd maintainers.
- Added binary-all dists files. This was after discussion with buildd
people (Phil) regarding how "Architecture: all" autobuilding might be
able to work in future. We've let it go onto the main mirror network
in case other people have use for the information. The rationale is
that the arch-dependent packages files sometimes lag in their .all deb
versions in order to maintain installability of packages which are
waiting for architecture specific debs. The binary-all metadata will
always contain the latest versions of the .all packages, thus allowing
wanna-build to work out what needs doing. This is, of course, just a
first step towards being able to build .all debs on the buildds.
- We had some discussion about accepting ddebs into the archive but it
needs major changes to dak. We might accept them into experimental as
a first step for early testing but there is no code yet to support
this.
- We checked the implementation of the smart upload server from last
years Google summer of code again but it needs some fixes and more
testing before it can be included into dak.
- For multiarch and XZ handling, we have asked for updates to python-apt
in stable which will simplify adding these features. Assuming that
the SRMs accept the update which the apt team are helpfully preparing
for us, we'll add these as soon as we can get the package updated on
franck.
- As there have been intermittent problems with the current tool which
generates the pdiff files (on occasion causing us to have to restart
the whole diff series), we looked into improving the situation. We
finally came up with the idea to store the affected files (Packages,
Sources, Contents) uncompressed in a local git repository and use gits
ability to output the needed ed scripts (which pdiffs are). The basic
idea would be to save the git commitid relating to the mirrorpushes we
did and then use that, combined with a call to "git difftool --extcmd
'diff --ed' --no-prompt" to output the ed scripts. This ought to be
more stable, and even better, we can replay whole series of pdiffs in
case there is some bug in them again.
As we are no git gurus ourself: Does anyone out there see any trouble
doing it this way? It means storing something around 7GB of
uncompressed text files in git, plus the daily changes happening to
them, then diffing them in the way described above, however the
archive will only need to go back for a couple of weeks and therefore
we should be able to apply git gc --prune (coupled with whatever way
to actually tell git that everything before $DATE can be removed) to
keep the size down.
- Many miscellaneous bug-fixes were included including control-suite
version constraints, some bugs with missing orig tarballs, security
build queue source handling and various other minor issues.
A couple of things which are left over to look at are:
- Changing projectb to use release codenames; the main obstacle to this
is ensuring that scripts calling the dak commands are doing the right
thing. We actually use the codenames on, for example, backports
already.
- Checking that the propup code (which deals with the case where a
higher version is uploaded to, say, stable than is in testing and
unstable) works properly
- A dak package to upload into unstable. Although that's coming Real
Soon Now, honest (especially as one of the ftpteam needs it...).
- Converting dak rm and cruft-report to use the information we now have
in the database so that queries are "live", not "as-of-last-dinstall"
due to them parsing the generated Packages files.
We are of course, happy to take patches for any of these, or other
items. And we will not bite (promise) if you help with cleaning up the
ftp.debian.org bugs list.
And with this we say goodbye and prepare to leave the great hosting that
LinuxHotel once more offered to us. Lets get back home folks.
[1] http://lists.debian.org/debian-project/2011/03/msg00039.html
[2] http://www.linuxhotel.de
[3] http://blog.ganneff.de/blog/2011/03/21/debian-ftpmaster-meeting-2011-1.html
[4] http://blog.ganneff.de/blog/2011/03/21/debian-ftpmaster-meeting-2011.html
[5] http://blog.ganneff.de/blog/2011/03/23/debian-ftpmaster-meeting-2011-2.html
[6] http://blog.ganneff.de/blog/2011/03/24/debian-ftpmaster-meeting-2011-3.html
[7] http://blog.ganneff.de/blog/2011/03/25/debian-ftpmaster-meeting-2011-4.html
[8] http://blog.ganneff.de/blog/2011/03/26/debian-ftpmaster-meeting-2011-5.html
[9] http://ftp-master.debian.org/#dak
[10] http://lists.debian.org/debian-devel-announce/2011/03/msg00013.html
--
bye, Joerg, for the FTP Team
Attachment:
pgpXxLGuarQTR.pgp
Description: PGP signature