[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Meeting Minutes, FTPMaster meeting March 2011



Hello world,

as previously announced[1] we had a FTPMaster meeting in the
LinuxHotel[2] in Essen during the week from March 21st till 27th. While
there have been[3] quite[4] a number[5] of blog[6] posts[7] about this
meeting[8], a more formal set of minutes seems to be called for, so here
we go.

Anyways, Mark Hymers, Torsten Werner and myself met on Sunday evening
whilst on Wednesday evening Ansgar Burchardt and, for the
wanna-build/buildd people, Phil Kern joined in. We took the opportunity
of a mostly undisturbed and quite nice work environment to enhance
various aspects of our setup and code, something that would have taken
considerably more time if done outside such a meeting.

During this week we did discuss a whole lot of different topics and did
a great set of changes to our codebase. More than all of us ever
expected in our most optimistic dreams. There have been nearly 300
commits with 94 files changed having 5175 insertions and 1453 deletions
including 16 changes to our database structure. Much of this is not of a
great interest to most of our readers, would bore the hell out of you
and massively bloat this mail. Which is already long. So please bear
with us if we sometimes give a summary only or fail to mention the more
embarassing bugfixes. If you do want more details to one point or topic,
don't hesistate to mail us (or check our git repository if it's code
related)[9].


And before we finally bore you with the details of our work, let us tell
you that there is again a good reason to send condolences to one of our
team members. Similar to our last meeting where we took the opportunity
to welcome a new FTPMaster, we are now torturing Ansgar Burchardt,
having made him FTP Assistant. And like our last promotion - he also
didn't run away screaming.

The main points of the meeting:

- DSA upgraded our main archive machine to Squeeze and started a move
  from PostgreSQL 8.4 over to the new 9.0. We had to make some changes
  to our codebase and database to get this working, but to our delight
  these changes were minimal. Taking it one step further, DSA  has
  provided a replication of the database from the ftpmaster host over to
  the DD accessible copy of our archive on ries, a major reason for
  switching to PostgreSQL 9.0.

  As this worked out quite nicely we took the opportunity of this meeting
  and also asked DSA to upgrade the backports and security machines to
  Squeeze and PostgreSQL 9.0. Which means that all our archive hosts are
  now the "latest and greatest" stable release and we can use features
  from there which we didn't yet have on Lenny.

- The DD accessible copy of the dak installation was moved from
  merkel.debian.org (which will be turned off by the end of this month)
  to the former ftp-master machine, ries.debian.org[10].

- The update frequency of the DD accessible copy changed. Database
  changes are reflected as immediately as the replication allows, while
  filesystem changes of all directories except ftp/ are transferred once
  every hour. The ftp/ directory is synced once from within dinstall.

  If you want to access the database on ries, use psql service=projectb,
  it will do the right thing for you. Should you use projectb access for
  one of your own projects, to gather whatever data it provides, keep in
  mind that we do not guarantee any kind of schema stability there. But if
  you can specify your needs we can always think about creating a view for
  you - or exporting the data in a defined format like
  RFC822/Yaml/whatever if it makes more sense or a direct projectb access
  is impractical. Just talk to us.

- We have imported all the metadata that is needed to generate Packages
  and Sources files as well as a full set of source contents into our
  database. We are already providing the source contents files right
  beside all the other Contents files. They are in a similar format to
  the binary contents files, one per suite/component pair
  (eg. dists/sid/main/Contents-Source.gz). This means it is now easy to
  look for similar files (name based) when searching other peoples
  code.

- A new generator for Packages and Sources files is also ready. We
  (will) put its output into http://ftp-master.debian.org/newdists/ for
  the next two to three weeks, to enable other people to check its
  output against the old way. We did diff the files and didn't find
  anything broken, besides a slightly different sorting, but it won't
  hurt to do the extra step.

- We moved the Contents files so their place makes more sense now and
  also split them by component. They are now inside
  main/contrib/non-free and the debian-installer ones have been renamed
  to Contents-udeb-$arch.gz. To keep those tools that are using the
  current location working, we have put symlinks into the old places,
  pointing at the Contents files of main. We intend to remove the last
  compatibility symlinks at the time Squeeze is moved to
  archive.debian.org. We probably remove them from experimental, sid,
  wheezy, and testing-proposed-updates earlier. None of these changes
  are done to the Contents files of Lenny and Squeeze (or their point
  releases), they will stay as they are.

- A new field, Built-Using, got implemented in dak and a patch for dpkg
  prepared. This will help us to be able to correctly follow
  requirements licenses like the GPL have, as a binary that is built
  using source from a different package can now declare this
  relationship. Our main customers for this will be cross-compilers and
  the d-i people. This enables us to get rid of a pretty ugly kludge we
  currently have behind the curtain, so-called $release-r0 suites, which
  have up until now been our way of dealing with this.

- Work on optimizing the NEW process for packages building differently
  named packages on different architectures started. In the future dpkg
  will add one extra entry to the .dsc file of every package containing
  a list of all "overrides" this source package can generate,
  independent of the architecture the packages are build one. This gives
  the archive a full list of needed override entries allowing us to
  process them once in NEW. This way we save multiple NEW iterations for
  additional binary uploads.
  Once more the main customer of this will be the linux kernel packages,
  but there are also  some library packages who will benefit from this
  change.

  There is a small, but deliberately choosen, disadvantage in the
  archive side implementation: unused overrides added this way will be
  deleted after two weeks time. In case a binary upload providing the
  packages for those overrides comes in later, another run of NEW will
  be needed. But as those uploads should be from a buildd we do not
  expect much trouble here, especially not as it just means one more
  round in NEW. Without this workaround, dak would have cleaned the
  overrides out immediately as it checks for unused overrides at every
  dinstall.  A two week period ought to be enough for everybody...

- In a discussion with the Debian Hurd porters it was decided that the
  Hurd port stays on FTPMaster until Wheezy is released. Should they
  have managed to get the port into a state that it is released together
  with all the others (probably as a technology preview), it is kept in
  the archive. Should they not manage this the port will be removed from
  the main archive and move fully to debian-ports.org. It may then
  reenter the main archive whenever it is ready to get released with the
  next release. (Obviously when we say "move to debian-ports" this does
  not  mean we expect the debian-ports people to "just eat it". They are
  running their archive and may have their own needs and pre-conditions
  prior to accepting a port, like getting help with the work that needs
  to be done or with the hardware for it, so any port who has to look
  for  new place should ensure to coordinate with the involved people.)
  In case it does not work out with debian-ports.org, the removal from
  main will still be done, but we are confident that the teams can work
  out something acceptable.

- Further on it was decided that the alpha port will be completely
  removed from Debian, while the hppa port will move over to
  debian-ports, at least for the lifetime of oldstable, which they want
  to continue to support. Of course, should anyone really want to
  support the alpha port, we will help migrate it elsewhere.
  Additionally we will support new architectures like armhf, sparc64,
  powerpc64, sh4 and s390x in case someone does the neccessary
  groundwork needed prior to an inclusion, and gets all the needed
  preconditions settled.  If porters want to discuss inclusion, they
  should contact us.

- We managed to speed up the creation of the Maintainers and Uploaders
  files. Instead of roughly 15 minutes to do that, it is now done within
  about a minute, additionally having the side effect that they are now
  even correct, the old code included outdated data in these files.

- The long standing project of enabling autosigning for the buildds also
  got finished. That means that packages successfully built can now be
  uploaded automatically, without waiting for the buildd admin to wake
  up/come back from holiday/whatever. The rules for this are simple
   * the buildd host must be maintained by DSA and have restricted access
   * the key must have a size of 4096 or higher and the private part
     must never leave the buildd
   * the key expires within 120 days
   * there are not more than 2 keys per buildd (so they can do a key
     rollover)
   * no network access during build time/for the build part

  We maintain a set of keyrings, one per architecture, together with a
  shell script which allows the wanna-build admins to add and remove
  keys as necessary.  Keys are restricted to single architecture uploads
  (and .all debs for future use).  This should speed up availability of
  binary packages for architectures not provided by the maintainer and
  also ease the load on the buildd maintainers.

- Added binary-all dists files.  This was after discussion with buildd
  people (Phil) regarding how "Architecture: all" autobuilding might be
  able to work in future.  We've let it go onto the main mirror network
  in case other people have use for the information. The rationale is
  that the arch-dependent packages files sometimes lag in their .all deb
  versions in order to maintain installability of packages which are
  waiting for architecture specific debs. The binary-all metadata will
  always contain the latest versions of the .all packages, thus allowing
  wanna-build to work out what needs doing.  This is, of course, just a
  first step towards being able to build .all debs on the buildds.

- We had some discussion about accepting ddebs into the archive but it
  needs major changes to dak. We might accept them into experimental as
  a first step for early testing but there is no code yet to support
  this.

- We checked the implementation of the smart upload server from last
  years Google summer of code again but it needs some fixes and more
  testing before it can be included into dak.

- For multiarch and XZ handling, we have asked for updates to python-apt
  in stable which will simplify adding these features.  Assuming that
  the SRMs accept the update which the apt team are helpfully preparing
  for us, we'll add these as soon as we can get the package updated on
  franck.

- As there have been intermittent problems with the current tool which
  generates the pdiff files (on occasion causing us to have to restart
  the whole diff series), we looked into improving the situation. We
  finally came up with the idea to store the affected files (Packages,
  Sources, Contents) uncompressed in a local git repository and use gits
  ability to output the needed ed scripts (which pdiffs are). The basic
  idea would be to save the git commitid relating to the mirrorpushes we
  did and then use that, combined with a call to "git difftool --extcmd
  'diff --ed' --no-prompt" to output the ed scripts. This ought to be
  more stable, and even better, we can replay whole series of pdiffs in
  case there is some bug in them again.

  As we are no git gurus ourself: Does anyone out there see any trouble
  doing it this way? It means storing something around 7GB of
  uncompressed text files in git, plus the daily changes happening to
  them, then diffing them in the way described above, however the
  archive will only need to go back for a couple of weeks and therefore
  we should be able to apply git gc --prune (coupled with whatever way
  to actually tell git that everything before $DATE can be removed) to
  keep the size down.

- Many miscellaneous bug-fixes were included including control-suite
  version constraints, some bugs with missing orig tarballs, security
  build queue source handling and various other minor issues.


A couple of things which are left over to look at are:
 - Changing projectb to use release codenames; the main obstacle to this
   is ensuring that scripts calling the dak commands are doing the right
   thing.  We actually use the codenames on, for example, backports
   already.
 - Checking that the propup code (which deals with the case where a
   higher version is uploaded to, say, stable than is in testing and
   unstable) works properly
 - A dak package to upload into unstable.  Although that's coming Real
   Soon Now, honest (especially as one of the ftpteam needs it...).
 - Converting dak rm and cruft-report to use the information we now have
   in the database so that queries are "live", not "as-of-last-dinstall"
   due to them parsing the generated Packages files.
 

We are of course, happy to take patches for any of these, or other
items. And we will not bite (promise) if you help with cleaning up the
ftp.debian.org bugs list.


And with this we say goodbye and prepare to leave the great hosting that
LinuxHotel once more offered to us. Lets get back home folks.

[1] http://lists.debian.org/debian-project/2011/03/msg00039.html
[2] http://www.linuxhotel.de
[3] http://blog.ganneff.de/blog/2011/03/21/debian-ftpmaster-meeting-2011-1.html
[4] http://blog.ganneff.de/blog/2011/03/21/debian-ftpmaster-meeting-2011.html
[5] http://blog.ganneff.de/blog/2011/03/23/debian-ftpmaster-meeting-2011-2.html
[6] http://blog.ganneff.de/blog/2011/03/24/debian-ftpmaster-meeting-2011-3.html
[7] http://blog.ganneff.de/blog/2011/03/25/debian-ftpmaster-meeting-2011-4.html
[8] http://blog.ganneff.de/blog/2011/03/26/debian-ftpmaster-meeting-2011-5.html
[9] http://ftp-master.debian.org/#dak
[10] http://lists.debian.org/debian-devel-announce/2011/03/msg00013.html

-- 
bye, Joerg, for the FTP Team

Attachment: pgpghx4Zfj1Ws.pgp
Description: PGP signature


Reply to: