Hello world, as you probably read on debian-project[1] there was a meeting of the FTPTeam in Fulda last weekend. Mark, Alexander and myself met from Friday til Sunday to discuss various topics we had on agenda - and to discover multiple new restaurants all around my place. :) And while I still miss out on Baklava (how can a turkish restaurant seriously run out of that?) I thought we shouldn't have you miss something that smells, tastes and looks like minutes, so here they are. I'm sorry for all the length of it, but flipping between a simple "We met, we did something, sod off" and this one, we somehow voted for a slightly longer edition. Have fun. :) [1] http://lists.debian.org/debian-project/2010/08/msg00314.html 1. New FTPMaster As you could already have read on d-d-a[2], there is a good reason to send condolences over to Torsten, as we took his absence as the best opportunity to promote him from FTP Assistant to FTPMaster. After all he couldn't run away screaming as he wasn't attending. [2] http://lists.debian.org/debian-devel-announce/2010/09/msg00006.html 2. Call for volunteers As usual when posting longish mails to d-d-a (yes, I know we are on -project) about our nice and beloved FTP Team, a call for volunteers is on order. And here it is! Ever felt compelled to do the hard groundwork? Ever wanted to help at a nicely central place inside Debian? Or just want to write some python code and still look for a good place to stick it in? Here we are. Come join the Nav^Wteam. Just sign over there to the right, or even easier, mail us. We won't bite you, thats for sure. At least not right away. :) The criteria are the same as always: You need to be a DD (except for coding only, though it helps to know the usual flow of a package) and you need to be able to deal with the existing team members. An occasional flame should also not disturb you, if you are working in the NEW queue you will stand between the people uploading and the packages entering the archive, rejecting something is not always liked much. (But you also get positive replies and thanks, to keep your spirits up :) ). And - if you get headaches when reading legal texts - we all do. But it is needed and things like NEW are mainly about that, the ftpteam is *the* one place to decide if something is ok for Debian to distribute or not, and you will have to take this decision. (Yes, there is more, but this is the master of the points you check). Obviously the other points I made in earlier mails, like [3], still apply too. [3] http://lists.debian.org/debian-devel-announce/2010/03/msg00003.html 3. BYHAND Turns out our BYHAND handling was broken since a few days. Seems like we never got that back alive after Essen. Shows how much BYHAND is used today, but with a perfect timing we had someone place a BYHAND upload in the queue, so Mark took the opportunity to fix it up. We also decided we do no longer want to have BYHAND. We want those things to be as automated as possible, and as such we invite the remaining few users of BYHAND (about 2 packages) to talk to us, so we can switch them over to the automated way other packages are using (for example d-i). This will help them as well as us as it'll mean their packages go through more quickly and don't have to wait for us to process them. 4. Volatile archive A while ago the volatile team approached us and asked if we can take over their archive. We did not exactly take it over, but starting with squeeze, the volatile suites will be integrated into the normal ftp.debian.org mirrortree. This weekend we enabled squeeze-volatile on ftp-master and setup the needed scripts so that the volatile team can fill it with packages whenever needed. Please note that the general handling of volatile starting with squeeze is now different to the way volatile worked in the past. All packages now have to pass stables proposed-updates queue before going into volatile. Stay tuned, the volatile team will send out more information about its handling later on, the exact policy how the suite is run is with them, not us. 5. Security archive This is one blocker in the way to a stable squeeze release as this archive is yet unable to process the dpkg v3 formats. Having recently upgraded backports to the current dak codebase I now know what I have to do with the security archive (same old code there), and my current schedule means it should be done real soon now. There is one anomaly in the security archive, namely the script used to actually release DSAs, which needs lots of work (it's BAD, mmmmmmmmmmmmkay?!) to continue working, but otherwise it should be the same amount of work as getting backports working. Additionally we discussed ways we can merge security into the normal FTPMaster archive. Obviously this is not a simple step to do because we must ensure that embargoed security issues stay embargoed until their normal release and such requires a good change in our database layout to deal with it. But we already have a good picture in which way to move and definitely follow this as a long term goal. This will have the advantage that things like out of sync orig.tar files won't be able to occur and migration of security releases to stable should be much smoother (will turn out to be a simple suite copy, instead of a move between two independent archives). 6. data.debian.org Right now this is slightly complicated to implement in a halfway sane way. The changes we need involve ensuring the location code is fully working, probably some database adjustments, and getting rid of some assumptions our code (may) make. This point and the merge idea from point 5 will probably best get addressed in a week long coding sprint of our team similar to the one we did last year in Essen. A very rough aim for that is currently "somewhere springtime 2011", but of course this should be after squeeze released. 7. changelog/metadata export We finished a script to export changelogs, README/NEWS.Debian and copyright files which are used by packages.debian.org code. Up to now pdo was extracting those itself, taking up lots of resources doing this, while occasionally going wrong. This is now offered for all FTPMaster run archives (security follows as soon as it is upgraded). We will make the export tree available publically soon, so whoever want to use the data can freely do so. DDs who have a service that want to use them should contact either me or (better) mirrors@debian.org to arrange a mirror of the exported data onto the Debian machine their service runs on. Also, always keep in mind that we are happy to export more metadata. When you have a service that regularly needs to generate some data based on our archive - talk to us. Describe what you do, what for, what kind of data you use and how we can help you (what we should export). If it is possible for us to do, we will help you out. We don't bite, don't waste precious CPU cycles all over the places. :) (All the better if you volunteer to write the neccessary code too, that almost guarantees that we don't bite. Python and dak, but talk to us, we can guide you to the right places in the code) 8. We changed our sending mail addresses. In the past we had at least "Debian Archive Maintenance <ftpmaster@ftp-master.debian.org>" "Debian FTP Masters <ftpmaster@ftp-master.debian.org>" "Archive Administrator <installer@ftp-master.debian.org>" as possible From headers from mails generated by dak. From now on this should be limited to "Debian FTP Masters <ftpmaster@ftp-master.debian.org>" so you may want to adjust your mailfilters accordingly. We will keep the installer@ address working for a week or two more, before we disallow any more mail send to it. This is also true for the backports archive (just replace domain). Additionally backports mail handling is split in two areas. For the technic you can reach the FTP Masters behind ftpmaster@backports.debian.org, while policy and all other questions, unless they better fit on one of the mailinglists, should be directed to team@backports.debian.org. 9. dak rm Alexander helped our removal tool to gain a new option. From now on we can close bugs associated to a package when doing a sourceful removal. Obviously this is not enabled by default, but an option we have to select whenever appropriate (not all removals mean all bugs can be closed, like when its "just" a source rename), but this can greatly help the QA team. 10. bts categorize And while he was already at the code, Alex also fixed our bts categorize script and ported it to a different bts soap whatever python library. So the usertags on ftp.debian.org should look right again, finally. 11. debian-ports Following a short chat I had with Aurelien Jarno about debian-ports.org and its archive, we discussed if FTPMaster could offer help with running such an archive. In general we aren't unhappy with it but we also do not want to end up the main force behind it. Together with the nature of the archive and the attached constraints (different architectures need to have the same source version - with different code/patches applied for example), the handling in dak is not straightforward nor entirely there yet. Most probably it will mean a set of mini archives, one per d-p architecture, and is as such attached directly to point 4 and 5. When the work for those is done, it will be relatively easy to provide the support d-p needs. Exact conditions for such work still need to be worked out, but basically something along the line that FTPMaster does the technic while someone else is actually responsible for it. So 2 or more DDs need to sign up for the work per arch, if they drop out and noone replaces, it gets removed, etc. Realistically we will not be able to offer this before late 2011. 12. rmadison / dak ls While this initially was (and still is) a tool provided by the QA people, we really dislike the current way it is running. Having to shell out to run a dak command, parse and send out its output is really not a way that is good (even though it works), and we want to provide a saner interface. As other services already provide SOAP interfaces to talk to (think of the BTS), this will be the way we will follow to. We intend to publish a clear API documentation and think of not only providing the equivalent of "dak ls", but also other querying tools like control-suite and override should be accessible. 13. dinstall replacement You might know that dinstall is "the thing ftpmaster let run 4 times a day getting us new things to upgrade to". What you might miss is that this is a nice 1280 lines shell scriptset, which even includes a pretty complete state machine - all in simple bash. Unfortunately it misses one feature, which was a nice way of synchronizing some steps we do in a nice way. While I actually know how to extend the scripts to do this, my fellow team members nearly threw me out of the window when I told them I want to extend the state machine and instead asked me to rewrite it all boringly in python muttering something about readability and sanity. While I do think my shell scripting pretty sane and readable, for the sake of going to the next restaurant instead of arguing I gave in and the rewrite is now on the todo list for "as soon as remotely possible, best two days ago", so I expect it to be done before end of October. 14. 3.0 git/bzr/$VCS First: For the following writeup we do not care if the VCS in question is git, bzr, hg or whatever. Current preference seems to be git, for its speed and flexibility, but the general arguments apply to any distributed VCS that could be used here. So interchange git with whatever you would like to see. We realize that this is a hot topic and understand that many people would like if they can *entirely* manage their packages using a distributed version control system, even as far as having a "git push" being the actual upload. We do understand the desire and reasons for wanting something like this but do not think the real usage of it - pushing entire repositories to ftp-master - is a goal we can currently support. The recent discussion, started a little before the DebConf Source Format BoF and then continued on debian-devel[4] pretty much listed the points the FTP Team has with this approach as Russ had discussed with us before. That is also the reason we did not participate any further - if there is nothing new to say there is no need to mail. :) So, to keep this already pretty long mail from exploding, our main trouble with such a package format is still the already mentioned work we have to do in the NEW queue. There needs to be a reasonable way to ensure Debian is actually allowed to distribute what we put on our mirror and having huge git repositories with lots of commits and branches will make NEW reviews near impossible with the current set of tools. How are we supposed to ensure that there wasn't a number of files with headers like "THESE FILES ARE FOR INTERNAL USE OF $COMPANY ONLY, IF YOU DISTRIBUTE THEM WE WILL SUE YOU"?[5] When the file was removed using git rm a hundred revisions ago, with a commit message not even clearly describing it? We would still distribute it, people could just checkout an old version and have it. -> Bad. Yes, shallow clones and allowing only something like one or two revisions in an upload seem to help, but then we don't really see the point in a new format. It would lose so much a full repository can give you, that one can directly use a 1.0 with patches or another of the existing 3.0 formats. Besides this we do invite people interested in a 3.0 $VCS format to work on the problems and are happy to participate, so we get to something we all can live with it - or all see it won't work out. But we will not be the drivers behind it, our plate is too full already. [4] http://lists.debian.org/debian-devel/2010/08/msg00244.html [5] Yes, we had such stuff in NEW, not only once, not just twice. Below here follow a few more points which might get hard to understand unless you are in dak / team internals anyways, but we list them nonetheless, as we had them over here and/or put them on todo for the future. There is nothing user visibly following, so you might skip if you want to. 15. control-suite sanity Right now there is no sane version checking done when we import new data into a suite using c-s. This means that in theory the release managers could put packages/versions from any suite into testing (say, an oldstable version into testing, an experimental version into testing), completly violating any version constraint the suites have when processing uploads. This is currently solved/worked around by having a very big (virtual) hammer flying above the release/volatile peoples heads - should they ever make use of this capability and take version from elsewhere than the allowed source suite the import process would be stopped by us until we have all code changed to fully check it. Obviously this was never needed, (britney being in the release teams hand since April 2008!), which is also the reason why it is still an open issue. But the acceptance of more archives and suites with varying teams responsible for increases the priority of this task, so we had a discussion how to properly design and code this. It seems having it implemented in the database is the way to go, but the exact details aren't written in stone yet. Mark agreed to take a hammer and chisel to do so. 16. .changes parsing We currently parse .changes files at multiple places. This is insane and actually also stupid as changes files are only trusted at the initial parsing done by process-upload (to be called with --initial then). Everything later on, even repeated p-u calls, must only use the database to gather the needed information, we can't trust the .changes anymore to contain correct data (just think about removed byhand parts as an example). This needs changes in code, but should be a good task for a new volunteer as it doesn't seem to be hard to do, just needs time. :) 17. dak config Our text config file for dak is a nuisance that actually complicates code in unneeded ways, duplicates information and is basically hated. We already moved many parts of it into the database (for example about all attributes of a suite), but should move nearly anything else over. Our goal is to stop with a config that only tells us where we find the database, rest is taken from there. Compared to our Essen meeting last year we are much nearer to that goal nowadays. 18. Contents We are still working on this, mainly due to time constraints on the side of the main driver behind this. Should be ready soon. 19. g-p-s (aptconfig class) When we moved to the new ftp-master host franck.debian.org we had been astonished by its speed - and immediately set out on the task of making it go slow. We did not yet succeed with it, but we managed to have many things run in parallel that were very slow before. One of those things are the slow apt-ftparchive runs. While this ran "serial" over all suites and architectures in the past, easily taking up 95% of the time a "dinstall" run takes, we wrote a script to run in parallel. Thus we now regenerate the Packages/Sources and Contents files 4 times an hour (yes, its that fast that we don't notice these often runs, and using the caching feature there enables even faster runs during dinstall). We just do not like the script that allows us to do this much, we need a real class to build up the apt config file that every apt-ftparchive needs to be fed, it is too much of a hack currently. This is also work in progress, currently with Mark. 20. highlight packages in NEW fixing rc bugs As it always happens that packages hitting release critical bugs need NEW processing, we plan to hilight these packages in the various output formats. This should make it easier for us to fastrack these packages and should also make it easier for the release team to review (and even veto) these packages during a freeze. 21. untag pending bugs if a package is rejected As the bugs fixed in packages in the NEW queue are currently (semi-) automatically taged "pending", Jan Hauke Rahm suggested to untag them, if a package get's rejected from NEW. There's already a patch floating arround, which needs to be reviewed and merged. -- bye, Joerg <rra> I don't know that "useful" is the best overall descriptor of the things Ganneff picks to put in his signature. :) <liw> obviously there is too little useful things said so he has to quote silly things instead * Myon sees Ganneff adding yet another liw quote
Attachment:
pgpPeWLHWb4rS.pgp
Description: PGP signature