[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

"Bits" from *ONE* ftpmaster


the following is basically what I wrote today in my blog. I just went
and integrated some of the comments I got and made it fit a mail.

Also, please take the subject as it reads: It's a mail and statement
From *me*. I only also post it via mail as some people do not read blogs
and might be interested in this. And I am interested in getting the "time
to kill?" part to them... :)

So as you might have read[0], I got the delegation of the
debadmin AKA FTPMaster group lately. I've got some pressure[1] to not
accept this delegation, but the response to my blog post about it had
been extremely positive, everyone said I should not deny it (and some
support statements still come in). Knowing to have such a support within
Debian is something I *really* *appreciate*, making it worth to spent time
on my various jobs.

Now, lets take this and write a little about what happened since I
gained that extra group:
  * Aj resigned[2] from anything that requires additional group
    privileges. As a DD run id ajt on a debian.org machine if you want
    to verify this, a non-DD can do this with 
    ldapsearch -x -h db.debian.org -b dc=debian,dc=org '(uid=ajt)' supplementaryGid

Work I did since then:
  * One of my first action was to finally finish the release
    transitions support, something which was waiting for it's final
    merge for a long time. The release team was waiting for this and
    did ping us (ftpteam) multiple times, but now it is there. You can
    read more about this feature in my list mail[3] and my initial
    blog post[4]. (Oh, and something I have missed about everywhere I
    wrote about this: This whole transitions thing is based on an idea
    from Marga!)

  * Added a mapping[5] in dak, requested by Luk.

  * Merged a patch from another ftpteam member, Thomas Viehmann, which
    fixes daks handling of Format 3.0 (something) uploads. Currently it
    rejects them, without the merged patch it just crashed.

  * Changed our daily cronscript to no longer VACUUM/VACUUM analyze
    projectb in every run. Postgres >= 8.1 has autovacuum, and its
    turned on, so we should be fine without extra runs of it. (And
    while I write daily cronscript or cron.daily - I mean dinstall,
    right now run twice a day).

  * Fixed a breakage in code introduced by Debian maintainers which led
    to process-accepted (one main part of the daily cronscript)
    breaking. It was triggered by some octave-$FOO package upload,
    where the ###censored### did add someone twice into the Uploaders:
    field. Not helpful, but now we ignore this case and just warn us.
    (And that package is said to be fixed now too, yay for people
    reading my blog :) ).

  * Added a script to expire old database dumps. We do dump our
    projectb (the main archive database) twice in every run
    (beginning/end), which made 4 dumps a day. While old dumps seemed
    to have been removed on an irregular schedule - the dumps took
    about 71Gb in total. Which pretty much filled up merkels /org, as
    they get synced there. Now, my script (reworked version of a script
    from Florian Reitmeir) works with an "incremental deletion" scheme:

    RULES = [ {`days':14, `interval':0},
              {`days':31, `interval':7},
              {`days':365, `interval':31},
              {`days':3650, `interval':365}, ]

    That means we now keep all files from the last 14 days. Then keep
    one per week. Then one per month and finally one per year for up to
    ten years. All the rest gets deleted. Space usage of our dumps now:
    4.8Gb. Thats 66Gb gained. Source is in dak bzr if you want it, MIT
    licensed. If you improvie it further I love to see patches.

  * Changed the way the projectb copy on merkel gets synced. Up to now
    it was run from a cronjob which syncs a number of files people want
    to see on merkel, including the database dumps, then reloads the
    database using the latest dump. This was run at a "random" time,
    scheduled to be shortly after dinstall. Problem with such a setup
    is that it will miss extra dinstall runs and also might import old
    data, as the timing of the cronjob meant it runs while dinstall on
    ftp-master is still doing its work. So bad bad. It is now done
    using the ssh trigger feature[6] I described a while ago in my
    blog. So our daily cronscript now tells merkel when it should sync
    its database again.

  * Coordinated with the qa group we now also push them, when the sync
    on merkel is done, so they can kick their cronjobs to reread
    projectb and just run whenever they get told that something
    changed. Yet another bad timing eliminated. (Well, will be done
    today). Oh, if you have something that depends on information from
    projectb / ftp-master and try to schedule it to run after dinstall
    - talk to me, we might just push you too.

  * Started talking[7] with the release team how britney, the script
    which manages our testing distribution, should be handled in the
    future. It currently is run fully under control of us ftpmasters
    (and reading hint files from the release team). Ideally this gets
    changed so that the release team has the control of the code and
    whatever gets run with it and then, in some way, maybe using ssh
    triggers, tells us to import a new dataset for testing. That needs
    some thinking, as obviously it should be limited to testing and we
    should make sure it doesn't break the archive, but I think that is

  * Prepared a patch that sanity checks the Provides: field, something
    we currently don't do. Mailed our internal team address about 3
    days ago, to let others look at it, no response yet.

The above should nicely sum up my work in the past few days. Most of it
pretty trivial, but still things that have to be done. I plan to keep
doing things, slowly going from the trivial stuff to more complicated
ones, but there is lots of stuff ahead. Some ideas, and this is a *very*
*incomplete* list just from my head, there are tons of things that one
could do, include

  * Finally allowing dpkg-sig support in .deb files. I once, years ago,
    made a patch for it, which never got applied. (Mostly my fault
    here). Need to rewrite that to fit the actual code.

  * Add a few more sections to our archive. There are multiple bugs
    asking for them, and some others also come to mind. Like java,
    ruby, octave, haskell to name a few, but there might be more
    needed. (Yes, longterm all sections should die and be replaced with
    debtags. I just don't think debtags is there yet?!)

  * Maybe finally drop m68k and add other architectures to the archive,
    like some $foo*bsd one. (Thats for after lenny of course, at the
    time arm also gets dropped).

  * Merge a patch I got from Myon which will keep the changed-by fields
    from .changes files in projectb.

  * Enable bzip2 support for sources, not only for the binary packages.
    I have a patch ready, it's not yet merged. Needs to test if
    anything breaks in stable if we apply it, before we can think about
    it. Worst case it has to wait until we are running lenny. (The
    archive runs on a machine with stable, so allowed features always
    have to be supported in stable).

  * Process the currently waiting requests for pseudo-packages and
    either reject or approve them. After thats done - talk to the BTS
    maintainers to hand the management of pseudo-packages over to them,
    as 99% of the usage of those is in the BTS.

  * Get dak to optionally mail the sponsor of a package too, if not
    already in the list of those getting a dak mail. Which isn't easy.
    One proposal is to just look at the gpg keyid and get the Debian
    uid from it, send mail there. I don't want this, as people are,
    unfortunately, able to disable mail receiving for their @debian.org
    address. Which would give us lots of bounces. No way we like that.
    Similar problems with "Just take one userid that looks like a mail
    address from their key".

  * Improve various parts run by our daily cronjob, as that one takes
    ages (about an hour today). Speed improvements are always nice
    there. (The really longterm plan is a whole new way how package
    uploads get handled by dak, eliminating the queues.)

  * Think about having more dinstall runs a day. Like - every 2 or 4 or
    6 hours could be nice. But that needs discussion within the ftpteam
    and also with the mirror people before it can be done. But 6 hours
    seems to be something nice to have, more might be also ok.

  * Coordinate with DSA to enable the continous AKA WAL archiving[8]
    for our database.

  * Make merkels sync be more instant, ie. no longer rely on a push of
    a database dump whenever dinstall runs, but something thats near
    realtime. Having changes done on ries immediately synced to merkel.

  * Discuss if we want a way of "source-only" uploads. Well, NO, im
    against source-only uploads, as that will lead to people not testing
    their stuff. But one can discuss "has to be uploaded like today, ie
    binaries included and then drop those binaries". That *might* be a
    way to go.

  * More (active) ftp-team member, maybe?!

Now, do you have any time to kill? Do you want to help? Some things you
could do:

  * Go look at our bug page[9] and start there. Someone, I think Don,
    already started using user categories. It would be nice if someone
    is willing to keep up with the bugs and put them in categories. And
    continue to do that in the future.

  * Generally help with our bug page. One nice task some people do, and
    possibly more could also do, is retitling bugs. Especially removal
    bugs tend to have a wrong title. The format is pretty specific...

  * Feel free to submit patches to me. I promise to look at them
    without too much delay and at least give you a hint what might be
    wrong or could be done better. And if it makes sense, doesn't break
    the archive, works, and is something dak should have - I will also
    merge it. Of course trivial patches are way easier in that regard,
    non-trivial ones should go via our ftpmaster address first, to
    leave others a chance to comment on. Bonus points if you provide
    bzr trees one can merge from. The bzr tree you should work with
    is http://ftp-master.debian.org/bzr/ftpmaster-dak/ but keep in mind
    that patches/bzr trees against outdated versions won't have much
    luck to get looked at. :)

  * Keep me happy. However you do it, just don't hug me. :)

I plan to post more of such texts, mainly to my blog. Will try to
summarise them every now and then and send them here. In case you want
to follow it more closely, I just modified my blog[a] to also have
per-category rss feeds, and there is a ftpmaster category. (As well as
some other categories). Feel free to subscribe your rss2mail there if
you don't want to read my whole blog. :)

--- URLS
0. http://lists.debian.org/debian-devel-announce/2008/04/msg00007.html
1. http://blog.ganneff.de/blog/2008/04/17/debadmin-delegation-should-i-a.html
2. http://azure.humbug.org.au/~aj/blog/2008/04/18#2008-04-18-on-freedom
3. http://lists.debian.org/debian-release/2008/04/msg00282.html
4. http://blog.ganneff.de/blog/2008/03/21/release-transitions.html
5. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=456570
6. http://blog.ganneff.de/blog/2007/12/29/ssh-triggers.html
7. http://lists.debian.org/debian-release/2008/04/msg00283.html
8. http://blog.ganneff.de/blog/2008/02/15/postgresql-continuous-archivin.html
9. http://bugs.debian.org/cgi-bin/pkgreport.cgi?pkg=ftp.debian.org;dist=unstable
a. http://blog.ganneff.de/blog/

bye, Joerg
graphviz (old license)
      * Burn it! Burn it all!! 

Attachment: pgpWJBcraTRZp.pgp
Description: PGP signature

Reply to: