Hi
the following is basically what I wrote today in my blog. I just went
and integrated some of the comments I got and made it fit a mail.
Also, please take the subject as it reads: It's a mail and statement
From *me*. I only also post it via mail as some people do not read blogs
and might be interested in this. And I am interested in getting the "time
to kill?" part to them... :)
So as you might have read[0], I got the delegation of the
debadmin AKA FTPMaster group lately. I've got some pressure[1] to not
accept this delegation, but the response to my blog post about it had
been extremely positive, everyone said I should not deny it (and some
support statements still come in). Knowing to have such a support within
Debian is something I *really* *appreciate*, making it worth to spent time
on my various jobs.
Now, lets take this and write a little about what happened since I
gained that extra group:
* Aj resigned[2] from anything that requires additional group
privileges. As a DD run id ajt on a debian.org machine if you want
to verify this, a non-DD can do this with
ldapsearch -x -h db.debian.org -b dc=debian,dc=org '(uid=ajt)' supplementaryGid
.
Work I did since then:
* One of my first action was to finally finish the release
transitions support, something which was waiting for it's final
merge for a long time. The release team was waiting for this and
did ping us (ftpteam) multiple times, but now it is there. You can
read more about this feature in my list mail[3] and my initial
blog post[4]. (Oh, and something I have missed about everywhere I
wrote about this: This whole transitions thing is based on an idea
from Marga!)
* Added a mapping[5] in dak, requested by Luk.
* Merged a patch from another ftpteam member, Thomas Viehmann, which
fixes daks handling of Format 3.0 (something) uploads. Currently it
rejects them, without the merged patch it just crashed.
* Changed our daily cronscript to no longer VACUUM/VACUUM analyze
projectb in every run. Postgres >= 8.1 has autovacuum, and its
turned on, so we should be fine without extra runs of it. (And
while I write daily cronscript or cron.daily - I mean dinstall,
right now run twice a day).
* Fixed a breakage in code introduced by Debian maintainers which led
to process-accepted (one main part of the daily cronscript)
breaking. It was triggered by some octave-$FOO package upload,
where the ###censored### did add someone twice into the Uploaders:
field. Not helpful, but now we ignore this case and just warn us.
(And that package is said to be fixed now too, yay for people
reading my blog :) ).
* Added a script to expire old database dumps. We do dump our
projectb (the main archive database) twice in every run
(beginning/end), which made 4 dumps a day. While old dumps seemed
to have been removed on an irregular schedule - the dumps took
about 71Gb in total. Which pretty much filled up merkels /org, as
they get synced there. Now, my script (reworked version of a script
from Florian Reitmeir) works with an "incremental deletion" scheme:
RULES = [ {`days':14, `interval':0},
{`days':31, `interval':7},
{`days':365, `interval':31},
{`days':3650, `interval':365}, ]
That means we now keep all files from the last 14 days. Then keep
one per week. Then one per month and finally one per year for up to
ten years. All the rest gets deleted. Space usage of our dumps now:
4.8Gb. Thats 66Gb gained. Source is in dak bzr if you want it, MIT
licensed. If you improvie it further I love to see patches.
* Changed the way the projectb copy on merkel gets synced. Up to now
it was run from a cronjob which syncs a number of files people want
to see on merkel, including the database dumps, then reloads the
database using the latest dump. This was run at a "random" time,
scheduled to be shortly after dinstall. Problem with such a setup
is that it will miss extra dinstall runs and also might import old
data, as the timing of the cronjob meant it runs while dinstall on
ftp-master is still doing its work. So bad bad. It is now done
using the ssh trigger feature[6] I described a while ago in my
blog. So our daily cronscript now tells merkel when it should sync
its database again.
* Coordinated with the qa group we now also push them, when the sync
on merkel is done, so they can kick their cronjobs to reread
projectb and just run whenever they get told that something
changed. Yet another bad timing eliminated. (Well, will be done
today). Oh, if you have something that depends on information from
projectb / ftp-master and try to schedule it to run after dinstall
- talk to me, we might just push you too.
* Started talking[7] with the release team how britney, the script
which manages our testing distribution, should be handled in the
future. It currently is run fully under control of us ftpmasters
(and reading hint files from the release team). Ideally this gets
changed so that the release team has the control of the code and
whatever gets run with it and then, in some way, maybe using ssh
triggers, tells us to import a new dataset for testing. That needs
some thinking, as obviously it should be limited to testing and we
should make sure it doesn't break the archive, but I think that is
doable.
* Prepared a patch that sanity checks the Provides: field, something
we currently don't do. Mailed our internal team address about 3
days ago, to let others look at it, no response yet.
The above should nicely sum up my work in the past few days. Most of it
pretty trivial, but still things that have to be done. I plan to keep
doing things, slowly going from the trivial stuff to more complicated
ones, but there is lots of stuff ahead. Some ideas, and this is a *very*
*incomplete* list just from my head, there are tons of things that one
could do, include
* Finally allowing dpkg-sig support in .deb files. I once, years ago,
made a patch for it, which never got applied. (Mostly my fault
here). Need to rewrite that to fit the actual code.
* Add a few more sections to our archive. There are multiple bugs
asking for them, and some others also come to mind. Like java,
ruby, octave, haskell to name a few, but there might be more
needed. (Yes, longterm all sections should die and be replaced with
debtags. I just don't think debtags is there yet?!)
* Maybe finally drop m68k and add other architectures to the archive,
like some $foo*bsd one. (Thats for after lenny of course, at the
time arm also gets dropped).
* Merge a patch I got from Myon which will keep the changed-by fields
from .changes files in projectb.
* Enable bzip2 support for sources, not only for the binary packages.
I have a patch ready, it's not yet merged. Needs to test if
anything breaks in stable if we apply it, before we can think about
it. Worst case it has to wait until we are running lenny. (The
archive runs on a machine with stable, so allowed features always
have to be supported in stable).
* Process the currently waiting requests for pseudo-packages and
either reject or approve them. After thats done - talk to the BTS
maintainers to hand the management of pseudo-packages over to them,
as 99% of the usage of those is in the BTS.
* Get dak to optionally mail the sponsor of a package too, if not
already in the list of those getting a dak mail. Which isn't easy.
One proposal is to just look at the gpg keyid and get the Debian
uid from it, send mail there. I don't want this, as people are,
unfortunately, able to disable mail receiving for their @debian.org
address. Which would give us lots of bounces. No way we like that.
Similar problems with "Just take one userid that looks like a mail
address from their key".
* Improve various parts run by our daily cronjob, as that one takes
ages (about an hour today). Speed improvements are always nice
there. (The really longterm plan is a whole new way how package
uploads get handled by dak, eliminating the queues.)
* Think about having more dinstall runs a day. Like - every 2 or 4 or
6 hours could be nice. But that needs discussion within the ftpteam
and also with the mirror people before it can be done. But 6 hours
seems to be something nice to have, more might be also ok.
* Coordinate with DSA to enable the continous AKA WAL archiving[8]
for our database.
* Make merkels sync be more instant, ie. no longer rely on a push of
a database dump whenever dinstall runs, but something thats near
realtime. Having changes done on ries immediately synced to merkel.
* Discuss if we want a way of "source-only" uploads. Well, NO, im
against source-only uploads, as that will lead to people not testing
their stuff. But one can discuss "has to be uploaded like today, ie
binaries included and then drop those binaries". That *might* be a
way to go.
* More (active) ftp-team member, maybe?!
Now, do you have any time to kill? Do you want to help? Some things you
could do:
* Go look at our bug page[9] and start there. Someone, I think Don,
already started using user categories. It would be nice if someone
is willing to keep up with the bugs and put them in categories. And
continue to do that in the future.
* Generally help with our bug page. One nice task some people do, and
possibly more could also do, is retitling bugs. Especially removal
bugs tend to have a wrong title. The format is pretty specific...
* Feel free to submit patches to me. I promise to look at them
without too much delay and at least give you a hint what might be
wrong or could be done better. And if it makes sense, doesn't break
the archive, works, and is something dak should have - I will also
merge it. Of course trivial patches are way easier in that regard,
non-trivial ones should go via our ftpmaster address first, to
leave others a chance to comment on. Bonus points if you provide
bzr trees one can merge from. The bzr tree you should work with
is http://ftp-master.debian.org/bzr/ftpmaster-dak/ but keep in mind
that patches/bzr trees against outdated versions won't have much
luck to get looked at. :)
* Keep me happy. However you do it, just don't hug me. :)
I plan to post more of such texts, mainly to my blog. Will try to
summarise them every now and then and send them here. In case you want
to follow it more closely, I just modified my blog[a] to also have
per-category rss feeds, and there is a ftpmaster category. (As well as
some other categories). Feel free to subscribe your rss2mail there if
you don't want to read my whole blog. :)
--- URLS
0. http://lists.debian.org/debian-devel-announce/2008/04/msg00007.html
1. http://blog.ganneff.de/blog/2008/04/17/debadmin-delegation-should-i-a.html
2. http://azure.humbug.org.au/~aj/blog/2008/04/18#2008-04-18-on-freedom
3. http://lists.debian.org/debian-release/2008/04/msg00282.html
4. http://blog.ganneff.de/blog/2008/03/21/release-transitions.html
5. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=456570
6. http://blog.ganneff.de/blog/2007/12/29/ssh-triggers.html
7. http://lists.debian.org/debian-release/2008/04/msg00283.html
8. http://blog.ganneff.de/blog/2008/02/15/postgresql-continuous-archivin.html
9. http://bugs.debian.org/cgi-bin/pkgreport.cgi?pkg=ftp.debian.org;dist=unstable
a. http://blog.ganneff.de/blog/
--
bye, Joerg
graphviz (old license)
[...]
Changes
* Burn it! Burn it all!!
Attachment:
pgpka_kDTCQlm.pgp
Description: PGP signature