Bits from the listmasters
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi everyone,
As you might have already read from several posts on debian-devel or
debian-user, lists.debian.org has a new spam filter setup. This was
done during the listmaster@ + owner@bugs meeting in Essen, Germany.
But to review everything in a single message:
Internals
- ---------
* Amavis-Setup
The new spam filter setup of lists.debian.org includes the use of
amavisd-new. We are using a feature called policy-banks, where we
have grouped all 180 mailing lists into the following policy banks
plus a few more administrative ones:
* bug * lang-greek
* en-ht * lang-hu-fi
* en-lt * lang-indic
* lang-arabic * lang-indonesic
* lang-asian * lang-romanic
* lang-esperanto * lang-scandinavic
* lang-french * lang-slavic
* lang-germanic
Each policy bank has its own spam filtering setup. Most of it can be
looked at, as it is checked into svn[1]. To find out to which policy
bank a list belongs, look for the X-Virus-Scanned header in the email.
The advantage of this new setup is that now we can distinguse between
different list types, and can set filters and scorings for each
list type on its own.
Mails to each list can be "ham", "maybe-spam" and "spam". For
borderline messages (maybe-spam) we are currently implementing a
queueing mechanism, which allows us to delay these mails for a while
and on recheck them after a defined time has passed.
Many thanks for the help with this setup goes to Alexander Wirt, who
provided several patches for our setup.
Gandalf
~~~~~~~
Don Amstrong is currently implementing a new greylisting daemon we
want to use on lists.debian.org. You now might ask, why another
greylisting daemon? We were inspired by the sort of postfix-weight is
working, but think it has some design flaws. Also we consider some
reporting mechanism back from spamassassin back into the greylisting
daemon quite helpful. Stay tuned, as we want this feature going live
rather soon.
Considering of lurker as webfrontend
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The listmaster team currently evaluating lurker as official
additional webfrontend for the mailing list archive. A few
show-stoppers have been found and documented in [2]. These have been
forwarded to the lurker upstream who is also Debian developer. We
hope to have these changes implemented rather soon, so we can also
offer lurker as an official web archive.
SVN on Alioth
~~~~~~~~~~~~~
We have moved some non-confidential configuration files to an SVN
repository on Alioth, including our SpamAssassin and amavisd-new
configuration. It can be viewed here[1]. If you want to help us with
spam filtering, see if you can improve the SpamAssassin rule files.
Send patches to them to listmaster@lists.debian.org.
Clean up
~~~~~~~~
We use smartlist[3] for running the lists. Smartlist consists of a
series of C programs, procmail and shellscripts. The setup was
deployed originally sometime in 1998 (judging from some file
timestamps) and since then it has evolved. Currently we have 180
lists and each of it has more than 30 files that define how it works
(maxsize, moderation, ...). That sums up to more than 6000 files we
have to maintain.
So now we have been cleaning up and linking identical files
together, reducing the differing configuration files to ideally one
file per list.
During the listmaster meeting this progress started, and about 1000
files have been linked together. The process of simplifying and
unifying configurations is still in process.
There are also some spam filter remnants in these configurations,
that are also being moved into the spamassassin-config.
Cooperation between bugs and lists
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
During the meeting in Essen Don Amstrong and Robert Blarson from
bugs.debian.org team were present. This helped quite a lot, as we were
able to ease the configuration of the spamfilters on both sides, so we
are now using mostly the same SpamAssassin config, which should
improve the spam situation on both sides.
Team members
- ------------
There has been quite a bit of restructuring of listmaster team
members, since we last sent out an official bits from the
listmasters. New members were added, some old members left the team.
All of them did tremendous good work as listmasters and we want to
thank them. Our thanks goes to Jaakko Niemi, Anand Kumria, Frans
Pop, Robert McQueen and Cesar Mendoza.
Recently we have also added three more team members, Don Armstrong,
David Moreno Garza and Thomas Viehmann, last one doing listarchives
only.
whitelist
- ---------
While it is possible to post with an address which isn't subscribed
to the lists, we recommend that you subscribe to our white-list
(http://lists.debian.org/whitelist/) so that our system recognizes
you. This will reduce the risk of false positives causing your mail
to be dropped.
Statistics
- ----------
Listmasters Joey Schulze and Cord Beermann recently blogged about
lists.debian.org and posted some nice statistics. They can be viewed
at [4] and [5].
How to help listmasters against spam
- ------------------------------------
* If you notice a spam in the list archives, press the 'Report As
Spam'-Button.
* If you run some spam-protecting mechanisms like
+ greylisting
+ tdma (challenge-response system)
+ virus scanner
+ spamscanner
make sure that it doesn't reject mails from murphy.debian.org (and
master.debian.org), as our bounce detection software is likely to
unsubscribe you. From the figures above you can see that we filter
a lot of spam and malware, but as we run a
posting-is-open-for-everyone policy, there will always new kinds of
junk that will pass our filters.
* Do not ever report spam received through our lists to third parties
services. They are likely to blacklist us or complain to our ISP,
both of which result in degraded performance for yourself and
others. This is also likely to cause tension between us, our
sponsors and their ISPs.
* Report spam that gets to you through our filters to
report-listspam@lists.debian.org. Please leave all the headers
untouched. The best method is to bounce (as in mutt) them. There is a
plugin for thunder^Wicesomething to do that at
http://mailredirect.mozdev.org/ . DO NOT do this automagically. If
you want to help us, you must make personally sure that the things
you report are REALLY spam.
* If you receive lots of spam and know how to stop it through
procmail or spamassassin, send us (listmaster@lists.debian.org) a
note with the recipe, or contact us in OFTC #debian-lists
* If you really want to use some kind of auto-responder, make sure
that it is sane, and interprets the Lists and Precedence-Headers
correctly so it ignores our mails. If we find that your mail
address issues automatic responses to the list or subscribers,
we'll unsubscribe you from all lists.
* Don't subscribe to our lists with a forwarding mail address, if
something goes wrong with the mail address you are forwarding to,
it will be harder for us to find out exactly which address we have
to drop. Instead, please subscribe with the address on which you
will be reading the mail. You are free to send responses with
another address, so your receiving address isn't published.
* Please keep in mind that the mails to our public lists are
publically archived at lists.debian.org and many other services
on the net. This means that everything in your mail is public,
including your sending mail address.
If you have any questions regarding lists.debian.org, do not hesitate to
contact us at listmaster@lists.debian.org or via IRC on #debian-lists on
the OFTC network.
Greetings,
Martin Zobel-Helas (on behalf of the Debian Listmasters)
[1] http://svn.debian.org/wsvn/pkg-listmaster/
[2] http://wiki.debian.org/Teams/ListMaster/LurkerProblems
[3] http://packages.debian.org/smartlist
[4] http://www.infodrom.org/~joey/log/?200709091425
[5] http://cord.de/blog/index.php?d=29&m=09&y=07&category=6
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
iD8DBQFHC2muST77jl1k+HARAgMmAKCLGTFmB6SVFfEHWjy3mqfV8KjNxgCdGy4b
3lf/Vt+gMYTWw37P0FXmBaY=
=JS34
-----END PGP SIGNATURE-----
Reply to: