Re: How to speed up fetching mail
* Vineet Kumar <debian-user@virtual.doorstop.net> [020427 09:18]:
> * Karsten Heymann (karsten.heymann@gmx.de) [020426 04:56]:
>
> Ah yes, another getmail convert! =D
The friendly debian maintainer warning in 'man fetchmail' convinced me
:)
> Are you using spamc/spamd or calling spamassassin directly? I don't
> have any hard evidence, but the word on the street is that using spamd
> is much, much faster for the scanning portion. As this is probably
> your bottleneck, try that. If you're already using spamc/spamd, then
> I'm not sure where to go. Some more detail about your delivery process
> would help. Does getmail hand off to procmail/maildrop/etc? What is
> your mailbox format (mbox vs. maildir?) I'd say that maildir would be
> faster in this case: no seeking; no locking.
Yes, I'm using spamd, procmail, no local mta, maildir, sanitizer, lbdb
and the duplicate mail recipe from procmail-lib. When running getmail in
verbose mode I see clearly that delivering to my mailbox takes much
longer than fetching from or deleting on the server.
> If it's still all too slow, you might want to try grabbing all the
> mail into a local maildrop and then processing it from there
> asynchronously. What I mean by that is that you could use getmail to
> jsut dump all the incoming mail into one big incoming maildir, no
> filtering of any kind. Then, offline, you could re-deliver those
> messages into your regular mail folders.
That's a really cool idea. Should I use mbox or maildir for this buffer
mailbox? If I use maildir, can I do th following to deliver locally
after fetching:
#!/bin/bash
for mail in `find -type f ~/Maildir/Incoming`
do
cat $mail | procmail -Y -d karsten && rm $Mail
done
or can that break anything? (It's certainly not the way djb dreamed of
it :)
I'll attach getmailrc and procmailrc for reference.
Yours,
Karsten
--
Karsten Heymann <karsten.heymann@gmx.de> <karsten@ecology.uni-kiel.de>
CAU-University Kiel, Germany
Registered Linux User #221014 (http://counter.li.org)
# Please check if all the paths in PATH are reachable, remove the ones that
# are not.
PATH=$HOME/bin:/usr/bin:/usr/ucb:/bin:/usr/local/bin:.
MAILDIR=$HOME/Maildir # You'd better make sure it exists
DEFAULT=$MAILDIR/inbox/
LOGFILE=$MAILDIR/log/MailLogfile
# These mails are quite big, move them before anything else.
:0
* ^To: ct-register-post@listserv.heise.de
Newsletter/heise-register/
#|add-heise-register
# drop duplicate Mail - deactivated
#INCLUDERC=/usr/share/procmail-lib/dupcheck.rc
# Let spamassassin do it's job
:0fw
| spamc
:0e
{
EXITCODE=$?
}
:0:
* ^X-Spam-Status: Yes
unwanted/
# Harvest all mail adresses into lbdb's adress database:
:0hc
| lbdb-fetchaddr
# Sanitize all mails
ANOMY=/usr/bin/sanitizer
:0 fw
| $ANOMY
:0 # Anything for root
* ^To: root
Rootmail/rootmail/
:0
* ^From:.*@fsoe.uni-kiel.de
Fsoe/fsoe/
:0
* ^From:.*@ecology.uni-kiel.de
Fsoe/fsoe/
# Security Announcments are too important to be stuffed into the Lists
# directory.
:0
* ^X-Mailing-List: <debian-security-announce@lists.debian.org>
security/
:0
* ^Subject:.*\[suse-security-announce\]
security/
# Process mailing lists
BASE=Lists
:0 # cpu-users digest
* ^List-Id:.*cpu-users.lists.sourceforge.net
| formail +1 -ds safecat $MAILDIR/$BASE/cpu-users/tmp $MAILDIR/$BASE/cpu-users/new
:0 # Anything about cpu
* ^Subject:.*cpu
cpu/
:0
* ^Mailing-List:.*oezk-info
$BASE/oezk-info/
# Handle the Debian Lists
DEBLISTS=mentors|user|sparc|laptop|python|news
:0:
* $^X-Mailing-List: <debian-($DEBLISTS)@lists\\.debian\\.org>
* ^X-Mailing-List: <debian-\/[^@]+
$BASE/debian-$MATCH/
# Now the SuSE-Lists
SUSELISTS=security|isdn
:0:
* ^Subject:.*\[suse-security\]
* $^Subject:.*\[suse-($SUSELISTS)\]
* ^Subject:.*\[suse-\/[^@]+\]
$BASE/suse-$MATCH/
# Process newsletters and the like
BASE=Newsletters
:0 # Anything from gmx
* ^From: GMX
$BASE/gmx/
:0
* ^From:.*billing@kdt.de
$BASE/kdt-rechnungen/
:0 # Anything from people at KDT
* ^From:.*@kdt.de
$BASE/kdt/
[default]
verbose=1
timeout=120
readall=1
delete=1
message_log=~/Maildir/log/getmaillog
postmaster="|/usr/bin/procmail -Y -d karsten"
#use_apop=1
[mail.kdt.de]
server=mail.kdt.de
username=karsten.heymann
password=XXXXXXXXXX
[pop.gmx.net]
server=pop.gmx.net
username=2123776
password=XXXXXXXXXX
[mail.ecology.uni-kiel.de]
server=mail.ecology.uni-kiel.de
username=karsten
password=XXXXXXXXXX
[pop3.web.de]
server=pop3.web.de
username=karsten.heymann
password=XXXXXXXXXXX
Reply to: