Re: How to speed up fetching mail

[Karsten Heymann <karsten.heymann@gmx.de>]

* Vineet Kumar <debian-user@virtual.doorstop.net> [020427 09:18]:
> * Karsten Heymann (karsten.heymann@gmx.de) [020426 04:56]:
> 
> Ah yes, another getmail convert! =D

The friendly debian maintainer warning in 'man fetchmail' convinced me
:)
 
> Are you using spamc/spamd or calling spamassassin directly? I don't
> have any hard evidence, but the word on the street is that using spamd
> is much, much faster for the scanning portion. As this is probably
> your bottleneck, try that. If you're already using spamc/spamd, then
> I'm not sure where to go. Some more detail about your delivery process
> would help. Does getmail hand off to procmail/maildrop/etc? What is
> your mailbox format (mbox vs. maildir?) I'd say that maildir would be
> faster in this case: no seeking; no locking.

Yes, I'm using spamd, procmail, no local mta, maildir, sanitizer, lbdb
and the duplicate mail recipe from procmail-lib. When running getmail in
verbose mode I see clearly that delivering to my mailbox takes much
longer than fetching from or deleting on the server.

> If it's still all too slow, you might want to try grabbing all the
> mail into a local maildrop and then processing it from there
> asynchronously.  What I mean by that is that you could use getmail to
> jsut dump all the incoming mail into one big incoming maildir, no
> filtering of any kind.  Then, offline, you could re-deliver those
> messages into your regular mail folders.

That's a really cool idea. Should I use mbox or maildir for this buffer
mailbox? If I use maildir, can I do th following to deliver locally
after fetching:

#!/bin/bash
for mail in `find -type f  ~/Maildir/Incoming`
do
  cat $mail | procmail -Y -d karsten && rm $Mail
done

or can that break anything? (It's certainly not the way djb dreamed of
it :) 

I'll attach getmailrc and procmailrc for reference.

Yours,

Karsten


-- 
Karsten Heymann <karsten.heymann@gmx.de> <karsten@ecology.uni-kiel.de>
CAU-University Kiel, Germany
Registered Linux User #221014                  (http://counter.li.org)

# Please check if all the paths in PATH are reachable, remove the ones that
# are not.

PATH=$HOME/bin:/usr/bin:/usr/ucb:/bin:/usr/local/bin:.
MAILDIR=$HOME/Maildir	# You'd better make sure it exists
DEFAULT=$MAILDIR/inbox/
LOGFILE=$MAILDIR/log/MailLogfile

# These mails are quite big, move them before anything else.
:0
* ^To: ct-register-post@listserv.heise.de
Newsletter/heise-register/
#|add-heise-register

# drop duplicate Mail - deactivated
#INCLUDERC=/usr/share/procmail-lib/dupcheck.rc

# Let spamassassin do it's job
:0fw
| spamc

:0e
{
    EXITCODE=$?
}

:0:
* ^X-Spam-Status: Yes
unwanted/

# Harvest all mail adresses into lbdb's adress database:
:0hc
| lbdb-fetchaddr

# Sanitize all mails
ANOMY=/usr/bin/sanitizer
:0 fw
| $ANOMY



:0				# Anything for root
* ^To: root
Rootmail/rootmail/

:0
* ^From:.*@fsoe.uni-kiel.de
Fsoe/fsoe/

:0
* ^From:.*@ecology.uni-kiel.de
Fsoe/fsoe/

# Security Announcments are too important to be stuffed into the Lists
# directory.
:0
* ^X-Mailing-List: <debian-security-announce@lists.debian.org>
security/

:0
* ^Subject:.*\[suse-security-announce\]
security/

# Process mailing lists

BASE=Lists

:0				# cpu-users digest
* ^List-Id:.*cpu-users.lists.sourceforge.net
| formail +1 -ds safecat $MAILDIR/$BASE/cpu-users/tmp $MAILDIR/$BASE/cpu-users/new

:0				# Anything about cpu
* ^Subject:.*cpu
cpu/

:0
* ^Mailing-List:.*oezk-info
$BASE/oezk-info/

# Handle the Debian Lists

DEBLISTS=mentors|user|sparc|laptop|python|news

:0:
* $^X-Mailing-List: <debian-($DEBLISTS)@lists\\.debian\\.org>
* ^X-Mailing-List: <debian-\/[^@]+
$BASE/debian-$MATCH/

# Now the SuSE-Lists

SUSELISTS=security|isdn

:0:
* ^Subject:.*\[suse-security\]
* $^Subject:.*\[suse-($SUSELISTS)\]
* ^Subject:.*\[suse-\/[^@]+\]
$BASE/suse-$MATCH/


# Process newsletters and the like
BASE=Newsletters

:0				# Anything from gmx
* ^From: GMX
$BASE/gmx/

:0
* ^From:.*billing@kdt.de
$BASE/kdt-rechnungen/

:0				# Anything from people at KDT
* ^From:.*@kdt.de
$BASE/kdt/

[default]
verbose=1
timeout=120
readall=1
delete=1
message_log=~/Maildir/log/getmaillog
postmaster="|/usr/bin/procmail -Y -d karsten"
#use_apop=1

[mail.kdt.de]
server=mail.kdt.de
username=karsten.heymann
password=XXXXXXXXXX

[pop.gmx.net]
server=pop.gmx.net
username=2123776
password=XXXXXXXXXX

[mail.ecology.uni-kiel.de]
server=mail.ecology.uni-kiel.de
username=karsten
password=XXXXXXXXXX

[pop3.web.de]
server=pop3.web.de
username=karsten.heymann
password=XXXXXXXXXXX