Re: Parallelizing fetchmail

[Michelle Konzack <linux4michelle@freenet.de>]

Hello Daniele,

Am 2006-05-21 18:52:44, schrieb Daniele Cortesi:

> Procmail checks every message for spam and viruses, introducing some
> seconds of latency, mainly because of DNSRBL checks of spamc.

OK

> The disadvantage of this is that fetchmail launches only one procmail
> for each message and waits for it termination. This leads to a very long
> delay when downloading many messages.

this is, because fetchmail downloads only one message
at a time and wait for the returnstatus of the MDA.

I have the same problem, since filtering eats up to 6
seconds per message...

What I have done?

I let download fetchmail all messages into ONE maildir-box and the I
run a background script which look for messages older then 2 minutes
and spam up to 10 filtersessions...

fnc()
{
  procmail -d <$MSG
}

and in a loop I do something like

for X in $MSGDIR/* ; do
   while [ <condition which checks for maximum 10 sessions of fnc> ] ; do 
     sleep 1
   done
   if [ <some condition which check for the age of the message> ] ; then
     fnc $X
   fi
done

I start tis script all 3 minutes to check, whether
messages are in the $MSGDIR and then run it.

This speed up the downloading rapidly...

fetchmail + procmail can handel only 500-2000 messages per hour

fetchmail alone with a simpel procmailrc file like

----8<-------------------------
MAILDIR=$HOME/Maildir_download
DEFAULT=$MAILDIR/
----8<-------------------------

make more then 8000 messages per hour on a 1 MBit ADSL

Greetings
    Michelle Konzack


-- 
Linux-User #280138 with the Linux Counter, http://counter.li.org/
##################### Debian GNU/Linux Consultant #####################
Michelle Konzack   Apt. 917                  ICQ #328449886
                   50, rue de Soultz         MSM LinuxMichi
0033/6/61925193    67100 Strasbourg/France   IRC #Debian (irc.icq.com)