spamfilter package -- help requested

To: debian-user <debian-user@lists.debian.org>
Cc: bcwhite@pobox.com
Subject: spamfilter package -- help requested
From: kmself@ix.netcom.com
Date: Fri, 3 Nov 2000 12:16:05 -0800
Message-id: <[🔎] 20001103121605.A6661@ix.netcom.com>

I've installed and tweaked the spamfilter package (Lars Wirzenius's
procmail recipies).  While I'm largely happy with the filters (I see
*no* spam, though logs show over 80 spam messages blocked since early
September -- despite lifting my ISP's spam filter (largely ineffective),
and posting liberally to Usenet, several mailing lists, with multiple
online references to my email address.

Enough advocacy.

For anyone else using the package, I'd appreciate assistance with...

The problems:

  - Mail sent *from* me *to* several mailing lists, and echoed to me
    from the list, is classified as local outgoing mail, and isn't
    filtered to the appropriate list folder.  This occurs only for
    certain lists.  debian-user, for example, works fine.  The lsec
    list, as an example, doesn't.  Instead, the mail is appended to the
    auto-generated greylist.  I believe this is governed by the mainline
    script, I'm attaching my local copy.

    Diagnostics from the procmai.log follow below.

    I've modified the mainline script to try to correct this without
    success to date.

  - Non-blacklist spam.  Mail which is identified as being spam w/o
    being listed in my blacklist appears to get /dev/null'ed.  I'd
    rather it went to a spam folder.  I believe this is goverened by the
    spam.rules script.  I'm attaching my local copy.

  - Debugging procmail filters is....difficult.  My understanding is
    that I should be able to take a message, say, dumped to the backup
    directory, and run it through procmail with:

      $ cat message | procmail procmailrc-file

    ...which will filter one message 'message' through procmail using
    the procmail rc file 'procmailrc-file'.  But it don't seem to work
    for me.  Clue?

I'm also attaching my main .procmailrc file, though it's little changed
from defaults.

lsec processing log -- this should be filtered to greylist, is instead
being applied to the local greylist.

========================================================================
From owner-lsec@mail.GNU/Linux-Consulting.com Fri Nov 03 11:56:26 2000
 Subject: [lsec] test, ignore
  Folder: /home/karsten/.procmail/backup/msg.3ZH                           3136
procmail: Unlocking "/home/karsten/.procmail/run/backup.lock"
procmail: Executing " test x$DEBUG = xyes"
procmail: Match on " test x$DEBUG = xyes"
procmail: Locking "/home/karsten/.procmail/run/backup.lock"
procmail: Executing " cd $PM_BACKUP && rm -f dummy `ls -t | sed -e 1,200d`"
procmail: Assigning "LASTFOLDER= cd $PM_BACKUP && rm -f dummy `ls -t | sed -e 1,200d`"
From owner-lsec@mail.GNU/Linux-Consulting.com Fri Nov 03 11:56:26 2000
 Subject: [lsec] test, ignore
  Folder:  cd $PM_BACKUP && rm -f dummy `ls -t | sed -e 1,200d`            3136
procmail: Unlocking "/home/karsten/.procmail/run/backup.lock"
procmail: Assigning "OBVIOUSLYTO=(((Original-)?(Resent-)?(To|Cc)):(.*[^a-zA-Z])?)"
procmail: Assigning "INCLUDERC=/home/karsten/.procmail/rules/priority.rules"
procmail: Match on "^Message-ID:.*(myhostname|ix.netcom.com)"
procmail: Match on "^From:.*(kmself(|,|$|@.*(ix.netcom.com|ix.netcom.com))|kmself|karsten|kar
sten@navel|root|root@navel)"
procmail: Match on ! "^Resent-From:"
procmail: Match on ! "^X-Loop:"
procmail: Match on ! "^X-Mailing-List:"
procmail: Match on ! "^(((Original-)?(Resent-)?(To|Cc)):(.*[^a-zA-Z])?)(kmself(|,|$|@.*(ix.ne
tcom.com|ix.netcom.com))|kmself|karsten|karsten@navel|root|root@navel)"
procmail: Assigning "LASTFOLDER= addpat --append $GREYLIST --header "To|Cc|Bcc|Reply-To" --ma
il $USER --from $OFFADDR --dest "greylist" $WHITELIST $GREYLIST"
From owner-lsec@mail.GNU/Linux-Consulting.com Fri Nov 03 11:56:26 2000
procmail: Executing " addpat --append $GREYLIST --header "To|Cc|Bcc|Reply-To" --mail $USER --
from $OFFADDR --dest "greylist" $WHITELIST $GREYLIST"
 Subject: [lsec] test, ignore
  Folder:  addpat --append $GREYLIST --header "To|Cc|Bcc|Reply-To" --ma    3136
procmail: Notified comsat: "karsten@: addpat --append $GREYLIST --header "To|Cc|Bcc|Reply-To"
 --mail $USER --from $OFFADDR --dest "greylist" $WHITELIST $GREYLIST"
procmail: Unlocking "/home/karsten/.procmail/run/procmail.lock"
========================================================================

-- 
Karsten M. Self <kmself@ix.netcom.com>     http://www.netcom.com/~kmself
 Evangelist, Zelerate, Inc.                      http://www.zelerate.org
  What part of "Gestalt" don't you understand?      There is no K5 cabal
   http://gestalt-system.sourceforge.net/        http://www.kuro5hin.org

#################################################################
#
# $Id: mainline,v 1.13 1998/05/22 21:07:22 bcwhite Exp $
#
# Spam Filter -- Mainline
#
#################################################################


# WARNING: If the global lockfile has been disabled, it is vital that
# every delivery rule in the sub-files (special.rules, list.rules,
# spam.rules, and user.rules) have a local lockfile (trailing ":").
# Otherwise, you could get corrupted mail.  The default delivery rules
# in this file all have local lockfiles where appropriate except for
# the spambox which is excluded because it is often set to /dev/null.
# Besides, who cares if spam gets corrupted?

# ------------------------------------------------------------------------ 
# If you are fairly new to procmail and plan to experiment a little
# bit it often helps to have a safety net of some sort.  Inserting the
# following two recipes above all other recipes will make sure that of
# all arriving mail always the last 356 messages will be preserved.
# In order for it to work as intended, you have to create a directory
# named $PM_BACKUP prior to inserting these two recipes.

:0 c: $BACKUPLOCK
* ? test x$DEBUG = xyes
$PM_BACKUP

:0 ic: $BACKUPLOCK
* ? test x$DEBUG = xyes
| cd $PM_BACKUP && rm -f dummy `ls -t | sed -e 1,200d`
# ------------------------------------------------------------------------ 


# ------------------------------------------------------------------------ 
#
# Provide some useful variables:
#
#	OBVIOUSLYTO	Similar to procmail's builtin "TO" but doesn't catch
#			some headers sometimes added when BCCing ourselves.
#
OBVIOUSLYTO=(((Original-)?(Resent-)?(To|Cc)):(.*[^a-zA-Z])?)
# ------------------------------------------------------------------------ 


# ------------------------------------------------------------------------ 
#
# Run the priority rules
#
INCLUDERC=$PM_RULES/priority.rules
# ------------------------------------------------------------------------ 



# ------------------------------------------------------------------------ 
# 
# Mailing lists. Most mailing lists have some characteristic pattern that
# can be tested. Failing that, test ^TOmailinglistaddress.
#
# This section requires heavy configuration and you _MUST_ get it right.
# You _MUST_ catch every mailing list you are on in this section.
#
:0 c
* $ ^$OBVIOUSLYTO($MYADDR|$PWDADDR)
* ? test x$LISTCC = xboth
{
	INCLUDERC=$PM_RULES/list.rules

	:0
	/dev/null
}

:0
* 1^0 !$ ^$OBVIOUSLYTO($MYADDR|$PWDADDR)
* 1^0 ?  test x$LISTCC = xlist
{
	INCLUDERC=$PM_RULES/list.rules
}

# svlug, cni-copyright, lsec, drop my posts:

:0
* (^TO|^X-BeenThere:.*)(svlug.*@|cni-copyright.*@|lsec.*@|galeon.*@)
{
	INCLUDERC=$PM_RULES/list.rules
}

# ------------------------------------------------------------------------ 


# ------------------------------------------------------------------------ 
#
# See if this mail is _from_ me, and if so, use it to update the greylist
#
:0
* $ ^Received:.*from.*$MYHOST
* $ ^Message-ID:.*$MYHOST
* $ ^From:.*$MYADDR
* ! ^Resent-From:
* ! ^X-Loop:
* ! ^X-Mailing-List:
* ! ^X-Been-There:
{

	# Catch list mail which may have slipped by.
	:0
	* (^TO|^X-BeenThere:.*)(svlug@|cni-copyright@|lsec@|galeon.*@)
	{
		INCLUDERC=$PM_RULES/list.rules
	}

	#
	# Since this message wasn't sent to me, just try to add people
	# to the greylist
	#

	:0
	* !$ ^$OBVIOUSLYTO$MYADDR
	| addpat --append $GREYLIST --header "To|Cc|Bcc|Reply-To" --mail $USER --from $OFFADDR --dest "greylist" $WHITELIST $GREYLIST

	#
	# This message was sent to me, so try to add people to the
	# greylist and then let the message through to the rest of the
	# filter
	#

	:0 c
	| addpat --append $GREYLIST --header "To|Cc|Bcc|Reply-To" --mail $USER --from $OFFADDR --dest "greylist" $WHITELIST $GREYLIST
}
# ------------------------------------------------------------------------ 



# ------------------------------------------------------------------------ 
#
# See if we've received this email before and discard it if so.
# Allowing duplicates to pass through just causes the same message
# to get filtered multiple times into the same location, which
# is not useful.
#
:0 Wh: $CACHELOCK
| formail -D 8192 $CACHE
# ------------------------------------------------------------------------ 


# ------------------------------------------------------------------------ 
#
# Rules that make changes to the incoming message
#
INCLUDERC=$PM_RULES/adjust.rules
# ------------------------------------------------------------------------ 



# ------------------------------------------------------------------------ 
#
# Blacklist. The file blacklist file contains a list of addresses (one per
# line) that are matched against the sender. If there is a match, then
# the sender is blacklisted and we discard the mail.
#
# Including the following two lines after the ":0 Whic" line will allow
# blacklisted people to still get through by providing the password.
#
# * !$ ^$OBVIOUSLYTO$PWDADDR
# * !$ ^Subject:.*$PASSWORD
#
# You may also wish to remove the "INCLUDERC" line so blacklisted
# address just go directly into bit-bucket.
#

:0 Whic
* !$ ^$OBVIOUSLYTO$PWDADDR
* !$ ^Subject:.*$PASSWORD
| chkmail --header "Received|From|Sender|Reply-To|Message-ID" $BLACKLIST

:0 a
{
	LOG="Black List: "

	INCLUDERC=$PM_RULES/black.rules

	:0
	$SPAMBOX
}
# ------------------------------------------------------------------------ 


# ------------------------------------------------------------------------ 
#
# Handle any user rules for non-blacklisted mail.
#

INCLUDERC=$PM_RULES/user.rules
# ------------------------------------------------------------------------ 


# ------------------------------------------------------------------------ 
#
# Whitelist. This is similar to the blacklist but with the opposite
# effect.  If we get this far, the letter is personal but it might
# still be spam, so we check the whitelist to see if it is from
# someone we know.  If it is, filter it accordingly.
#

:0 Whic
| chkmail --header "From|Sender" $WHITELIST $GREYLIST

:0 a
{
	LOG="White/Grey List: "

	INCLUDERC=$PM_RULES/white.rules

	:0:
	$DEFAULT
}
# ------------------------------------------------------------------------ 



# ------------------------------------------------------------------------ 
#
# Okay, at this point it hasn't been caught by any of the lists (black, white,
# or grey) so just try to do some intelligent processing.
#



# ------------------------------------
#
# It wasn't from anyone we know. Does it have the password? If so, assume it
# is valid and filter it appropriately.
#

:0
* $ (^Subject:.*$PASSWORD)|(^$OBVIOUSLYTO$PWDADDR)
{
	LOG="Password Provided: "

	INCLUDERC=$PM_RULES/password.rules

	:0:
	$DEFAULT
}
# ------------------------------------



# ------------------------------------
#
# Valid, but no password.
#
LOG="Unknown: "

INCLUDERC=$PM_RULES/nopassword.rules

LOG="(default) "

:0:
$NOPWBOX
# ------------------------------------------------------------------------

#
# $Id: spam.rules,v 1.5 1998/02/10 14:34:33 bcwhite Exp $
#
# Procmail filters for mail determined to be spam.
#
# If nothing happens to the message here, mail will just be dropped into
# the default $SPAMBOX
#
# Written by: Brian White <bcwhite@pobox.com>
#


#
# Send an "ignored" reply back to the originator of the message.  If the reply
# contains the keyword "PASSWORD", then substitute our mail password there.
#
# The "chkmail" test just makes sure that if the from address has been forged
# to appear to be coming from someone we know and trust, they don't get
# bombarded with spam replies.
#
# The theory of operation is this:  Discard any mail from mailer daemons, loops,
# or people forging mail "from me".  Then, if it did NOT appear to come from
# anybody we trust, send a reply (since "send" is a delivery rule, this will be
# the last action).
#
# If it appears to come from somebody we trust but not "me", then it falls
# through so it drops into $NOPWBOX.  Note that this message was already
# marked as spam for some reason such as a blocked domain in the "Message-Id"
# or similar field, so it is possible that this was not valid mail.  Still,
# better to be safe than sorry.  This has also been of use when somebody
# known (whitelisted) temporarily gets their domain on a blacklist.  Without
# this, their mail would have gotten lost and never replied to with an
# "ignore" message (because they are in the whitelist).  At least this way
# it gets filed and will be seen eventually.
#
# This can also be used to provide exceptions from the blacklist.  When
# something is blacklisted, it comes here.  Any messages from a whitelisted
# address will fall out the bottom.  If being used for this purpose, it
# may be best to include $GREYLIST in the "chkmail" line below.

:0
* 1^0   ^FROM_MAILER
* 1^0 $ ^X-Loop:.*$OFFADDR
* 1^0 $ ^(From|Sender|Reply-To):.*($MYADDR|$PWDADDR)
$SPAMBOX
# /dev/null

:0 cW
| chkmail --header "From|Sender" $WHITELIST $GREYLIST

# KMSelf Sun Oct  8 16:45:20 PDT 2000:  what does this rule do?
:0 eh
| (formail -rt -A"X-Loop: filter-$OFFADDR" -A"From: filter-$OFFADDR"; \
   sed -e "s/PASSWORD/$PASSWORD/" $SPAMREPLY) | $SENDMAIL -t

LOG="(spared!) "

:0
$NOPWBOX

###############################################################################
#
# $Id: default-settings,v 1.10 1998/01/13 05:20:04 bcwhite Exp $
#
# .procmailrc
#
# Originally by: Lars Wirzenius <liw@iki.fi>
# Modified extensively by: Behan Webster <behanw@pobox.com>
# Modified extensively by: Brian White <bcwhite@pobox.com>
#
###############################################################################

#
# Defaults
#
# This file is included so that any changes to the Debian package can have
# default values that work through the remainder of the rules even if the
# user does not incorporate those changes into the copy of this file in
# their home directory. Do not remove this line!
#
# To make the incorporation of changes easier, a copy of the original defaults
# file was placed under the "rules" subdirectory as "default-settings.orig".
# To find out what changes have been made to the package since the filter
# was installed into a user account, go to the local .procmail directory
# and do the following:
#
# diff -u rules/default-settings.orig /usr/lib/spamfilter/rules/default-settings
#
# This will give display of the differences. Once incorporation of any changes
# is complete, be sure to do a
#
# cp /usr/lib/spamfilter/rules/default-settings rules/default-settings.orig
#
# so that the next time you wish to do a diff it will be from the latest time
# the incorporation of changes was done.
#
INCLUDERC=/usr/lib/spamfilter/rules/default-settings

#
# Directories
#
PM_HOME=$HOME/.procmail
PM_ETC=$PM_HOME/etc
PM_RULES=$PM_HOME/rules
PM_TEXT=$PM_ETC
PM_VAR=$PM_HOME
PM_BACKUP=$PM_VAR/backup
PM_LOCK=$PM_VAR/run
PM_LOG=$PM_VAR/log
PM_RUN=$PM_VAR/run
PM_MAIL=$PM_VAR/spool

PATH=/usr/lib/spamfilter/bin:$HOME/bin:/usr/local/bin:/usr/bin:/bin

#
# Procmail variables
#
# If the global lockfile has been disabled, be sure to use local lockfiles
# where appropriate. You may wish to enable this if your load goes too
# high because of large volumes of mail. Some 'movemail' scripts (including
# the one available within the spamfilter package) require global locking.
#
MAILDIR=$MBOXDIR
DEFAULT=$MBOXDIR/mbox
LOCKFILE=$PM_LOCK/procmail.lock

#
# Move mail variables (used by /usr/lib/spamfilter/bin/movemail)
#
# MBOXDIR Destination to which to copy filtered mailboxes.
# MOVEDMSG Display xmessage with message counts? (yes/no)
#
MBOXDIR=$HOME/Mail
MOVEDMSG=yes

#
# Other definitions
#
CACHELOCK=$PM_LOCK/cache.lock
BACKUPLOCK=$PM_LOCK/backup.lock

#
# Default mailboxes. Once fully running, you may want to set these
# directly to /dev/null, thus discarding the mail altogether.
#
# You can also set any of these to $DEFAULT if you want it delivered
# directly into your standard inbox.
#
ADMINBOX=Admin
NOPWBOX=Nopassword
SPAMBOX=Spam-Filter

#
# This is my singing and dancing Procmail configuration file.
# In addition to the usual things, it tries to stamp out e-mail spam.
# I've also tried to make it easily configurable, but you should
# carefully read all of it.
#
# Features:
#
# discards mail from unknown people (unless they use password)
# lets people know when they can reach me without password
# easy vacation message (just create the message file)
# sorts bounces in separate folder
#
# This is rather an aggressive mail filter. You probably don't want
# to employ it unless you are really disgusted by the amount of junkmail
# you get.
#
# I'd be grateful if you dropped me a note if you use this file as the
# basis of your own configuration. See also the following URL's:
#
# * <URL:http://www.iki.fi/liw/mail-to-lasu.html>
# What you should know before sending me mail.
# * <URL:http://www.iki.fi/liw/mailfilter.html>
# A general description of my mail filtering scheme
# (describes this file on a bit higher level).
#
# Lars Wirzenius
#

#################################################################
# Configuration section
#
# You need to configure all the following variables. Also read
# through the rest of the file, because you may not like all
# the filtering decisions I've made.
#
# Any settings that could involve shell meta-characters (like
# the pipe "|" symbol used for "or" in regular expressions)
# should be placed within double-quotes.
#
#################################################################

#
# The name of the blacklist file. The file should have one address
# per line.
#
#BLACKLIST=$PM_ETC/blacklist.db
BLACKLIST="/var/spamdb/SpamDomains /var/spamdb/Spammers $PM_ETC/blacklist.db"

#
# The whitelist and the greylist. The greylist is like the whitelist,
# but my setup maintains it automatically. The format for both is the
# same as for the blacklist.
#
WHITELIST=$PM_ETC/whitelist.db
GREYLIST=$PM_ETC/greylist.db

#
# Password that unknown people (those not on the whitelist) must include
# in subject to reach me. It doesn't need to be a great secret or very
# complicated. It's fine to have any shortish, unusual word as the password.
# NOTE that this does _not_ apply to normal passwords, only for this one.
#
PASSWORD=dilcue

#
# USERID is your common userid
#
USERID=kmself
FQDN=ix.netcom.com

#
# OFFADDR is the official address (used in From and X-Loop headers for
# automatic replies).
OFFADDR=$USERID@$FQDN

# PWDADDR is the same, but the username has xyzzy appended (where
# xyzzy is the password). PWDADDR is an alternative to having the
# password in the subject. This circumvents some autoresponses
# that change the subject and have a different From and Sender
# addresses than the one I sent mail to. To use this, you need to
# have _personal_ letters have a Reply-to that is $PWDADDR. Don't
# do it for news articles or mailing lists, because then spammers
# will pick up the passworded address, and you don't want that.
# This does not work on all mail systems.
#
PWDADDR="$USERID-$PASSWORD@$FQDN"

#
# VACATION is the name of vacation file. Creating this file triggers
# vacation responses.
#
VACATION=$PM_TEXT/vacation.txt

#
# Name of the database of people who have recently sent me mail.
# You should probably remove this file when you create or change
# the vacation message. (Ditto for AUTORESPONSE.)
#
VACATIONDB=$PM_RUN/vacation.cache
AUTORESPONSEDB=$PM_RUN/autoresponse.cache

#
# Text file to send to people who should be using the password.
#
USEPASSWORD=$PM_TEXT/use-passwd.txt

#
# Text file to send back to spammers.
#
SPAMREPLY=$PM_TEXT/spam-reply.txt

#
# What comes after @ in the Message-ID for locally generated mail.
# (Used by the rules that filter locally generated mail.)
#
MYHOST="(myhostname|$FQDN)"

#
# Procmail logging.
#
LOGFILE=$PM_LOG/procmail.log
LOGABSTRACT=all
CACHE=$PM_RUN/procmail.cache
VERBOSE=yes

#
# Are we debugging? Set to "yes" or "no" (without the quotes).
# If set to yes, then mail is stored in a mail-backup folder in
# the home directory, and nothing is automatically deleted.
# However, automatic responses _will_ be sent.
#
DEBUG=yes

#
# The LISTCC variable sets how mail that is destined to both you and
# a mailing list should get delivered locally. Valid settings are:
#
# normal Ignore "list.rules" and deliver the mail as though
# it were coming only to you.
#
# both Put a copy of the mail in the appropriate list box
# (via "list.rules") and deliver another copy as though
# it were coming only to you.
#
# list Deliver the mail into the list (via "list.rules") as
# though you were not among the recipients.
#
LISTCC=both

#
# Okay, now load and run the primary filter.
#

# The mainline program does the following (up until delivery takes place):
# - if (DEBUG==yes) make backup of mail into $PM_BACKUP
# - execute $PM_RULES/priority.rules to handle special actions
# - check for mail sent by you and update the greylist
# - execute $PM_RULES/adjust.rules to update/alter mail
# - execute $PM_RULES/list.rules to filter out mailing lists
# - execute $PM_RULES/black.rules on mail with blacklisted addresses/domains
# - execute $PM_RULES/user.rules
# - execute $PM_RULES/white.rules on mail if it's from a whitelisted address
# - execute $PM_RULES/password.rules on mail if it contained the password
# - execute $PM_RULES/nopassword.rules (on any yet uncategorized mail)

INCLUDERC=/usr/lib/spamfilter/rules/mainline

Attachment: pgpFJGE0ErfxT.pgp
Description: PGP signature

Reply to:

Follow-Ups:
- Re: spamfilter package -- help requested
  - From: kmself@ix.netcom.com

Prev by Date: Re: New user rant..urr, questions.
Next by Date: Re: PS/2 mouse (Hewlett packard name on mouse) works with gpm ...not with X11
Previous by thread: Re: rotating Logfiles
Next by thread: Re: spamfilter package -- help requested
Index(es):
- Date
- Thread