Re: Networking -- use of two Internet connections for one server with round robin DNS -- web okay, but should I do mail this way too?

[Stan Hoeppner <stan@hardwarefreak.com>]

On 7/10/2011 8:31 PM, lee wrote:
> Stan Hoeppner <stan@hardwarefreak.com> writes:
> 
>> On 7/10/2011 7:26 AM, lee wrote:
>>> Stan Hoeppner <stan@hardwarefreak.com> writes:
>>>
>>>> On 7/9/2011 12:00 PM, lee wrote:
>>>>
>>>> Just checking for the existence of rDNS is no longer sufficiently
>>>> effective against bot spam from infected residential hosts.  This is
>>>> because many/most? ISPs have rDNS for most of their IP addresses,
>>>> whether dynamic or static.
>>>
>>> Well, most rejects are because the HELO checks fail.  There are only a
>>> very few that fail because of the rDNS check.  There isn't much SPAM
>>> getting through; I'm getting less than one message per day.
>>
>> If your EHLO check is first it would make sense that it will reject more
>> than the rDNS check.  Reverse the order and you may see that metric
>> reversed.  It's good to hear you're not seeing much with your setup.
>> I'd guess you have low mail flow on that host.
> 
> Yes, the HELO checks are first.  It seems to make sense that way.

Most MTAs lookup the hostname long before receiving EHLO.  One can
reject mail sooner if configured to not wait for SMTP logging info
(HELO, MAIL FROM, etc).  If you wait it doesn't really matter as you
have no speed gain.  This 'trick' would only be applicable to extremely
high volume MX hosts, i.e. > 300 connects/second.  Such systems likely
have upstream firewalls killing most of this traffic beforehand though.
 Lots of different ways to kill spam.

> What do you consider low mail flow?

Personally?  Generally less than 50,000 connects/day.  If you ask an
admin at one of the universities with 30k+ students, he'd probably tell
you anything less than 500k connects/day is low mail flow.  This answer
depends on who you ask.

>>>> http://www.hardwarefreak.com/fqrdns.pcre
>>
>> I take it you are you really new to managing a mail server.  dnsbls have
>> been around forever, and every mail OP uses one or another, if not 5 or
>> more.
> 
> That they are around for a long time doesn't mean that I have to like
> them or to have others decide what mail to accept or not to accept.

That they have been around a long time, and have a solid reputation for
blocking spam and not ham, is the key.  I don't know why you wouldn't
'like' them.  I think you simply need more exposure to spam fighting and
the excellent free tools available to you.

Notice I didn't mention SORBS or FiveTen, et al--horrible reputation for
blocking ham.  Start with Spamhaus' dnsbls and branch out from there, if
needed.

>> Have you heard of SpamAssassin?  Both restrictions make
>> reject/keep decisions for you.  Using this PCRE table is no different in
>> that regard.
> 
> Spamassassin seems to be doing a good job; I don't know about your
> table.  Both ways of filtering make decisions for me --- that's the
> idea.

The same is true of configuring EXIM/Postfix/etc to reject based on lack
of PTR, incorrect HELO, etc.  It's called automation.  You already allow
your MTA to make block/accept decisions for you.  Using external or
other tools is no different in this regard.

Snowshoe spammers obey all the SMTP and DNS rules--correct rDNS, HELO,
etc.  Good luck fighting snowshoe spam without automation such as ZEN,
DBL, local lists, etc.

>>>> This Postfix PCRE table consists of 1600+ rDNS patterns of residential
>>>> broadband/SOHO ISPs around the world, and is extremely effective at
>>>> killing bot spam, while putting very little load on your server.
>>>
>>> Sounds like it must have taken quite some work to put the list together,
>>> and it would need to be maintained.  
>>
>> The table was built over a relatively long period of time, and does take
>> a small amount of time to maintain.  ISPs don't add new residential rDNS
>> patterns very often.  When we spot a new one a regex is created to match
>> it.  Changes average about one add every 1 to 2 months.
> 
> Hm, that's a pretty low rate.

Changes occur at approximately the same rate as changes in the target
set of worldwide ISP rDNS patterns.  They simply don't change often once
you have the initial table built.  About once a month an ISP somewhere
in the world will start using a different rDNS pattern than before, or,
more often, they simply introduce a new pattern.  When one of the
fqrdns.pcre users identifies it, I add a regex to the table to match it,
and only it, which keeps FPs low or nonexistent.

>>> Won't graylisting work as well?
>>
>> I see than indeed you are new.  Greylisting will usually defeat bot spam
>> as bots never retry.  The problem is the delivery delay introduced
>> (minutes to hours).  This doesn't work for those ordering last minute
>> air fare and need to print their boarding pass.  With greylisting that
>> boarding pass email may arrive an hour later.  Greylisting also sucks
>> system resources due to the triplet database.
> 
> Since when can anyone take a given delivery time of emails for granted?
> I can see people being stupid enough to do that, though.  The delay with
> graylisting remains a disadvantage.

For most SMTP mail systems that are properly configured, successful
delivery occurs within a few seconds to a couple of minutes, depending
on source and destination geographic location and the current load on
each system.  The misconfigured systems, including those with a poor
greylisting implementation or other poorly implemented anti-spam
countermeasures, are the ones that inject significant delay.

For example, on my personal MX thus far today:

=== Delivery Delays Percentiles ===================================
  0%      25%      50%      75%      90%      95%      98%     100%
0.72     1.92     2.40     3.00     5.40     7.15     9.14    14.00
===================================================================

we have 50% of all in/outbound messages delivered in less than 2.5
seconds and all messages delivered in 14 seconds or less.  This is a
well configured MTA.  Keep in mind it does have significant anti-spam
features, most of them custom.  It does not make use of a content filter
such as SpamAssassin, or any policy daemons, however, which helps keep
delays relatively low.  It does use a custom header_checks TCP server
which does add a second or so of additional delay as it queries 3 RHSBL
servers.  And this is on 11 year old hardware.

>> The fqrdns.pcre table gives most of the "catch" performance of
>> greylisting without the downsides.
> 
> I can see why you like it.  How do you make sure that mail you want to
> receive isn't rejected when using the table?

Because no one should be receiving email directly from residential PCs,
most which have dynamic IP addresses, some static addresses.  They
should never be sending email directly to MX hosts.  Only bot infected
PCs do that.  This table targets residential type rDNS strings, which
identify the PC as being residential, or less commonly, SOHO.  In either
case, they should be relaying email through their ISP's mail relay,
which we state in the reject messages in the table.

-- 
Stan