Re: Challenge-response mail filters considered harmful (was Re: Look at these update from M$ Corporation.)
On Mon, Aug 04, 2003 at 12:00:11AM +0100, Karsten M. Self wrote:
> on Sun, Aug 03, 2003 at 08:37:49PM +0200, David Fokkema (dfokkema@ileos.nl) wrote:
> > On Sun, Aug 03, 2003 at 05:13:26AM +0100, Karsten M. Self wrote:
> > > Spam is a growing, heck, exploding problem. No doubt.
> > > Challenge-response (C-R) is a flawed tactic, for the following
> > > reasons.
> > >
> > > 0. Weak, and trivially abused, verification basis.
> > >
> > > The 'FROM:' header of email can be, and routinely is, spoofed.
> > > It offers no degree of authentication or evidence of identity.
> >
> > As filtering is a spam-reduction system, so is C-R. The chance of
> > receiving spam from a whitelisted address is, in the experience of tmda
> > users, very rare. And even if it happens, it only adds up to low
> > statistics.
>
> C-R uses the "From:" header (with implementation-specific variations) as
> a key. While a given key is going to have a relatively low likelihood
> of being cleared by a given user, there are keys which will have a high
> likelihood of being cleared. Off the top of my head, @microsoft.com,
> @aol.com, @ebay.com, @*.gov, and other major commercial, financial, and
> governmental institutions, would be likely to be cleared by a large
> number of users.
Please read my response: 'the chance of receiving spam from a
whitelisted address is, in the experience of tmda users, very rare.' To
check this statement, please read the faq at tmda.sourceforge.net. The
statement is the one made in this faq. Since we are discussing C-R as a
possible solution _at present_, I will not discuss what keys would
likely be cleared by a large number of users, as this is _not_ done by
tmda users.
> C-R moves you back to square one of the fact that SMTP can't provide
> authentication of email headers. At the very least, contextual analysis
> of headers (as Alan admits) is necessary. If you're already taking this
> step, heuristic and Bayesian methods are a low-overhead next step, which
> have proven to be highly effective and accurate.
I wholeheartedly agree with your first statement, however, it doesn't
appear to be a problem since we have filtering and C-R. I have no
references to back me up, but I _think_ tmda would cost a lot
less CPU cycles than my SA setup, so I doubt the low overhead.
> > The spammer is affected in the same way as it is affected by filtering:
> > it sees revenues going down, fast.
>
> No. The spammer sees delivery effectiveness diminished at the margin --
> that is, on a message-by-message basis. Which is precisely the same
> effect as _any_ content or context-based filtering achieves. C-R
> doesn't lose in this regard, it just doesn't win.
I never said that it would win. I said: 'the same way'.
> Tools such as
> Teergrube and RBL _do_ fundamentally shift the balance against the
> spammer.
Great, I'll check them out sometime.
> > Additionally, if a spammer wants to deal with challenges, it has to
> > not only use valid addresses, it has to use addresses which he can
> > read replies from. That way, it is usually easy to report to his ISP.
>
> Wrong. See response to 0. above.
Wrong. My statement: '_if_ a spammer wants to deal with challenges,
etc.' If a spammer doesn't want to deal with challenges, it has to use a
whitelisted address, which must be tailored to a group of users, such as
us at debian-user.
> > > 2. Privacy violation.
> > >
> > > A record of our correspondence is being maintained by a third
> > > party who has no business knowing of the transaction. Many
> > > people will refuse to respond to C-R requests for this reason.
> >
> > I don't see this point.
>
> Several of the C-R proponents are making a number of assumptions. Most
> of the _general_ discussion (that is, outside this mailing list) has
> concerned service-model enterprise models in which C-R is provided and
> hosted by a third-party, which is then aquiring a rather interesting
> database of communications patterns, which _must_ be maintained on a
> persistent basis. Not the sort of thing I'd like to have available to
> an arbitrary subpeona request.
Well, a lot of people are using pop boxes these days. Keeps a lot of
communications stored on the server. And who says intermediate mail
servers don't keep copies? The only problem with third-party C-R might
be that you _know_ that third-party keeps copies (at least for a while).
That's why I would prefer running C-R on my home server. I dislike
copies being held by third parties as much as you do. Well, I don't know
that. I just dislike it. I empty my pop boxes once per minute and don't
keep anything on the server.
> Even TMDA is server-side based. This places it beyond the immediate
> control of the typical home email user.
It is. However, serveral client-side implementations of C-R are
available (see faq @ tmda.sourceforge.net) but there are numerous
advantages to server-side implementations. Most notably, delay times for
the challenge.
> > > 3. Less effective at greater burden than reciever-side
> > > whitelisting.
>
> Note that what I'm discussing here is a system where the reciever
> determines, and accumulates, his or her own whitelist. This is the
> system I use, it is completely transparent to the sender. Essentially:
> I'm doing my own dirty work rather than outsourcing. The advantage is
> that this gives me a greater degree of control over the outcome. The
> incremental cost is very low, and the potential offset-loss (missed
> message) is very high.
Agreed. Although the 'very high' depends on the willingness of people to
answer challenges. TMDA can be configured to inform you of people who
haven't answered challenges, but this folder will also hold a lot of
spam which would have been filtered... It would defeat the purpose of
C-R considerably. However, no mail has to be lost.
> > > A C-R system is essentially an outsourced whitelist system. The
> > > difference between a C-R system and a self-maintained whitelist
> > > is that the latter is:
> > >
> > > - Maintained by the mail recipient, rather than a third party.
> > > - Is the responsibility of the mail recipient, rather than the
> > > sender.
> > > - Places the burden on the recipient to add new addresses to
> > > allow/deny lists.
> > >
> > > I might add that I myself use a mix of whitelisting and spam
> > > filtering (via SpamAssassin) to filter my own mail with a very
> > > high level of accuracy, in terms of true positives, true
> > > negatives, false positives, and false negatives.
> >
> > I don't see the third party.
>
> The third party is the C-R service provider, see response to 2. above.
Yes, thank you. However, the whitelist of C-R, depending on the
implementation, might be totally in control of the recipient, the way I
would like it to be.
> > > 4. High type II error (beta).
> > >
> > > Because of numerous issues in sender-compliance with C-R
> > > systems, C-R tends to a high false postive rate. This is known
> > > as type II error, in statistical tests, and is denoted by beta.
> > >
> > > The mechanics of C-R systems lead to a fairly high probability
> > > that users of such
> >
> > I don't know anything about this.
>
> You or Alan.
Well, thank you for putting me on his side. I'm more on your side than
on his. If I have given the impression that I'm not a reasonable person
either, I sincerely apologise and I would like you to tell me so. Maybe
I can change my behaviour or think things over, ;-)
> This is among the aspects of C-R systems which makes it completely
> unsuitable for any practical use, without safeguards such as periodic
> review of queued messages (a practice Alan specifically rejects) or
> secondary pass mechanisms.
A statement like 'completely unsuitable' is rather a strong one.
However, I do agree with the basic problem involved. However, this also
applies to filtering. Messages are known to generate a false positive.
On the other hand, for C-R, senders are known to not reply to
challenges. But, Alan is just a guy who thinks that if someone doesn't
repsond to a challenge, he isn't worth talking to. Mind you, I'm not of
that kind. I'm of the kind that thinks that _if_ people would not object
to facing challenges, C-R would be a very nice system. But, as
discussion here has showed, it is a very big if. Too big for me, I value
your opinions too much.
> A false positive (type II error) is ham mail that is incorrectly tagged
> or otherwise treated as spam. C-R as defined assumes that mail is spam
> until otherwise determined.
> By contrast, my own filtering system assumes a mail is of unknown nature
> until otherwise determined. As various filters (whitelists, blacklists,
> spamlists, Spamassassin, mailing list filing, etc.) are applied, the
> incoming message stream is categorized and filed. At the end of the
> stream, a message isn't either spam or ham, it's grey (unknown). These
> messages are then manually evaluated. I receive a handful of such
> messages per day, recruiters, some spam, and contacts from previously
> unknown correspondents or new addresses of known correspondents. It
> takes only a few seconds to deal with these. The bulk (40-80+
> messages/daily) of my spam is automatically transferred to a spam folder
> where a quick scan (10 seconds) is all that's necessary to ensure that
> something hasn't been misfiled.
Agreed, a nice system.
> Alan apparently defines as an identity spam / unwanted mail anything
> that arrives from someone who doesn't comply with a C-R procedure. This
> is not a pragmatic definition.
Agreed.
>
> For standard definitions and descriptions, see:
>
> http://mathworld.wolfram.com/TypeIIError.html
> http://www.acponline.org/journals/ecp/novdec01/primer_errors.htm
Thanks!
> > > 5. Potential denial of service.
> > >
> > > C-R systems can be used in a denial-of-service or "Joe" attack
> > > on an innocent third party. In fact, this is likely to start
> > > happening shortly as C-R becomes more widespread.
> > >
> > > How? Simply: Spammer spoofs a legitimate sending address (this
> > > is already commonplace). C-R systems then send out a challenge
> > > to this address. With only 1% penetration of C-R, the victim of
> > > the C-R/Spam attack is deluged with 100,000 challenge emails.
> > > This could likely lead to lawsuits or other legal challenges.
> >
> > This can also be done right now. I guess that people who've had their
> > mail addresses spoofed know this because they receive a lot of bounces,
> > angry mails, replies from stupid people, etc. And if a spammer really
> > wants to start a DoS attack, why using such an elaborate way if he can
> > spoof whatever address he'd like?
>
> It's not the intentional use of C-R by the spammer, it's the unintended
> consequence. Though spamming a user population known to have a high
> utilization of C-R (say, subscribers of ISPs or organization known to use C-R
> methods) could be intentional.
I agree with you. I only think that _without_ C-R, you still get a lot
of bounces, angry mails, etc. It's not funny both ways.
> > > 6. C-R - C-R deadlock
> > >
> > > This is almost funny.
> > >
> > > How do two C-R system users ever start talking to each other?
> > >
> > > - User A sends mail to user B. While user B's address is then
> > > known to A, user B's C-R server's mail is not.
> > >
> > > - User B's C-R system sends a challenge to A...
> > >
> > > - ...who intercepts the challenge with A's C-R system, which
> > > sends a challenge to user B's C-R system...
> > >
> > > No, I didn't think this one up myself, see Ed Felton's "A
> > > Challenging Response to Challenge-Response":
> > >
> > > http://www.freedom-to-tinker.com/archives/000389.html
> > >
> > > Bypassing this deadlock then opens an obvious loophole for
> > > spammers to exploit.
> >
> > This must be very, very old. Uh, no... it's not. This guy obviously
> > hasn't read through tmda.sourceforge.net. I'm sorry, tmda doesn't have
> > a loophole. Only automatic whitelisting and dated addresses which, in
> > the experience of tmda users, are not a risk. And even if they were a
> > slight risk, C-R is a spam_reduction_ system.
>
> This and other responses assume "well-designed" C-R systems. Current
> experience with vacation responders and spam-notification filters
> provide strong empirical evidence that a significant number of C-R
> systems will in fact _not_ get this right.
Yes, it does assume well-designed systems. For the sake of discussing
concepts, I would like to stick to well-designed systems.
> The TMDA FAQ item 1.9 lists fifteen similar systems. I'd appreciate
> your comments on the specifics of design, implementation, design
> integrity, and handled mail volume, of these systems. As well as the
> other C-R systems not listed in the TMDA FAQ.
I'm just a guy who's read up on C-R systems and likes the concept
(again, the big if) but am not a user and certainly not preaching its
wide-spread use. I like what I know of tmda's implementation with, for
the sake of people not having to face challenges if they only _reply_ to
your mail, dated addressing. I don't know the other fifteen but for the
fact that some are commercial and use M$ Outlook, which I distrust
heavily. But then again, my father installed a nice spam filter for this
same MUA and I was no longer able to send mails to him. It wasn't even
processed by the filter, it was held back. Full of bugs.
As for C-R systems not in the FAQ: there is a program called
Mail-Sentry-Program or something like that, which I don't like at all:
mail is sent to /dev/null, senders have to resend their mail and copy
paste a password, instead of just replying to a challenge. Also, the
challenge is not phrased very elegantly.
> > > 7. Potential integration into spam email harvest systems.
> > >
> > > One commonplace piece of advice for avoiding spam is to not
> > > respond to opt-out, aka email validation testing, requests.
> > >
> > > C-R spoofing on the part of spammers would simply hijack a
> > > presumption that C-R requests were valid to provide spammers
> > > with higher-quality mailing lists.
> >
> > TMDA _always_ includes the original e-mail, so the recipient of the
> > challenge can check if it really was him sending a mail.
>
> See above response. This assumes a well-designed system. My experience
> with C-R demonstrates the contrary. In any event, habituation and
> uncertainty on the part of the typical recipient of a challenge moot
> this point. See the current rash of identity theft / CC theft scams
> based on "updating your account information". C-R at best promotes bad
> personal identity protection practices.
I totally agree: badly-designed C-R systems should be purged, and people
should be educated.
> > > 8. Likly consequences.
> > >
> > > The C-R user is likely to find their own address added to
> > > blocklists from many users and/or mailing list adminstrators
> > > burned by malformed, or simply unwanted, C-R requests.
> >
> > If not set up correctly, this might happen.
>
> First, see response to 6., regarding "well-designed" systems.
>
> Second, this factor is entirely outside the bounds of the C-R system, it
> is a reflection of the independent response of individuals and
> organizations to receiving C-R challenges. C-R definitionally cannot
> accomodate this.
A good implementation and configuration (using dated addresses) will
lead to so very few challenges that this is unlikely. However, you are
totally right: it is outside the C-R system and depends on the feeling
of individuals toward C-R.
> Third, item 1.10 in the TMDA FAQ directly contradicts you: Spamassassin
> tags TMDA challenges as spam. Spamassassin filters are a concensus
> hueristic based on the aggregate spambase contributed by Spamassassin
> developers and users, modulo local filterns and Bayesian modifications.
> The concensus reality, then expressed by Spamassassin users is that C-R
> challenges are spam.
If I read it correctly, it says: marking e-mail sent _to_ tmda-tagged
addresses. So, IIUC, if _you_ use tmda _and_ SA, reconfigure SA to let
challenge _replies_ go through. So, it doesn't say that SA will block
challenges, since the test FROM_ADDRESS_ENDS_IN_NUM or whatever it is
called doesn't apply: From: dfokkema@ileos.nl, Reply-To:
dfokkema-14142341@ileos.nl. But, again, it is a risk if people massively
decide that challenges are spam.
> Beyond any semiotic arguments of what spam is or isn't, if the
> operational reality is that Spamassassin reflects the opinion of SA
> users and developers and treats C-R transactions as spam, then my
> statement 8. is validated.
Indeed.
> > > 9. Mailing list burden.
> > >
> > > C-R systems typically misfunction on mailing lists in one of
> > > two ways, neither of which is acceptable:
> > >
> > > 1. The C-R sends a challenge to the list for messages received.
> > >
> > > 2. The C-R sends a challenge to each individual listmember for
> > > the first post received.
> > >
> > > In both cases, the burden is placed on a party who could care
> > > less about the benefits of the C-R system. Several lists of my
> > > aquaintance have taken to permanently banning any users who
> > > exhibit use of misconfigured C-R systems.
> >
> > Those C-R systems are set up _very_ incorrectly.
>
> See response to 6. regarding "well-designed" systems.
Again, correct.
> > > 10. Fails to address techno-economic underpinnings of spam.
> > >
> > > Spam exists for one reason: it's profitable.
> > >
> > > It's profitable because technology allows the costs of sending
> > > a large number of mail messages to be lower than the revenues
> > > available for doing so.
> > >
> > > Any effective spam remedy must attack one or the other side (or
> > > both) of this equation: raise the costs or reduce the
> > > technological effectiveness, on the one side, or reduce
> > > revenues on the other.
> > >
> > > C-R, as with most recipient-side filtering systems, imposes
> > > negligible incremental overhead on the spammer. A delivery is
> > > made, the spam server moves on, the cost is a single SMTP
> > > connection for a fractional second. Collateral costs are high:
> > > for legitimate senders, spoofed reply addresses, mailing lists,
> > > and retaliatory actions on the C-R user.
> > >
> > > A truly effective spam defense must attack the techical and
> > > economic aspects, in as unobtrusive a manner as possible.
> > >
> > > The one system which seems to best fit this requirement is the
> > > Teergrub -- the spam tar-baby, FAQ at:
> > >
> > > http://www.iks-jena.de/mitarb/lutz/usenet/teergrube.en.html
> > >
> > > A teergrubing mailserver costs a spammer multiple SMTP
> > > connections, an inherently finite resource, for possibly hours.
> > > Workarounds on the part of the spammer are possible, but all
> > > result in higher costs, reduced delivery, or both. The net
> > > effect is essentially a delivery payment requirement, though
> > > the payment is in the form of time and configuration on the
> > > part of the spammer. Collateral damage is low -- if a
> > > teergrube _does_ unintentionally filter a legitimate sender,
> > > the only cost is a single (or very small number of) delayed
> > > delivery. This and other issues are covered at the FAQ above,
> > > read it before posing hypothetical problems.
> >
> > You mention that workarounds are possible, but result in higher costs.
> > The same is true for TMDA: by requiring spammers to set up a C-R
> > auto-reply system, they make their mails traceable to their origin. I
> > don't think spammers can spoof a valid mail address _and_ capture the
> > replies.
>
> Incorrect. See response to 0.
Response to 0 states that they have to capture valid whitelisted
addresses for other spammed addresses. Also more expensive than blindly
spoofing one address and sending a spam mailing.
> > To read a nice pro C-R page, read tmda.sourceforge.net. Check their FAQ,
> > for example, or read the introduction. In particular, the part about
> > 'Won't senders just refuse to confirm their messages?' is nice, if you
> > read through to the part about using 'dated' addresses.
>
> First, the apparent outcome here is distinctly different from the
> experience reported in the FAQ.
Indeed.
> Second, given that both you and Alan here fail to even understand what
> Type II errors are, let alone contenance their existence in the face of
> C-R, I find the evidence provided by you to be less than convincing.
I'm sorry to hear that. I think this discussion has already passed the
point of emotional involvement.
I think I _do_ understand what Type II errors are. I did only know them
as 'false positives'. I accept their existence, especially in the face
of C-R. C-R can't be a viable solution unless the people you want to
accept mail from don't oppose challenges. As long as they do, C-R is not
to be used.
Furthermore, I did and do not provide evidence. I only provide
reasonings and pointers to tmda, where (I gather) people share their
experiences. But, just maybe, they could all be like Alan and really
don't _care_ if people don't reply challenges. However, in the FAQ
there's a statement about this which I'm sure you've read.
> Third, in edge cases (mass distribution, mailing lists, malformed
> messages, poorly designed systems, spammer abuse of C-R concepts), this
> behavior is very likely to increase, rather than decrease. FAQ item 1.5
> specifically assumes direct personal communications, which is in many
> cases invalid.
The FAQ states that you _should not_ use C-R with mailing lists to avoid
angering people. Mass distribution from unknown source which is not spam
is ... unsolicited commercial e-mail?
Malformed messages, poorly designed systems and spammer abuse hit us
all. For example, I believe you mentioned that you also use
whitelisting. A spammer can both defeat your whitelist and a C-R
whitelist.
> Fourth, the issues addressed in the TMDA FAQ are specific to TMDA and
> cannot be generalized to other systems. Specifically, several comments
> of Alan regarding his system design directly contradict statements of
> the TMDA FAQ, particularly regarding message loss.
You are very right. I myself think that Alan's system should not be held
as a fair example of a well-designed C-R system.
Filtering, as a generalized system, might or might not be effective. It
is very much dependent on the methods used. But, Bayes filtering seems
to be very effective and is what I use.
> Peace.
Thank you. Peace to you too.
David
PS: If I have given other people the impression that I'm not reasonable
and that this discussion has turned into a did, didn't argument in which
I had part, I'm very sorry.
PPS: Sorry to clobber up bandwidth, again... But I though this actually
_was_ a discussion.
Reply to: