[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Delayed mails on debian-user [Now mostly fixed]



'lo,

I got some emails asking me what teergrubing was after this post which I
managed to delete before replying to. Hopefully the people who were
wondering about it will catch this message.

On Thu, Aug 19, 2004 at 03:40:34AM +1000, Pascal Hakim wrote:
> Hi,
> <snip>
> 	Delays were caused on this list by the high number of gmail
> subscribers, which caused some gmail servers to start teergrubing
> lists.debian.org.
> <snip>

Firstly, that sentence above is incorrect. It should have be: "Delays on
this list were caused by the high number of gmail subscribers on this
list, combined with the fairly high volume of mails that comes out of
it".

Basically, what's currently happening is that mails getting sent to
gmail.com are taking a _long_ time to get processed. While we get an
answer almost straight away on HELO, RCPT TO:, or Mail From, we have to
wait until we get a reply at the end of the DATA section of the email.
I've done some tests, and it seems to be taking at least 5 minutes to
give us an OK code after the message has been sent.

While this is not a problem if you are only sending a couple of emails,
when you get to the levels that lists.debian.org has to deal with, it
starts being a problem. We've setup our MTA so that it can be trying to
deliver up 270 messages at any point in time. By holding a connection
open for a long period of time, processes that would normaly be
delivering mail just sit there waiting for a successfull delivery
message. Since there weren't enough processes to deliver messages, the
queues were getting longer and longer causing some people to have
extreme delays in getting emails from lists. The listmaster team stuck a
band-aid on the problem by having some processes set aside to be used
uniquely by gmail. That way the problem now only affects people with an
address at gmail and no others.

If we look at some quick numbers, we realise quite quickly that we're
still having quite a problem. In the last 90 minutes, 350 messages were
accepted by gmail. Since there are 50 processes dedicated to delivering
mail there, it's taking on average 12 minutes for a message to an
individual gmail subscriber to go through.

We still don't know what's causing the delays when we send mail to
gmail. There's two main possibilities I can think of. (a) is that gmail
is doing some content filtering after mail reception but before
accepting a message. This way it can bounce messages it believes to be
spam. (b) is that google is holding connections open to what it thinks
are propbably spammers. This is what teergrubing refers to. The idea
behind it is that most spammers do not use real SMTP clients, but
rather, something that mostly works. By holding a connection open for a
long time, a spammer is prevented from sending more spam out, as it has
one less process and one less port it can send spam from. As we can see
from what's happening to us, this is a very effective technique in
stopping someone from delivering mail.

There's a fairly interesting article about teergrubing at:
http://www.iks-jena.de/mitarb/lutz/usenet/teergrube.en.html

At the moment, I'm guessing that (b) happened. I'm still trying to get
more information, so it's hard for me to make more than a guess at this
point however.

	Cheers,

Pasc



Reply to: