iptables ip_conntrack bugs? (was: persistant connections?)
Well, nobody seems to be intrested :(
The short story is:
The firewall box will suddenly refuse connections.
For people on low budgets, with a limited amount of RAM, the (only)?
option (should such a scenario suddenly become reality) is to reboot
the box. And that is seldom acceptable.
My analysis may be wrong, due to my lack of deep knowledge on the
subject, but here it is anyyway.
On Sat, 19 Jan 2002, Cristian Ionescu-Idbohrn wrote:
> Hi,
>
> I'm bugged by the fact that I see this connection (see below) on my
> firewall box. The windoze client that initiated it is switched off
> since a few days back.
>
> I searched the net trying to find som sort of explanation to the
> phenomenon, but what I found was more or less dead ends, sotr of :(
>
>
> # cat /proc/net/ip_conntrack | grep x.x.x.x
> tcp 6
> 182837
> ESTABLISHED
> src=x.x.x.x dst=z.z.z.z
> sport=1309 dport=443
> [UNREPLIED]
> src=z.z.z.z dst=y.y.y.y
> sport=443 dport=1309
> use=1
>
> x.x.x.x - windoze client ip address on the internal network
> y.y.y.y - firewall box, outside ip address
> z.z.z.z - some web server running Microsoft-IIS/5.0
>
>
> Has anyone seen this before?
> Does anyone know what's going on behind the curtains?
> Is there a way to kill these things?
> Is there anything I can do to prevent this kind of thing (aside from
> not using buggy windoze clients and not browsing to buggy web-servers
> and not using buggy software to surf the net)?
>
> I don't like rebooting the firewall box (which will cure the problem)
> every time anything like this occurs.
Anyhow, after reading recent postings on netfilter@lists.samba.org,
like this one:
||| Date: Fri, 4 Jan 2002 17:03:12 -0800
||| From: "Perser, Jerry" <jerry.perser@spirentcom.com>
||| To: "'netfilter@lists.samba.org'" <netfilter@lists.samba.org>
||| Subject: Clearing /proc/net/ip_conntrack
|||
||| Is there a way to clear the conntrack table?
|||
||| I am doing some performance measurement on iptables. The tool I am
||| using does not send a RST to close the connection. After each test
||| iteration, iptables shows all the connections ESTABLISHED. Current
||| aging timer would take 5 days to clear. I really don't want to reboot
||| linux after each test run.
or this one:
||| Date: Fri, 21 Dec 2001 11:19:13 -0500
||| From: Aldo S. Lagana <alagana@discmail.com>
||| To: netfilter@lists.samba.org
||| Subject: Try # 2 ip_conntrack - connections never seem to die...
|||
||| In ip_conntrack, I see many connections with GIANT timeout values for a
||| server that is SHUT OFF?!?
|||
||| Here are some:
||| tcp 6 94781 ESTABLISHED src=192.168.100.233 dst=X.X.X.X sport=4500
||| dport=25 src=63.76.212.67 dst=X.X.X.X sport=25 dport=4500 [ASSURED]
||| use=1
||| tcp 6 82785 ESTABLISHED src=192.168.100.233 dst=X.X.X.X sport=2480
||| dport=25 src=X.X.X.X dst=X.X.X.X sport=25 dport=2480 [ASSURED] use=1
And it smells like bugs in netfilter. If you afford lots of RAM,
you'll be hit later, rather than sooner. The result is that packets
are thrown away with a:
ip_conntrack: maximum limit of XXX entries exceeded
message (see):
http://netfilter.samba.org/documentation/FAQ/netfilter-faq-3.html#ss3.6)
when the max limit:
# cat /proc/sys/net/ipv4/ip_conntrack_max
4096
is reached. It looks like it eats up 350 bytes of non-swappable
memory/tracked connection.
One of the problems (see nr. 1 below) seems to lie in
(linux/net/ipv4/netfilter/ip_conntrack_proto_tcp.c):
50 static unsigned long tcp_timeouts[]
51 = { 30 MINS, /* TCP_CONNTRACK_NONE, */
52 5 DAYS, /* TCP_CONNTRACK_ESTABLISHED, */
53 2 MINS, /* TCP_CONNTRACK_SYN_SENT, */
54 60 SECS, /* TCP_CONNTRACK_SYN_RECV, */
55 2 MINS, /* TCP_CONNTRACK_FIN_WAIT, */
56 2 MINS, /* TCP_CONNTRACK_TIME_WAIT, */
57 10 SECS, /* TCP_CONNTRACK_CLOSE, */
58 60 SECS, /* TCP_CONNTRACK_CLOSE_WAIT, */
59 30 SECS, /* TCP_CONNTRACK_LAST_ACK, */
60 2 MINS, /* TCP_CONNTRACK_LISTEN, */
61 };
TCP_CONNTRACK_ESTABLISHED == 5 DAYS
Several people agrees on 5 hours should be more than plenty.
The problem will show it's ugly face when you do:
# cat /proc/net/ip_conntrack
and/or:
# grep conn /proc/slabinfo
ip_conntrack 9180 9290 368 929 929 1
And the winners ;-) are:
1. Same huge timeout (5 days) on these:
tcp 6 164529 ESTABLISHED src=192.168.4.1 dst=192.168.8.2
sport=1093 dport=136 [UNREPLIED] src=192.168.8.2 dst=192.168.4.1
sport=136 dport=1093 use=1
2. Huge CLOSE timeouts on tcp connections (more than 50 hours on the
one below), and they _should_ be <= 10 sec:
tcp 6 18302357 CLOSE src=202.52.x.x dst=202.52.x.x
sport=1026 dport=110 src=202.52.x.x dst=202.52.x.x
sport=110 dport=1026 [ASSURED] use=1
or 5 hours on this one:
tcp 6 1773924 CLOSE src=202.131.x.x dst=202.52.x.x
sport=1110 dport=80 [UNREPLIED] src=202.52.x.x dst=202.131.x.x
sport=80 dport=1110 use=1
These problems are ACKed by members of the netfilter team:
||| Date: Mon, 7 Jan 2002 14:17:30 +0100 (CET)
||| From: Jozsef Kadlecsik <kadlec@blackhole.kfki.hu>
||| To: Sneppe Filip <Filip.Sneppe@cronos.be>
||| Cc: netfilter@lists.samba.org, netfilter-devel@lists.samba.org
||| Subject: RE: strange ip_conntrack values! (fwd)
|||
||| On Mon, 7 Jan 2002, Sneppe Filip wrote:
|||
||| > I think the answer to your questions may be in a mail from Harald
||| > after the netfilter developer meeting in Enschede (search for
||| > "Summary of netfilter devel workshop" in the archives):
|||
||| No, he's right. There is a bug and I'm working on finding it.
||| As usual, it's not clear how one could trigger it.
|||
||| Regards,
||| Jozsef
So the question is: Is there anybody involved with solving this?
Cheers,
Cristian
Reply to: