Hi,
Netfilters clamp-mss-to-pmtu (as used in "iptables -A FORWARD -p tcp --tcp-flags
SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu") sets MSS in packets that had no MSS set
before.
So instead of sending packets of standard TCP MSS 536 Bytes to a host that didn't set
a MSS at all, packets with a potentially higher MSS (like 1452 on my DSL connection)
will be sent to that host. That might fail if the host only accepts packets with a MSS
of 536 (like http://research.microsoft.com). If that host also doesn't send a "ICMP
fragmentation needed" packet (like research.microsoft.com ...) the connection will fail.
Clamp-mss-to-pmtu really shouldn't set a MSS if none was set before.
RFC 879 says "HOSTS MUST NOT SEND DATAGRAMS LARGER THAN 576 OCTETS UNLESS THEY
HAVE SPECIFIC KNOWLEDGE THAT THE DESTINATION HOST IS PREPARED TO ACCEPT LARGER
DATAGRAMS." This means that the standard MTU is 576 and the standard MSS 536 and any
server not setting a MSS can expect to only receive packages with a MSS of 536 bytes.
If the clamping sets a MSS > 536 connections might fail.
I stumbled upon this problem in debian bug #541658[1] ("[iceweasel] cannot open
research.microsoft.com" - only worth reading for entertainment purposes) and, after that
bug was closed, analysed it in my blog[2] until a friend of mine found out why the page
loads when clamping mss to pmtu is disabled or restricted to a range (like with
"iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -m tcpmss --mss 1400:1536 -j TCPMSS
--clamp-mss-to-pmtu") but doesn't load with "simple" clamping. His really great and
detailed analysation of the problem may be seen at [3].
He also looked into the kernel code (net/netfilter/xt_TCPMSS.c) and found the
kind of contradicting comments "Never increase MSS, even when setting it, as doing so
results in problems for hosts that rely on MSS being set correctly." (line 93) and
"MSS Option not found ?! add it.." (line 116).
So instead of just leaving a packet without MSS option untouched a new MSS, that might
be much greater than the default 536, is set, even though the guy who wrote that seemed
to be aware that MSS may not be increased.
This bug most probably affects all kernel version from at least 2.6.26 (probably also
much older versions) up to now. It seems like also some hardware-routers that probably
use Linux are affected.