Weird routing / arp / ppp problem - low upload after debian upgrade [long]
Hi,
After upgrade from old patched etch, my clients cannot browse internet anymore (upload is ok but download not bigger than few kbps ) - problem occurs randomly - other services that use small packets like voip work perfectly.
Here's my detailed problem.
I have pppoe concentrator serving several hundreds of computers . Every user can have public IP (directly on the pppoe tunnel without snat/dnat) or snated private IP. Those clients with public IP are proxy_arp'ed so world can see them. Incoming traffic goes on imq0 and outgoing on eth0 - traffic shaping looks fine
This is typical example of firewall rule generated for public IP :
iptables :
iptables -t filter -A FORWARD -i ppp+ -s 217.17.10.250 -j ACCEPT
iptables -t filter -A FORWARD -d 217.17.10.250 -j ACCEPT
shaping :
iptables -t mangle -A UPLOAD -p all -o eth0 -s 217.17.10.250 -j CLASSIFY --set-class 2:246
tc filter add dev imq0 parent 1: protocol ip u32 match ip dst 217.17.38.250 flowid 1:246
tc class add dev imq0 parent 1:2 classid 1:246 htb rate 128kbit ceil 4096kbit burst 4096kbit prio 5 quantum 8
tc qdisc add dev imq0 parent 1:246 handle 246:0 sfq perturb 10
tc class add dev eth0 parent 2:2 classid 2:246 htb rate 128kbit ceil 4096kbit burst 4096kbit prio 5 quantum 8
tc qdisc add dev eth0 parent 2:246 handle 246:0 sfq perturb 10
for private IP we have almost the same but there's SNAT in the iptables part. Every client has the same formula for generating iptables firewall.
My problem is following - totally random clients are having problems with download. If I use mikrotik bandwidth tester from internet to their computer it gives transfer like Xmbits upload (from their side) and 10-15kbps in direction to the client. Problem ONLY occurs when they are behind their client router. If they connect via pppoe directly to my server - problem disappeares.
Bandwidth tester uses big packets so they are fragmented. If I use packets like ping - they have nice transfer and everything is reachable from them. The problem on the side of client looks like they can browse internet but google.com loads for like 20 minutes but voip works fine. Moreover if i take client router and place it in the other place of my "lan" (my lan is 100% bridged with mikrotik) , it usually works.
What i've triple checked :
- generation of iptables/tc rules
- pppoe MTU (1480 or 1492 - both working )
- mss - path mtu discovery packets are not blocked, everything looks fine
What i suspect :
- some arp problem maybe ?
Problem began right after i've changed my old Etch server on 2.6.15 witch patched iptables and kernel with patch-o-matic into clean 2.6.32 squeeze with everything from apt. My sysctl.conf along with pppoe-server-options is attached at the end of this message.
I've done also tcpdump sniff on the clients interface many times and nothing drags my attention.
Typical arp entry for snated IP is :
? (10.100.0.25) at <incomplete> on eth1
? (10.100.0.25) at <from_interface> PERM PUB on eth1
Typical arp entry for public IP looks like :
? (217.17.10.250) at <from_interface> PERM PUB on eth0
Any clues will be VERY appreciated.
debian-firewall , please Cc to me as I'm not subscribed please.
Regards
WZ
------------sysctl.conf---------------
kernel.panic = 3
net.core.rmem_max = 131071
net.core.wmem_max = 131071
net.ipv4.conf.all.arp_announce = 0
net.ipv4.conf.all.arp_ignore = 0
net.ipv4.conf.all.proxy_arp = 1
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.arp_announce = 0
net.ipv4.conf.default.arp_filter = 0
net.ipv4.conf.default.arp_ignore = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.eth0.arp_announce = 0
net.ipv4.conf.eth0.arp_ignore = 0
net.ipv4.conf.eth0.proxy_arp = 1
net.ipv4.conf.eth1.arp_announce = 0
net.ipv4.conf.eth1.arp_ignore = 0
net.ipv4.conf.eth1.proxy_arp = 0
net.ipv4.conf.eth2.proxy_arp = 0
net.ipv4.ip_forward = 1
net.ipv4.ip_local_port_range = 1024 4999
net.ipv4.neigh.default.base_reachable_time = 1036800
net.ipv4.neigh.default.gc_thresh1 = 1024
net.ipv4.neigh.default.gc_thresh2 = 8192
net.ipv4.neigh.default.gc_thresh3 = 32768
net.ipv4.neigh.default.ucast_solicit = 4
net.ipv4.neigh.eth0.base_reachable_time = 1036800
net.ipv4.neigh.eth0.ucast_solicit = 4
net.ipv4.neigh.eth1.ucast_solicit = 4
net.ipv4.neigh.eth2.ucast_solicit = 4
net.ipv4.neigh.imq0.ucast_solicit = 4
net.ipv4.neigh.imq1.ucast_solicit = 4
net.ipv4.neigh.lo.ucast_solicit = 4
net.ipv4.netfilter.ip_conntrack_max = 132760
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close_wait = 10
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close = 5
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 43200
net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait = 30
net.ipv4.netfilter.ip_conntrack_tcp_timeout_last_ack = 30
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv = 60
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 20
net.ipv4.tcp_dsack = 0
net.ipv4.tcp_ecn = 0
net.ipv4.tcp_fack = 1
net.ipv4.tcp_mem = 393216 524288 786432
net.ipv4.tcp_rmem = 4096 87380 174760
net.ipv4.tcp_sack = 0
net.ipv4.tcp_syncookies = 0
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_wmem = 4096 16384 131072
-----------------------------------tcpdump of client -------------(myptr is the client's public IP)------------
23:11:45.110682 IP ew-in-f104.1e100.net.www > 190.myptr.com.33773: Flags [.], seq 966809112:966810542, ack 3627825205, win 122, length 1430
23:11:49.509890 IP 190.myptr.com.32876 > sip.voice.gtsenergis.pl.sip: SIP, length: 369
23:11:49.512774 IP sip.voice.gtsenergis.pl.sip > 190.myptr.com.32876: SIP, length: 366
23:11:53.811644 IP 190.myptr.com.49186 > 94.245.115.184.3544: UDP, length 61
23:11:53.853959 IP 94.245.115.184.3544 > 190.myptr.com.49186: UDP, length 109
23:11:55.110798 IP ew-in-f104.1e100.net.www > 190.myptr.com.33773: Flags [.], seq 0:1430, ack 1, win 122, length 1430
23:11:56.383790 IP 151.59.26.182.46119 > 190.myptr.com.33260: Flags [F.], seq 3286477939, ack 3609663927, win 65364, length 0
23:11:56.622196 IP 158.129.20.136.35137 > 190.myptr.com.32923: Flags [F.], seq 840311840, ack 4022529708, win 65373, length 0
23:12:00.277498 IP 190.myptr.com.isakmp > ip-89.171.11.42.static.crowley.pl.isakmp: isakmp: phase 1 I ident
23:12:04.529779 IP 190.myptr.com.32876 > sip.voice.gtsenergis.pl.sip: SIP, length: 369
23:12:04.532416 IP sip.voice.gtsenergis.pl.sip > 190.myptr.com.32876: SIP, length: 366
23:12:05.111138 IP ew-in-f104.1e100.net.www > 190.myptr.com.33773: Flags [.], seq 0:1430, ack 1, win 122, length 1430
23:12:05.659030 IP 130pc240.sshunet.nl.https > 190.myptr.com.32786: Flags [R.], seq 3389053229, ack 421162466, win 0, length 0
23:12:09.608646 IP 190.myptr.com.isakmp > ip-89.171.11.42.static.crowley.pl.isakmp: isakmp: phase 1 I ident
23:12:10.174398 IP 190.myptr.com.33580 > 10.10.123.30.www: Flags [S], seq 3710035758, win 8192, options [mss 1452,nop,wscale 8,nop,nop,sackOK], length 0
23:12:13.187047 IP 190.myptr.com.33580 > 10.10.123.30.www: Flags [S], seq 3710035758, win 8192, options [mss 1452,nop,wscale 8,nop,nop,sackOK], length 0
23:12:15.111714 IP ew-in-f104.1e100.net.www > 190.myptr.com.33773: Flags [.], seq 0:1430, ack 1, win 122, length 1430
23:12:19.548671 IP 190.myptr.com.32876 > sip.voice.gtsenergis.pl.sip: SIP, length: 369
23:12:19.551750 IP sip.voice.gtsenergis.pl.sip > 190.myptr.com.32876: SIP, length: 366
23:12:23.548358 IP 190.myptr.com.33568 > 10.10.123.30.www: Flags [S], seq 2757323080, win 8192, options [mss 1452,nop,wscale 8,nop,nop,sackOK], length 0
23:12:25.112024 IP ew-in-f104.1e100.net.www > 190.myptr.com.33773: Flags [.], seq 0:1430, ack 1, win 122, length 1430
23:12:26.548632 IP 190.myptr.com.33568 > 10.10.123.30.www: Flags [S], seq 2757323080, win 8192, options [mss 1452,nop,wscale 8,nop,nop,sackOK], length 0
23:12:32.191530 IP 139.91.70.35.19609 > 190.myptr.com.33196: Flags [F.], seq 604234015, ack 3123515410, win 17520, length 0
-----------------pppoe server options-----------------------
plugin radius.so
plugin radattr.so
auth
require-chap
lcp-echo-interval 10
lcp-echo-failure 5
ms-dns 217.17.10.208
ms-dns 217.17.10.10
proxyarp
noipx
mtu 1460
mru 1460
--
Wojciech Ziniewicz
http://www.rfc-editor.org/rfc/rfc2324.txt
Reply to: