Too many sockets in SYN_RECV
Hello Debian-isp,
On almost idle server (12 GB RAM, 2x 6-core xeon, utilized to about
15-20%) I am running nginx serving one small php file over php5-fpm.
It is ad tracking site, so it has many connections per second from all
over the world.
During the peak hours (about 1500 conns/s) the response time goes high
- I want to get the nginx status page and I often get connection
timeout. I guess it is because there are too many connections waiting
in SYN_RECV state (it tops at 512 - is it possible to increase it).
But the server is almost idle, stracing nginx shows that it is
normally processing incoming connections. I tried increasing the
number of nginx worker processes, but it didn't help.
As I have learned, connection is ready for accept() by nginx only
after it is ESTABLISHED, so these SYN_RECV must be waiting in the
kernel for some reason...
Debian squeeze, 2.6.32-5-amd64, php & nginx from dotdeb
bill3:/home:# munin-run netstat_tcpstates
TIME_WAIT.value 38369
CLOSE_WAIT.value 0
SYN_SENT.value 0
FIN_WAIT1.value 364
CLOSE.value 0
ESTABLISHED.value 2263
FIN_WAIT2.value 3895
SYN_RECV.value 463
CLOSING.value 14
LAST_ACK.value 53
bill3:/home:# ss -nlt
Recv-Q Send-Q Local Address:Port Peer Address:Port
0 511 xx.xx.xx.xx:80 *:*
bill3:/home:# netstat -ts
Tcp:
113348 active connections openings
1840559456 passive connection openings
2363047 failed connection attempts
11536549 connection resets received
2234 connections established
2692177651 segments received
2990428444 segments send out
53602607 segments retransmited
151889 bad segments received.
261748080 resets sent
TcpExt:
2360438 resets received for embryonic SYN_RECV sockets
3 packets pruned from receive queue because of socket buffer overrun
643 ICMP packets dropped because they were out-of-window
48 ICMP packets dropped because socket was locked
914787009 TCP sockets finished time wait in fast timer
164848867 TCP sockets finished time wait in slow timer
86861297 passive connections rejected because of time stamp
388715 packets rejects in established connections because of timestamp
1112821217 delayed acks sent
615599 delayed acks further delayed because of locked socket
Quick ack mode was activated 12384186 times
310610 times the listen queue of a socket overflowed
310610 SYNs to LISTEN sockets dropped
387813160 packets directly queued to recvmsg prequeue.
5771 bytes directly in process context from backlog
7711423 bytes directly received in process context from prequeue
977281196 packet headers predicted
311065 packets header predicted and directly queued to user
739893500 acknowledgments not containing data payload received
697503290 predicted acknowledgments
1 times recovered from packet loss due to fast retransmit
19583 times recovered from packet loss by selective acknowledgements
2380 bad SACK blocks received
Detected reordering 9 times using FACK
189 congestion windows fully recovered without slow start
2 congestion windows partially recovered using Hoe heuristic
2688626 congestion windows recovered without slow start by DSACK
5521689 congestion windows recovered without slow start after partial ack
123 TCP data loss events
TCPLostRetransmit: 2
4272 timeouts after reno fast retransmit
2405072 timeouts after SACK recovery
124467 timeouts in loss state
19632 fast retransmits
38 forward retransmits
2137269 retransmits in slow start
33938539 other TCP timeouts
9706 SACK retransmits failed
202 packets collapsed in receive queue due to low socket buffer
15999828 DSACKs sent for old packets
13914 DSACKs sent for out of order packets
11364774 DSACKs received
77 DSACKs for out of order packets received
5 connections reset due to unexpected SYN
32975997 connections reset due to unexpected data
120171 connections reset due to early user close
2272680 connections aborted due to timeout
TCPSACKDiscard: 40431
TCPDSACKIgnoredOld: 2049268
TCPDSACKIgnoredNoUndo: 1429462
TCPSackShifted: 156
TCPSackMerged: 315
TCPSackShiftFallback: 6461253
--
bYE, Marki
Reply to: