[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Too many sockets in SYN_RECV

Hello Debian-isp,

On almost idle server (12 GB RAM, 2x 6-core xeon, utilized to about
15-20%) I am running nginx serving one small php file over php5-fpm.
It is ad tracking site, so it has many connections per second from all
over the world.

During the peak hours (about 1500 conns/s) the response time goes high
- I want to get the nginx status page and I often get connection
timeout. I guess it is because there are too many connections waiting
in SYN_RECV state (it tops at 512 - is it possible to increase it).
But the server is almost idle, stracing nginx shows that it is
normally processing incoming connections. I tried increasing the
number of nginx worker processes, but it didn't help.
As I have learned, connection is ready for accept() by nginx only
after it is ESTABLISHED, so these SYN_RECV must be waiting in the
kernel for some reason...

Debian squeeze, 2.6.32-5-amd64, php & nginx from dotdeb

bill3:/home:# munin-run netstat_tcpstates
TIME_WAIT.value 38369
CLOSE_WAIT.value 0
SYN_SENT.value 0
FIN_WAIT1.value 364
CLOSE.value 0
ESTABLISHED.value 2263
FIN_WAIT2.value 3895
SYN_RECV.value 463
CLOSING.value 14
LAST_ACK.value 53

bill3:/home:# ss -nlt
Recv-Q Send-Q             Local Address:Port               Peer Address:Port
0      511               xx.xx.xx.xx:80                            *:*

bill3:/home:# netstat -ts
    113348 active connections openings
    1840559456 passive connection openings
    2363047 failed connection attempts
    11536549 connection resets received
    2234 connections established
    2692177651 segments received
    2990428444 segments send out
    53602607 segments retransmited
    151889 bad segments received.
    261748080 resets sent
    2360438 resets received for embryonic SYN_RECV sockets
    3 packets pruned from receive queue because of socket buffer overrun
    643 ICMP packets dropped because they were out-of-window
    48 ICMP packets dropped because socket was locked
    914787009 TCP sockets finished time wait in fast timer
    164848867 TCP sockets finished time wait in slow timer
    86861297 passive connections rejected because of time stamp
    388715 packets rejects in established connections because of timestamp
    1112821217 delayed acks sent
    615599 delayed acks further delayed because of locked socket
    Quick ack mode was activated 12384186 times
    310610 times the listen queue of a socket overflowed
    310610 SYNs to LISTEN sockets dropped
    387813160 packets directly queued to recvmsg prequeue.
    5771 bytes directly in process context from backlog
    7711423 bytes directly received in process context from prequeue
    977281196 packet headers predicted
    311065 packets header predicted and directly queued to user
    739893500 acknowledgments not containing data payload received
    697503290 predicted acknowledgments
    1 times recovered from packet loss due to fast retransmit
    19583 times recovered from packet loss by selective acknowledgements
    2380 bad SACK blocks received
    Detected reordering 9 times using FACK
    189 congestion windows fully recovered without slow start
    2 congestion windows partially recovered using Hoe heuristic
    2688626 congestion windows recovered without slow start by DSACK
    5521689 congestion windows recovered without slow start after partial ack
    123 TCP data loss events
    TCPLostRetransmit: 2
    4272 timeouts after reno fast retransmit
    2405072 timeouts after SACK recovery
    124467 timeouts in loss state
    19632 fast retransmits
    38 forward retransmits
    2137269 retransmits in slow start
    33938539 other TCP timeouts
    9706 SACK retransmits failed
    202 packets collapsed in receive queue due to low socket buffer
    15999828 DSACKs sent for old packets
    13914 DSACKs sent for out of order packets
    11364774 DSACKs received
    77 DSACKs for out of order packets received
    5 connections reset due to unexpected SYN
    32975997 connections reset due to unexpected data
    120171 connections reset due to early user close
    2272680 connections aborted due to timeout
    TCPSACKDiscard: 40431
    TCPDSACKIgnoredOld: 2049268
    TCPDSACKIgnoredNoUndo: 1429462
    TCPSackShifted: 156
    TCPSackMerged: 315
    TCPSackShiftFallback: 6461253


  bYE, Marki

Reply to: