Re: Network issue
Hi.
Please do not top post. This is a mailing list, not a corporate e-mail
spamfest.
On Wed, Aug 29, 2018 at 01:18:52PM -0500, Nicholas Geovanis wrote:
> Hi, I've never used OVH. How certain are you that the e1000 network driver
> is the correct one?
> Under VMWare/ESX the network driver choice can be crucial, for example.
1) You cannot run Xen in VSphere/ESXi.
2) No sane public provider will use VSphere/ESXi for hosting, the costs
can dim a budget of a small country.
3) e1000e may be bad, but vmxnet3 will make oom-killer an everyday
reality. SR-IOV is a way to go.
Now, to the issue at hand.
> On Wed, Aug 29, 2018 at 6:30 AM Kevin DAGNEAUX <kevin.dagneaux@fiitelcom.fr>
> wrote:
Please note that you seem to have "link down" first:
> > Aug 28 15:50:32 ovh-1 kernel: e1000e: enp1s0 NIC Link is Down
and 'malfunctioning' netfilter rules next.
> > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
> > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=YY.YY.YY.YY
> > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2229 DF PROTO=TCP
> > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
Observe 'RST' flag at each and every 'DROPPED' message.
I find it highly unlikely that whatever PHP load test you did it
involved sending mass amounts of TCP Reset packets.
It seems that the following scenario has much higher probability:
1) You fired your load generator application.
2) Your hosting provided immediately got a signal of your typical DOS
attack (not to be confused with DDOS) coming from that seems to be a
typical mobile phone.
3) DDOS protection kicked in:
a) Isolating your server from the network to stop DOS.
b) Sending forged TCP RST to your server to break existing connections
*and* termninate unneeded Apache workers (or whatever you have there).
c) Banning the initiator of DOS (i.e. you on mobile network) temporarily.
4) Real network outage of your server was 2 seconds (time between "link
down" and "link up").
Reco
>
>
> > Hi,
> >
> > I've a server in OVH datacenter, on this server i've 7 VMs, on 1 of them
> > in run Apache.
> > To debug a slow upload (who was ~2Mo/s instead 12Mo/s) i've installed an
> > HTML5/PHP speed test application.
> > When i use this app, i've no problem in general, but, when a make a speed
> > test from a source who have more bandwith than the server (the server is
> > limited at 100Mb/s by OVH and i make the test from a 4G+ network where i've
> > ~150Mb/s of bandwith), in this case, the DOM0 lost his network connection
> > (like the ethernet cable is unplugged) until i reboot the server.
> >
> > When i check the syslog of DOM0, i see that iptables drop incomming packet
> > on port 80 instead of routing them to the VM.
> >
> > This is my iptables script i use on DOM0 :
> >
> > #!/bin/bash
> >
> > IPT="/sbin/iptables"
> >
> >
> > ###########################################################################################
> > # Filter
> >
> > ## Remise par defaut des regles
> > $IPT -t filter -P INPUT ACCEPT
> > $IPT -t filter -P FORWARD ACCEPT
> > $IPT -t filter -P OUTPUT ACCEPT
> >
> > ## On purge les tables
> > $IPT -t filter -F
> >
> > ## On autorise lo
> > $IPT -t filter -A INPUT -i lo -j ACCEPT
> >
> > ## On ouvre les ports nécéssaires au DOM0
> > $IPT -t filter -A INPUT -m tcp -p tcp --dport 22 -j
> > ACCEPT ## SSH
> > $IPT -t filter -A INPUT -m udp -p udp --dport 53 -j
> > ACCEPT ## DNS
> > $IPT -t filter -A INPUT -m icmp -p icmp --icmp-type 8 -j
> > ACCEPT ## Ping
> > $IPT -t filter -A INPUT -s 10.0.0.0/24 -j ACCEPT
> >
> > ## On accepte si la connexion est déjà établie
> > $IPT -t filter -A INPUT -m conntrack --ctstate ESTABLISHED -j ACCEPT
> >
> > ## On log ce qui n'a pas été matché par les règles précédente
> > $IPT -A INPUT -p tcp -j LOG --log-prefix "DROPED packets "
> >
> > ## On bloque tout le reste
> > $IPT -t filter -P INPUT DROP
> >
> >
> > ############################################################################################
> > # Nat
> >
> > ## Remise par defaut des regles
> > $IPT -t nat -P PREROUTING ACCEPT
> > $IPT -t nat -P POSTROUTING ACCEPT
> > $IPT -t nat -P INPUT ACCEPT
> > $IPT -t nat -P OUTPUT ACCEPT
> >
> > ## On purge
> > $IPT -t nat -F
> >
> > ### Routage des ports entrants pour la VM "mails"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22030 -j DNAT --to
> > 10.0.0.30:22 ## SSH
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 25 -j DNAT --to
> > 10.0.0.30:25 ## SMTP
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 587 -j DNAT --to
> > 10.0.0.30:587 ## SMTP SUBMISSION
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 465 -j DNAT --to
> > 10.0.0.30:465 ## SMTP SSL
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 143 -j DNAT --to
> > 10.0.0.30:143 ## IMAP
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 993 -j DNAT --to
> > 10.0.0.30:993 ## IMAP SSL
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 9930 -j DNAT --to
> > 10.0.0.30:9930 ## IMAP SSL
> >
> > ### Routage des ports entrants pour la VM "sql"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22020 -j DNAT --to
> > 10.0.0.20:22 ## SSH
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 3306 -j DNAT --to
> > 10.0.0.20:3306 ## MariaDB
> >
> > ### Routage des ports entrants pour la VM "files"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22010 -j DNAT --to
> > 10.0.0.10:22 ## SSH
> >
> > ### Routage des ports entrant pour la VM "web"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22040 -j DNAT --to
> > 10.0.0.40:22 ## SSH
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 80 -j DNAT --to
> > 10.0.0.40:80 ## HTTP
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 443 -j DNAT --to
> > 10.0.0.40:443 ## HTTPS
> >
> > ### Routage des ports entrants pour la VM "monitor"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22050 -j DNAT --to
> > 10.0.0.50:22 ## SSH
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 850 -j DNAT --to
> > 10.0.0.50:80 ## HTTP
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 855 -j DNAT --to
> > 10.0.0.50:443 ## HTTPS
> >
> > ### Routage des ports entrants pour la VM "comm"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22060 -j DNAT --to
> > 10.0.0.60:22 ## SSH
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 5222 -j DNAT --to
> > 10.0.0.60:5222 ## Jabber
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 5269 -j DNAT --to
> > 10.0.0.60:5269 ## Jabber
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 5280 -j DNAT --to
> > 10.0.0.60:5280 ## Jabber
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 5281 -j DNAT --to
> > 10.0.0.60:5281 ## Jabber
> >
> > ### Routage des ports entrants pour la VM "secure"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22070 -j DNAT --to
> > 10.0.0.70:22 ## SSH
> >
> > ### Routage des ports entrants pour la VM "net"
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 22080 -j DNAT --to
> > 10.0.0.80:22 ## SSH
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 8388 -j DNAT --to
> > 10.0.0.80:8388 ## shadowsocks
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p udp --dport 8388 -j DNAT --to
> > 10.0.0.80:8388 ## shadowsocks
> > $IPT -t nat -A PREROUTING -d XX.XX.XX.XX -p tcp --dport 11094 -j DNAT --to
> > 10.0.0.80:1194 ## OpenVPN
> >
> > ### Autorise les VMs a accéder a internet
> > $IPT -t nat -A POSTROUTING -s 10.0.0.0/24 -j MASQUERADE
> >
> > And this is what i have in syslogs :
> >
> > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
> > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=YY.YY.YY.YY
> > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2226 DF PROTO=TCP
> > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
> > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
> > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=YY.YY.YY.YY
> > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2227 DF PROTO=TCP
> > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
> > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
> > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=YY.YY.YY.YY
> > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2228 DF PROTO=TCP
> > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
> > Aug 28 15:50:32 ovh-1 kernel: DROPED packets IN=enp1s0 OUT=
> > MAC=ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ:ZZ SRC=YY.YY.YY.YY
> > DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=55 ID=2229 DF PROTO=TCP
> > SPT=9610 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
> >
> > *** I've several hundred of similare line where only the ID change, then :
> > ***
> >
> > Aug 28 15:50:32 ovh-1 kernel: e1000e: enp1s0 NIC Link is Down
> > Aug 28 15:50:32 ovh-1 systemd-networkd[20998]: enp1s0: Lost carrier
> > Aug 28 15:50:34 ovh-1 systemd-networkd[20998]: enp1s0: Gained carrier
> > Aug 28 15:50:34 ovh-1 kernel: e1000e: enp1s0 NIC Link is Up 100 Mbps Full
> > Duplex, Flow Control: None
> > Aug 28 15:50:34 ovh-1 kernel: e1000e 0000:01:00.0 enp1s0: 10/100 speed:
> > disabling TSO
> >
> > The log show that the network is UP but the server is offline.
> > In the VM's logs i don't have abnormal things.
> >
> > OVH have changed the motherboard and the ethernet cable but the problem
> > still persist, the syslog show that it's probably a software problem.
> >
> > Do you have idea of what happen or a way to help me to debug this problem?
> >
> > Thank you in advance.
> >
> > Kevin
> > --
> >
> > DAGNEAUX Kevin
> > Service informatique
> > 03 29 36 88 85
> > kevin.dagneaux@fiitelcom.fr
> >
Reply to: