Re: IP performance question
Hi.
On Sun, 24 May 2015 13:26:52 +0200
Petter Adsen <petter@synth.no> wrote:
> > On Sun, 24 May 2015 11:28:36 +0200
> > Petter Adsen <petter@synth.no> wrote:
> >
> > > > On Sun, 24 May 2015 10:36:39 +0200
> > > > Petter Adsen <petter@synth.no> wrote:
> > > >
> > > > > I've been trying to improve NFS performance at home, and in that
> > > > > process i ran iperf to get an overview of general network
> > > > > performance. I have two Jessie hosts connected to a dumb switch
> > > > > with Cat-5e. One host uses a Realtek RTL8169 PCI controller, and
> > > > > the other has an Intel 82583V on the motherboard.
> > > > >
> > > > > iperf maxes out at about 725Mbps. At first I thought maybe the
> > > > > switch could be at fault, it's a really cheap one, so I
> > > > > connected both hosts to my router instead. Didn't change
> > > > > anything, and it had no significant impact on the load on the
> > > > > router. I can't try to run iperf on the router (OpenWRT),
> > > > > though, as it maxes out the CPU.
> > > > >
> > > > > Should I be getting more than 725Mbps in the real world?
> > > >
> > > > A quick test in my current environment shows this:
> > > >
> > > > [ ID] Interval Transfer Bandwidth
> > > > [ 3] 0.0-10.0 sec 1.10 GBytes 941 Mbits/sec
> > > >
> > > > Two hosts, connected via Cisco 8-port unmanaged switch, Realtek
> > > > 8168e on one host, Atheros Attansic L1 on another.
> > > >
> > > > On the other hand, the same test, Realtek 8139e on one side, but
> > > > with lowly Marvell ARM SOC on the other side shows this:
> > > >
> > > > [ ID] Interval Transfer Bandwidth
> > > > [ 3] 0.0-10.0 sec 534 MBytes 448 Mbits/sec
> > > >
> > > > So - you can, definitely, and yes, it depends.
> > >
> > > That last one, would that be limited because of CPU power?
> >
> > That too. You cannot extract that much juice from a single-core ARM5.
> > Another possibility is that Marvell is unable to design a good chipset
> > even in the case it would be a matter of life and death :)
>
> That might be why I'm not using the Marvell adapter :) I remember
> reading somewhere that either Marvell or Realtek were bad, but I
> couldn't remember which one, so I kept using the Realtek one since I
> had obviously switched for a reason :)
Both are actually. Realtek *was* good at least 5 years ago, but since
then they managed to introduce multiple chips that are managed by the
same r8169 kernel module. Since then it became a matter of luck. Either
your NIC works flawlessly without any firmware (mine does), or you're
getting all kinds of weird glitches.
> > > > Try the same test but use UDP instead of TCP.
> > >
> > > Only gives me 1.03Mbits/sec :)
> >
> > iperf(1) says that by default UDP is capped on 1Mbit. Use -b option on
> > client side to set desired bandwidth to 1024m like this:
> >
> > iperf -c <server> -u -b 1024M
> >
> > Note that -b should be the last option. Gives me 812 Mbits/sec with
> > the default UDP buffer settings.
>
> Didn't notice that. I get 808, so close to what you get.
Good. The only thing is left to do is to apply that 'udp' flag to NFS
clients, and you're set. Just don't mix it with 'async' flag, as Bad
Things ™ can happen if you do so (see nfs(5) for the gory details).
> > > > Increase TCP window size (those net.core.rmem/wmem sysctls) on
> > > > both sides.
> > >
> > > It is currently 85KB and 85.3KB, what should I try setting them to?
> >
> > Try these:
> >
> > net.core.rmem_max = 4194304
> > net.core.wmem_max = 1048576
>
> OK, I've set them on both sides, but it doesn't change the results, no
> matter what values I give iperf with -w.
Now that's weird. I picked those sysctl values from one of NFS
performance tuning guides. Maybe I misunderstood something.
> > > > Try increasing MTU above 1500 on both sides.
> > >
> > > Likewise, increase to what?
> >
> > The amount of your NICs support. No way of knowing the maximum unless
> > you try. A magic value seems to be 9000. Any value above 1500 is
> > non-standard (so nothing is guaranteed)
> >
> > A case study (that particular NIC claims to support MTU of 9200 in
> > dmesg):
> >
> > # ip l s dev eth0 mtu 1500
> > # ip l s dev eth0 mtu 9000
> > # ip l s dev eth0 mtu 65536
> > RTNETLINK answers: Invalid argument
> >
> > Of course, for this to work it would require to increase MTU on every
> > host between your two, so that's kind of a last resort measure.
>
> Well, both hosts are connected to the same switch (or right now, to the
> router, but I could easily put them back on the switch if that
> matters). One of the hosts would not accept a value larger than 7152,
> but it did have quite an effect: I now get up to 880Mbps :)
Consider yourself lucky as MTU experiments on server hardware usually
lead to a long trip to a datacenter :)
> Will this setting have an impact on communication with machines where
> the MTU is smaller? In other words, will it have a negative impact on
> general network performance, or is MTU adjusted automatically?
Well, back in the old days it was simple. Hardware simply rejected all
frames which MTU exceeded their settings (i.e. 1500).
Since then they introduced all those smart switches which presumably
fragment such frames to their MTU (i.e. 1 big frame transforms into
several small ones). Which is considered CPU-costly and ineffective,
but no packets are lost that way.
Still, since MTU is strictly L2 category (according to OSI Model), and
you managed to send this very e-mail I'm replying to - big MTU should
not hamper your ability to communicate to the outside world. That means
that about the only problem you should encounter will manifest yourself
as soon as you introduce third computer into your LAN.
> And what is the appropriate way of setting it permanently - rc.local?
/etc/network/interfaces supports 'mtu' setting. So doing it Debian way
means you locate 'eth0' (or whatever) interface definition and set MTU
there. The only caveat I can think of is that 'mtu' setting does not
seem to be supported for 'dhcp'. See interfaces(5).
Reco
Reply to: