Re: Bonding not using all 3 interfaces?
Florius <floriusrk@gmail.com> wrote:
> I'm setting up a Ceph storage with Proxmox 5.0 beta, based on Debian
> Stretch. It's based on 3 Dell PowerEdge R610's, bonded with 3 NIC's.
> (I'll get back on this in a bit), all 3 with PERC H700 and RAID1
> Kingston SSD. Writing speeds directly on the SSD give me roughly
> 300MB/s (which is still slow, as it should reach 500MB/s). The switch
> they are connected to is a HP PowerConnect 2824
> The issue I'm having is that when I test my storage speeds I'm only
> getting 80MB/s via Ceph. If testing directly with netperf I'm getting
> 112MB/s. (I followed the following guide for testing:
> https://blog.codecentric.de/en/2014/03/ceph-object-storage-fast-gets-benchmarking-ceph/)
> Anyone that might have a clue what I'm doing wrong/what is wrong with
> the setup?
802.3ad or LACP does *not* round-rob the packages across the individual
links nor does it any kind of load-balancing based on traffic. The
distribution is static and deterministic, based on the hashing policy
you select on both the server and the switch. But any one data flow can
only ever run over one NIC and one NIC alone, which is why your netperf
tests caps at 112MB/s.
The normal hashing function for a layer2-based approach is like this:
(src-MAC XOR dst-MAC) modulo numer-of-NICs-in-bond
You selected a hash policy consisting of layer3 (IP) and layer4 (TCP/UDP
Port) on your servers, which will cover the outgoing packages but I
don't see you configuring the same policy on the switch.
The default configuration for most switches is to use a hash based on
layer2 (MAC address).
This means if you are really unlucky and depending on the MAC addresses
of you servers, the hashing might put every packet on the same NIC.
How to improve this:
a) Select a hash policy which involves the IP. You already did this for
the bond interface on the servers, you also have to do this on the
switch.
b) Make sure to use consecutive IPs for the CEPH bond. For example:
server1 gets x.x.x.11, server2 x.x.x.12 and server3 x.x.x.13
This way you make sure not all packages between servers are put on
the same NIC (or NICs).
Grüße,
Sven.
--
Sigmentation fault. Core dumped.
Reply to: