Bonding not using all 3 interfaces?
Hello,
I'm setting up a Ceph storage with Proxmox 5.0 beta, based on Debian
Stretch.
It's based on 3 Dell PowerEdge R610's, bonded with 3 NIC's. (I'll get
back on this in a bit), all 3 with PERC H700 and RAID1 Kingston SSD.
Writing speeds directly on the SSD give me roughly 300MB/s (which is
still slow, as it should reach 500MB/s).
The switch they are connected to is a HP PowerConnect 2824
The issue I'm having is that when I test my storage speeds I'm only
getting 80MB/s via Ceph.
If testing directly with netperf I'm getting 112MB/s. (I followed the
following guide for testing:
https://blog.codecentric.de/en/2014/03/ceph-object-storage-fast-gets-benchmarking-ceph/)
I have setup bonding, exactly the same on all 3 nodes. All configs can
be found below.
I tried using 'mtu 9000' in the config, but then I got packet loss and
total connection loss. Unkown what's causing it but I disabled it again.
Anyone that might have a clue what I'm doing wrong/what is wrong with
the setup?
Note: Current setup will be for testing only, once everything is
succesfull we will invest in 10G NIC's.
=======
root@kvm-01:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback
iface eno1 inet manual
iface eno2 inet manual
iface eno3 inet manual
iface eno4 inet manual
auto bond1
iface bond1 inet manual
slaves eno2 eno3 eno4
bond_miimon 100
bond_mode 802.3ad
bond_lacp_rate 1
bond_xmit_hash_policy layer3+4
auto bond0
iface bond0 inet manual
slaves eno1
bond_miimon 100
bond_mode 802.3ad
bond_lacp_rate 1
bond_xmit_hash_policy layer3+4
auto vmbr0
iface vmbr0 inet static
address X.X.X.X
netmask 255.255.255.240
gateway X.X.X.X
bridge_ports bond0
bridge_stp off
bridge_fd 0
auto vmbr1
iface vmbr1 inet static
address 10.0.0.2
netmask 255.255.255.0
bridge_ports bond1
bridge_stp off
bridge_fd 0
=======
Switch configuration:
# sh run
Running configuration:
; J4903A Configuration Editor; Created on release #I.08.98
interface 2
no lacp
exit
interface 3
no lacp
exit
interface 4
no lacp
exit
interface 6
no lacp
exit
interface 7
no lacp
exit
interface 8
no lacp
exit
interface 10
no lacp
exit
interface 11
no lacp
exit
interface 12
no lacp
exit
trunk 2-4 Trk1 LACP
trunk 10-12 Trk2 LACP
trunk 6-8 Trk3 LACP
snmp-server community "public" Unrestricted
vlan 1
name "DEFAULT_VLAN"
untagged 16-17,23-24
ip address dhcp-bootp
no untagged 1,5,9,13-15,18-22,Trk1-Trk3
exit
vlan 40
name "OOB"
untagged 21,Trk1-Trk3
ip address 10.0.0.254 255.255.255.0
exit
vlan 300
name "IPMI"
untagged 13-15,18-19
tagged 24
exit
vlan 133
name "VLAN133"
untagged 1,5,9,22
tagged 24
exit
vlan 135
name "VLAN135"
untagged 20
tagged 24
exit
spanning-tree Trk1 priority 4
spanning-tree Trk2 priority 4
spanning-tree Trk3 priority 4
=======
root@kvm-02:~# cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 14:fe:b5:d2:43:1b
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 3
Actor Key: 9
Partner Key: 53
Partner Mac Address: 00:15:60:fe:46:80
Slave Interface: eno2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 14:fe:b5:d2:43:1b
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: 14:fe:b5:d2:43:1b
port key: 9
port priority: 255
port number: 1
port state: 63
details partner lacp pdu:
system priority: 18048
system mac address: 00:15:60:fe:46:80
oper key: 53
port priority: 0
port number: 6
port state: 61
Slave Interface: eno3
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 14:fe:b5:d2:43:1d
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: 14:fe:b5:d2:43:1b
port key: 9
port priority: 255
port number: 2
port state: 63
details partner lacp pdu:
system priority: 18048
system mac address: 00:15:60:fe:46:80
oper key: 53
port priority: 0
port number: 7
port state: 61
Slave Interface: eno4
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 14:fe:b5:d2:43:1f
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: 14:fe:b5:d2:43:1b
port key: 9
port priority: 255
port number: 3
port state: 63
details partner lacp pdu:
system priority: 18048
system mac address: 00:15:60:fe:46:80
oper key: 53
port priority: 0
port number: 8
port state: 61
======
Reply to: