[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

VXLAN remote mac flapping between IP's

Hi all,

I have several debian servers running multicast vxlan for a qemu/kvm
cluster in a fully routed mini-datacenter.
Each server has multiple links to multiple top of rack switches,
configured as ecmp point-to-points with /30's
Additionally, each server is running ospf via FRR and thus has routes
to each other /30 in this cluster.
Each server also has a lo:1 address set, but I don't think that's
coming into play here...

I've been seeing my dmesg on all members getting spammed with messages
like this:

[169833.510533] vxlan_1gig1: 46:65:af:13:b7:d1 migrated from to
[169833.511121] vxlan_1gig1: 46:65:af:13:b7:d1 migrated from to

I'm seeing this across all vxlan interfaces on each server.
The mac that's flapping is the remote address for the 1gig1 vxlan
interface on one of the other peers, but
.218 and .114 are both 10gig addresses on that same remote machine,
which doesn't make sense as they're connected to different switches,
and that vxlan is bound to a 1gig interface...

>From the documentation, I thought that kernel vxlan interfaces were
tied to a specific interface.
However, 150.114 is on a 10gig switch, and .218 is as well, whereas
this vxlan interface is bound to one of the 1gig links.

It looks like the kernel is seeing better-cost routes via the 10gig
interfaces, and then.... ignoring the device parameter?
There's no packetloss going on, and performance is unaffected. I'd
just like to know more about why this is happening.
I always have syslog and my graylog server, but both the ring buffer
and systemd journal are basically useless because of this constant

This is how I'm building the vxlan interface in
/etc/network/interfaces (identical config across all servers, just
IP's changed).
I know I'm doing things less than ideally, but am I missing something
big here? Any tips for a better implementation?

auto lo
iface lo inet loopback

auto lo:1
iface lo:1 inet static

allow-hotplug 1gig1
iface 1gig1 inet static
mtu 1500
# 1gig1

allow-hotplug 10gig1
iface 10gig1 inet static
mtu 1500
# 10gig1

allow-hotplug 10gig2
iface 10gig2 inet static
mtu 1500
# 10gig2

auto vxlan_1gig1
iface vxlan_1gig1 inet manual
up exec `ip link add vxlan_1gig1 type vxlan id 250 group
dstport 4789 ttl 2 dev 1gig1; ip link set vxlan_1gig1 up; ip addr add dev vxlan_1gig1`
down exec `ip link set vxlan_1gig1 down; ip link del vxlan_1gig1`
# vxlan via 1gig1 - primary corosync network

auto vxlan_10gig1
iface vxlan_10gig1 inet manual
up exec `ip link add vxlan_10gig1 type vxlan id 251 group
dstport 4789 ttl 2 dev 10gig1; ip link set vxlan_10gig1 up; ip addr
add dev vxlan_10gig1`
down exec `ip link set vxlan_10gig1 down; ip link del vxlan_10gig1`
# vxlan via 10gig1 - secondary corosync network, vm migration network

auto vxlan_10gig2
iface vxlan_10gig2 inet manual
        up exec `ip link add vxlan_10gig2 type vxlan id 252 group dstport 4789 ttl 2 dev 10gig2; ip link set vxlan_10gig2
up; brctl addif vmbr0 vxlan_10gig2`
        down exec `ip link set vxlan_10gig2 down; ip link del vxlan_10gig2`
# vxlan via 10gig2 - vm lan

auto vmbr0
iface vmbr0 inet static
bridge_ports vxlan_10gig2
bridge_stp off
bridge_fd 0
# vm network

I'd appreciate any help on this or ideas for further
troubleshooting/improvements, I'm deeply confused at this point but
that probably just means I have something setup wrong.


Reply to: