Re: Bonded gigabit cards

[Eric Nodwell <nodwell@physics.ubc.ca>]

We use bonded ethernet on a (now somewhat old) cluster running Debian.
Once you get it set up, it works like a charm.

Bonding doubles your throughput, but does nothing for latency, which
is where Myrinet is particulary strong.  I don't have any numbers
offhand.

There is practically no CPU overhead from using bonded networking
(above and beyond normal network overhead).  It's a very trivial thing
from the point of view of the kernel.

You're correct, that you construct two physically independent
networks, each with their own switches.  You could use either of these
networks in normal (non-channel-bonded) mode.

Channel bonding is transparent to practically all network protocols,
but channel bonding does *not* interoperate with non-channel-bonded
networking.  (The non-channel bonded machines will not receive every
second packet.)  One consequence is that network booting does not work
(DHCP, BOOTP, Etherboot, PXE, whatever).  This can be an
inconvenience.  For example, we use FAI to automatically install
nodes, but if for example we have a disk failure on a node, we have to
schedule a cluster shutdown to switch to non-channel-bonded to
reinstall the node.

I had problems with the ifenslave package in Debian stable, and
backported the version from the testing branch, which worked fine.

Here's my entry in /etc/network/interfaces:

iface eth0 inet static
    address 192.168.253.01
    netmask 255.255.255.0
    broadcast 192.168.253.255
    gateway 192.168.253.250

auto bond0
iface bond0 inet static
    address 192.168.253.01
    netmask 255.255.255.0
    broadcast 192.168.253.255
    gateway 192.168.253.250
    up ifenslave bond0 eth0
    up ifenslave bond0 eth1
    post-down ifconfig eth1 down
    post-down ifconfig eth0 down

Notice that I have configurations for both channel-bonded and
non-channel-bonded networking (same ip, which isn't necessary).  I can
switch to non-channel-bonded with

ifdown bond0
ifup eth0


cheers,
Eric


On Thu, Aug 07, 2003 at 12:47:00PM -0700, Josh Lauricha wrote:
> I am doing research for a small cluster to be built for use at UCR's
> bioinformatics department. Since this cluster will mainly be used in
> blast or similar applications that mainly involve a few transfers of
> large chunks of data, I am of the opinion that Myrinet cards would be
> a massive waste of money. Instead, we plan on using dual bonded gigabit
> channels. Is anyone aware of the CPU overhead incurred by bonded gigabit
> devices?
> 
> After skimming through the bonding documents with the kernel, it appears
> that boding is possible over any switches, however it seems to be
> implying that this is for high availability boding, rather than high
> performance bonding. From previous posts to this list, I seem to recall
> that its possible to have eth0 and eth1 connected through different
> switches on different subnets and relying one the kernel bonding to
> handle the rest, is this accurate?
> 
> 
> Thanks in advance,