[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

non-bonded multiple NICs w/ an unmanaged switch

 I have a cluster of 8 dual Opterons, and one unmanaged D-link 16port gigE
switch.  The Opterons have dual gigE on their mobos, and right now, channel
bonding is enabled.  This is a bit bogus, because both receiving NICs will
get a copy of every packet, I think. (Both NICs get the same MAC address when
bonded, and that's what switches keep track of.)  I do get ~10 or 20% higher
TCP throughput than without bonding, so it is helping a bit.  Somewhat
surprisingly, UDP packets don't seem to be getting duplicated.  I tested
with nc -u, talking to nc -l -u, and stuff I typed was only received once.
Maybe I'm wrong about the switch duplicating the packets, but I certainly
don't get twice the bandwidth.

 Anyway, I've been thinking about what can be done with an unmanaged switch.
I've considered arp table hacks like the U. Kentucky flat-network idea
(google for KLAT2), but not in enough detail to come up with anything good.
Maybe half the nodes could talk to one of the the master nodes NICs, and
half to the other, if the master node has separate MAC addresses and doesn't
use bonding?  This could be useful if there is significant openMosix or NFS
traffic, and not just all<->all MPI traffic.

 I've also thought of having two subnets, and, with each
node having one NIC in each net. could be used for all normal
traffic (NFS, ssh, etc.), while could be exclusively for MPI.  It
might be more convenient to hack config files to get NFS or openMosix using
different IP addresses from everything else, though.

 Anyway, has anyone done anything like this, or want to expand on this idea?
I haven't thought of a useful search string to google on for this kind of
thing yet, so if anyone knows any good web pages about this, I'd love to see

#define X(x,y) x##y
Peter Cordes ;  e-mail: X(peter@cor , des.ca)

"The gods confound the man who first found out how to distinguish the hours!
 Confound him, too, who in this place set up a sundial, to cut and hack
 my day so wretchedly into small pieces!" -- Plautus, 200 BC

Attachment: signature.asc
Description: Digital signature

Reply to: