I have a cluster of 8 dual Opterons, and one unmanaged D-link 16port gigE switch. The Opterons have dual gigE on their mobos, and right now, channel bonding is enabled. This is a bit bogus, because both receiving NICs will get a copy of every packet, I think. (Both NICs get the same MAC address when bonded, and that's what switches keep track of.) I do get ~10 or 20% higher TCP throughput than without bonding, so it is helping a bit. Somewhat surprisingly, UDP packets don't seem to be getting duplicated. I tested with nc -u, talking to nc -l -u, and stuff I typed was only received once. Maybe I'm wrong about the switch duplicating the packets, but I certainly don't get twice the bandwidth. Anyway, I've been thinking about what can be done with an unmanaged switch. I've considered arp table hacks like the U. Kentucky flat-network idea (google for KLAT2), but not in enough detail to come up with anything good. Maybe half the nodes could talk to one of the the master nodes NICs, and half to the other, if the master node has separate MAC addresses and doesn't use bonding? This could be useful if there is significant openMosix or NFS traffic, and not just all<->all MPI traffic. I've also thought of having two subnets, 10.0.0.0 and 10.1.0.0, with each node having one NIC in each net. 10.0.0.0 could be used for all normal traffic (NFS, ssh, etc.), while 10.1.0.0 could be exclusively for MPI. It might be more convenient to hack config files to get NFS or openMosix using different IP addresses from everything else, though. Anyway, has anyone done anything like this, or want to expand on this idea? I haven't thought of a useful search string to google on for this kind of thing yet, so if anyone knows any good web pages about this, I'd love to see links. -- #define X(x,y) x##y Peter Cordes ; e-mail: X(peter@cor , des.ca) "The gods confound the man who first found out how to distinguish the hours! Confound him, too, who in this place set up a sundial, to cut and hack my day so wretchedly into small pieces!" -- Plautus, 200 BC
Attachment:
signature.asc
Description: Digital signature