I would appreciate any advice that people might have about gigabit ethernet. What sort of performance boost can I realistically expect? What particular NIC's and/or switches seem to work well?
Well, if you primarily want to relieve network congestion, and your app is as CPU-bound as you can make it, you have other options than to just go with GigE. You could go with multiple FE NICs in FNN / Klos / Hypercube topologies and that may do what you need right there.
That having been said, you can get into inexpensive GigE NICs (like the Intel Pro/1000MT for a 32-bit PCI bus) and 16-port Gigabit switches from D-Link, etc. for fairly short dough, relatively speaking. You won't get 1000Mbps out of those (on Linux). I average about 300-400 Mbps with a similar config.
But if you want to compare it to the price of your CPUs be prepared for disappointment. Even inexpensive GigE gear is considerably more expensive than its FE counterparts. And CPU overhead for the GigE drivers is substantial; that will probably solve your CPU / Network load imbalance. For 10 nodes, you have many options available to you. The GigE NICs will give you better performance, but not better price/ performance.
*********************************** Kurt Keville Debian Beowulf Users Group (DeBUG) http://www.extreme-linux.com Cluster On! ***********************************