network reliability testing

[Pavel Jurus <jurus@cs.cas.cz>]

Last few days i get random errors in mpich communication on my cluster
with typical error message looking like this:

p2_2983:  p4_error: socket_recv_on_fd: invalid data type %d

Since such messages appears only recently, quite randomly and after
several hours of computation I guess that the reason is some hardware
problem. I tried some tests (like prime95) on all the nodes but found
nothing so far. The most suspected component is the switch or maybe
cabels now. Does anyone know about some tool that could put heavy load
on my internal cluster network and test whether communication is ok?

Pavel Jurus