[sailer@bnl.gov: Network Throughput]

[Bulent Murtezaoglu <bm@acm.org>]

    tps> ... As part of
    tps> the traffic going through the box, some streams have 1000k
    tps> window size for a certain reason. ...

This is the TCP window?  Are you sure both sides can use the window 
scale option?

[...]
    tps> PS: This is really something to do with the window size and
    tps> WAN latency.  

If everything is set up right TCP _should_ adapt and crank up the
bandwidth utilization as the transfer progresses.  Latency alone
should not hurt you much for long transfers, lossage+high latency might.
(all of the top of my head, grans of salt recommended). 

    tps> The box does well when traffic goes in one NIC
    tps> and out the other, as long as the end point is local When it
    tps> hits the WAN, it all dies. [...]

Maybe some router/firewall admin along the way is broken?  Try either
sniffing the wire or turning off path mtu discovery outright.  If
something on the path is dropping packets with the DF bit set and the
resulting ICMP message is not finding its way back to your box you'll
effectively be feeding large packets into a black hole.  The
proc.txt file in the Documentation directory of the kernel source
should give you the info to tweak your kernel's behaviour by poking
stuff into the files in /proc.

Pls. let us know what you end up finding.

cheers,

BM