Managing disperse servers

[Keith Edmunds <kae@midnighthax.com>]

I'd be interested in hearing what others do to manage a number of
geographically-disperse servers. We currently use Nagios/Munin for
monitoring, which we're happy with, but system management is more
challenging.

How do others approach the problem of security updates? Up until now,
we've done this manually with some help from 'cssh' for some servers;
however, that solution doesn't scale as the number of servers increases.
We're reluctant to have servers automatically install updates. We're
looking at CfEngine and Puppet, but I'd be interested in hearing of other
approaches.

I'm also interested in hearing of other techniques for managing multiple,
mostly-similar (but not identical) systems. We're currently managing about
40 such servers, so not a huge number, but we're expecting that number to
grow and we want to put some tools and techniques in place before we drown
in trying to manually manage them.

Thanks,
Keith