Re: Upgrading thousands of boxes via APT
Ben wrote:
> I was wondering if there's a Debian specific tool that could facilitate
> managing thousands of machines via APT. I'm aware that many people would
> recommend a sync option, where 1 machine serves as the master, and the
> others sync off of that. Perhaps that is the only reliable approach, but
> I thought I'd just check in w/ the list and see what people recommend.
There is no single standardized tool to do this. Some customized
script writing will be required. However there are lots of options
and lots of possibilities.
Unfortunately you did not say whether you were upgrading from Sarge to
Etch or if this is a routine daily installation of security upgrades
or if other conditions applied. I would suggest different things in
each of those cases. If you could say more about your environment
then better suggestions might be provided.
The majority of users have a small number of machines. The standard
solutions all center around the Debian Release Notes and upgrading
manually. That is the most flexible method but is of course the most
manual method. The numbers of administrators such as yourself with a
large number of machines is smaller. Also they usually have
customized environments making use of completely standardized tools
out of the box difficult. It is harder to make a generalized
solution. But custom solutions for any one particular environment are
almost always possible.
What I have done in the past (also with thousands of machines) to
provide security upgrades is to run a daily cron task that ran a an
upgrade script. I used a private mirror that I controlled. I staged
victim machines to get security upgrades immediately and other
machines received them after a waiting period if no problems were seen
on the "canary" machines. A standard Debian package that may be
useful in this case is fine 'cron-apt' however I found a custom script
solution to be better in my case.
However security upgrades are nice, tidy special cases. Configuration
files don't change. Package names don't change. Very little changes.
But for distribution changes from Sarge to Etch it is more complicated
to automatically upgrade machines. More is needed than tools designed
for security upgrades can provide. In those cases I think only a
custom script upgrade process can work successfully.
Are all of your machines identical? Are there small numbers of known
variations? Are there large numbers of large variations? Desktops?
Servers? A mix? Of course the more similar the pool of machines to
upgrade automatically then the easier this will be but one of the
strengths of Debian systems is the ability to handle gracefully a lot
of variations.
Assuming that you have thousands of machines running Sarge and that
some variation exists but that most are very similar then it is fairly
easy to create a script to automate the upgrade Sarge to Etch. I have
done this several stable Debian releases previously. I would be happy
to provide further information from my own experience and I am sure
that others on the list would as well. Start small and test the
script on a representative machine. Fix any issues found. Work
slowly through several more machines. Gain confidence is the process.
Increase the rollout to the large pool of machines. Finish off any
exception machines that were held off during the original deployment.
It will be done before you know it!
I did not provide details here because they would be overwhelming. If
you (or others) are interested then please keep the dialog going. It
is an interesting topic.
Bob
Reply to: