Re: how to upgrade dozens of debian servers

[Craig Sanders <cas@taz.net.au>]

On Fri, Nov 15, 2002 at 10:40:21AM +0800, Patrick Hsieh wrote:
> I have some debian servers and hav a pain when these is security upgrade 
> package available, for I have to check and upgrade them one by one, making 
> sure they are in safe status.
> 
> I wonder how the administrator manage dozens or even hundreds of debian 
> servers in this case? Any tool or administration tips?

i use simple scripts like the following.  they're not perfect, but they do the
job and they're quite adequate for a scripted install of security updates.

you still have to watch it and be ready to answer any questions or occasionally
hit enter, the advantage of using the scripts is that you don't forget any of
the machines because the script makes sure you do them all in order.   so you
can work on other stuff while the scripts are running, checking it occasionally
to see if you need to answer any questions.  for some upgrades (i.e. where
there are no questions to answer) it is completely automated.

for debconf questions, you can answer them in advance using tools like
debconf-db.  apt can also be configured to use --force-confdef or
--force-confold etc when it runs dpkg.

this may not suit everyone, but it suits me - i prefer it to be semi-automated
like this because i just don't trust 100% automated upgrades.  i want me or one
of my assistant sysadmins to be actively involved and in control of the
upgrade.


btw, i have also used these scripts to do stuff like update /etc/hosts.allow,
install authorised_keys for ssh, and install custom config files.  in the
latter case, i create a template for the config file, use scp to copy it to
each remote machine, and then use a perl (or sed or awk but usually perl)
script to search and replace whatever is needed to localise it for each
machine.


anyway, enough discussion.  here are the scripts:


first there's the script which does the work.  usually this just runs ssh to
execute some commands.  sometimes it also does pre-upgrade setup work like
using scp to copy files or scripts to /tmp/ on the remote machine.

i do the first upgrade by hand so that i know what is going to happen before i
write the script.


---bind.sh---
#! /bin/sh

HOST="$1"

ssh $HOST '
PATH="/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin"

apt-get update
apt-get -y -u install bind
'
---bind.sh---



then there's a wrapper script which runs the work script on each host.  this
one rarely changes.  it gets the list of servers from a file called servers.txt
- file format is one hostname per line, blank lines and comments are ignored.

---upgrade-all.sh---
#! /bin/sh

SCRIPT="$1"
LOG="$1.done"
touch $LOG

for i in $(egrep -v "#|^[[:space:]]*$" servers.txt) ; do
  if ! grep -q "^$i " $LOG ; then
    if fping -q $i ; then
      echo $i:
      $SCRIPT $i && echo "$i $(date '+%Y-%m-%d %H:%M:%S')" >>$LOG
	else
	  echo "$i: not responding to fping"
	fi
  else
    echo "$i: skipped (already done)"
  fi
done
---upgrade-all.sh---


for a simple test of this system, create a script like "bind.sh" above (call it
"test.sh") but replace both of the apt-get lines with just "echo passed test"
and then run it as "./upgrade-all.sh ./test.sh".


note that the logging only logs whether the ssh command was executed.  it
doesn't log whether the upgrade completed successfully.  the purpose of the log
is to make it easy to run the script again without trying to upgrade hosts that
have already been upgraded.

to tell whether each upgrade has succeeded, you need to run upgrade-all.sh
using script to log a complete transcript of what happened.  see script(1) for
details.  script logs all output, including control-characters and escape codes
so dialog boxes (e.g. from debconf) can make the logfile difficult to read.

like so:

script -f ./test.log
./upgrade-all.sh ./test.sh
exit



craig

-- 
craig sanders <cas@taz.net.au>

Fabricati Diem, PVNC.
 -- motto of the Ankh-Morpork City Watch