Re: How to get Amanda and tape libraries to work?
On Thursday 15 December 2005 14:49, jpg wrote:
>Been reading docs and working on understanding amanda for about two
> weeks now and finally got it up and running with a tape
> jukebox/library; SORTOF.
>
>I can load/unload/label tapes in the library, and got a valid
> 'changer.conf', 'disklist', etc.
>
>However cannot get amanda and the 'tpchanger' chg-zd-mtx to backup
> any disk that is larger than the size of one tape in the library.
> Sort of defeats the whole purpose of using amanda and tape
> libraries.
This is a basic limitation of amanda, no disklist entry can be larger
than a tape because it cannot span one backup across multiple tapes.
The usual fix is to use tar, which allows you to break a large
partition into subdir entries, thereby making them fit.
>Plus no data is written to the disk cache, '/var/tmp', either when
> tape drive fails for whatever reason, as advertised in the amanda
> docs.
/Var/tmp might be considered a poor place to put the disk cache by some
as it should easy to check completion by checking that the assignment
is indeed empty when amdump has finished.
In any event, the default value of the size of this holding disk
'cache' reserved for use by incremental only backups in the event of a
tape failure is 100%, so no full dumps will be kept there by default.
You can change this in your amanda.conf file with the reserved
keyword, and its usage is explained in that files comments.
Doing so will also speed up your dumps by quite a bit by allowing the
cache to be used at the softwares full speed rather than speed limited
by the direct to tape method forced when it doesn't have an adequate
cache. Lack of cache also forces it to serialize the backups, where
with it, it can nicely parallelize much of it. Also, offload as much
of the compression to the client machines as possible since they can
all be doing compression at the same time, and the resultant files
then use less network bandwidth so they move faster when the smunching
is done.
Just make sure you specify different spindle numbers for each physical
disk in the disklist options so that it doesn't try to parallelize 2
or more dumps on the same physical disk, which would thrash the seeks
all over the drive while it was running.
As my / has only a 9% usage according to df, I've set the reserved to
20% of the 27GB free, so it will not ever run into this limitation
here. Amanda uses /dumps as this holding disk. Also I have 44
disklist entries, none of which exceed about 2Gb and it all Just
Works(tm) to vdisks located on a 180GB partition on a 200GB drive that
also functions as /var and swap. But with a gig of ram, swap has
never been touched. I didn't exect that, but thats how its worked
here.
As big, hundreds of GB disks are commodity items today, insufficient
holding disk cache is hard to explain. As for explanations and
justifications to TPTB that write the checks for these disks, its just
an accepted part of using the best disk archiver ever and its
performance was designed around having it available.
>There is tons of documentation on amanda, but I have yet to find a
> single occurrence of how to get amanda to backup large disk mounts
> to a library and automatically change tapes as it needs to. Let
> alone why no data is being written to the disk cache, for later
> writing to the tape when tape errors occur.
>
>Now I am not even sure amanda even was designed to do so.
See above. That said, there is a patch, written by John Stange that
does implement the tape spanning ability. Check this lists archives
for data and links to it. I'm told it does work quite well. However,
I find its little enough problem to just use amanda the way it was
designed to be used. That also works very well.
>Has anyone on the list ever gotten amanda and tape libraries to work?
> Or should I look at ufsdump and fsbackup instead?
>
>Here are the specifics:
>----------------------------------------------------------------
>Debian - sarge
>Kernel - 2.6.12.2
>User - backup
>group - backup
>Version of amanda - 2.4.4p3-3
A bit long in the tooth now, we're up to amanda-2.4.5p1-20051201.tar.gz
as of the first of the month. Because once I'd written a script to
configure and build it, I have been using the latest snapshots with
very little fuss to install them. It takes me about 5 minutes to do
the below description.
I su amanda, unpack the tarball right in the /home/amanda directory, cp
my little 15 line script into the tarballs tree, cd to it, run the
script, then hit a ctrl+d to get back to root, cd to the top of that
tree and do a make install, followed by an ldconfig, then su back to
amanda and run an amcheck Daily to see if there are any surprises.
Maybe one a year for 20 updates in that year, usually because of
'operator error' :-)
And, because distro packages have been known to contain less than
optimum configurations, we on the amanda-users list at amanda.org
don't generally recommend using a prepackaged amanda. Its often old,
and rarely configured for your use situation.
That script I alluded to (but its for vdisk usage, needs expanded a wee
bit for tape users):
----------------
#!/bin/sh
# since I'm always forgetting to su amanda...
if [ `whoami` != 'amanda' ]; then
echo
echo "!!!!!!!!!!!! Warning !!!!!!!!!!!!"
echo "Amanda needs to be configured and built by the user amanda,"
echo "but must be installed by user root."
echo
exit 1
fi
make clean
rm -f config.status config.cache
./configure --with-user=amanda \
--with-group=disk \
--with-owner=amanda \
--with-gnu-ld \
--prefix=/usr/local \
--with-tapedev="FILE:/amandatapes" \
--with-debugging=/tmp/amanda-dbg/ \
--with-tape-server=$FQDN \
--with-amandahosts \
--with-configdir=/usr/local/etc/amanda \
--with-config=Daily \
--with-gnutar=/usr/local/bin/tar
make
------------------------
The last line should point at your installed tar, and FWIW 1.14 is
broken, 1.13-19, 1.13-25, and 1.15-1 are fine. The $FQDN should be
replaced by the Fully Qualified Domain Name of the machine acting as
the server, do not, ever, use localhost as it will make recovery
operations difficult if not impossible.
>Tape Library - HP SureStore 12000e
>
>Contents of /etc/amanda/normal/amanda.conf
>=====================================================================
>========= org "Linux Workstation" # Title of report
>mailto "me" # recipients of report, space separated
>dumpuser "backup" # the user to run dumps under
>inparallel 4 # maximum dumpers that will run in parallel
>netusage 600 # maximum net bandwidth for Amanda, in KB per sec
>
># a filesystem is due for a full backup once every days
>dumpcycle 4 weeks # the number of days in the normal dump cycle
>tapecycle 6 tapes # the number of tapes in rotation
>
>bumpsize 20 MB # minimum savings (threshold) to bump level 1 -> 2
>bumpdays 1 # minimum days at each level
>bumpmult 4 # threshold = bumpsize * (level-1)**bumpmult
>
>runtapes 1
If you have a library, you could set runtapes to more than 1.
>tpchanger "chg-zd-mtx"
>changerdev "/dev/sg15"
>tapedev "/dev/nst0" # Linux: norewinding
>changerfile "/etc/amanda/normal/changer"
>
>tapetype HPC1553A # what kind of tape it is (see tapetypes below)
>labelstr "^MY-TAPE-[0-9][0-9]*$" # label constraint regex: all tapes
> must match
>
>diskdir "/var/tmp" # where the holding disk is
>disksize 1000 MB # how much space can we use on it
nowhere near enough space, and its not called "holdingdisk". So there
really isn't one specified here.
It should more or less resemble this, from my amanda.conf:
--------------------
holdingdisk hd1 {
comment "main holding disk"
directory "/dumps" # where the holding disk is
use -500 Mb # how much space can we use on it
# a non-positive value means:
# use all space but that value
chunksize 1Gb # size of chunk if you want big dump to be
# dumped on multiple files on holding disks
# N Kb/Mb/Gb split images in chunks of size N
# The maximum value should be
# (MAX_FILE_SIZE - 1Mb)
# 0 same as INT_MAX bytes
}
>infofile "/var/lib/amanda/normal/curinfo" # database filename
>logfile "/var/log/amanda/normal/log" # log filename
>
>indexdir "/var/lib/amanda/normal/index"
>
>#define tapetype SDT-5000 {
>#comment "Sony SDT-5000"
>#length 1584 mbytes
>#filemark 0 kbytes
>#speed 271 kps
>#}
>
>define tapetype HPC1553A {
> comment "HP SureStore12000E"
> length 3900 mbytes
> filemark 160 kbytes
> speed 453 kbytes
>}
>
>
>define dumptype striped_lvol-tar {
>program "GNUTAR"
>comment "striped_lvol partition dump with tar"
>options compress-fast, index, exclude-list
> "/etc/amanda/normal/striped_lvol.exclude" priority medium
>}
>
>=====================================================================
>=========
>
>Contents of /etc/amanda/normal/changer.conf
>=====================================================================
>========= changerdev=/dev/sg15
>firstslot=1
>lastslot=4
>cleanslot=-1
>autoclean=0
>#lastslot=5
>#cleanslot=6
># autoclean=1 #### Set to '1' or greater to enable
>#
># autocleancount=99 #### Number of access before a clean.
>#
># cleancycle=120 #### Time (seconds) to clean drive
> (default 120) havereader=0
>offline_before_unload=0
>OFFLINE_BEFORE_UNLOAD=0
>poll_drive_ready=10
>max_drive_wait=120
>unloadpause=20
>driveslot=0
>=====================================================================
>=========
>
>
>Output from mtx -f /dev/sg15 inquiry
>=====================================================================
>========= Product Type: Medium Changer
>Vendor ID: 'HP '
>Product ID: 'C1553A '
>Revision: '9503'
>Attached Changer: No
>=====================================================================
>=========
>
>
>Output from chg-zd-mtx -info
>=====================================================================
>========= 1 4 1
>=====================================================================
>=========
>
>
>Output from chg-zd-mtx -reset
>=====================================================================
>========= 1 /dev/nst0
>=====================================================================
>=========
>
>
>Contents of E-mail sent by amdump
>=====================================================================
>========= FAILURE AND STRANGE DUMP SUMMARY:
> hostname.c /striped_lvol lev 0 FAILED [dump larger than tape,
> 5968735 KB, but cannot incremental dump new disk] planner: FATAL
> cannot fit anything on tape, bailing out
>=====================================================================
>=========
>
>Packages and versions:
>=====================================================================
>========= ii mtx 1.2.16rel-4 controls tape
> autochangers ii amanda-client 2.4.4p3-3 Advanced Maryland
> Automatic Network Disk Arc ii amanda-common 2.4.4p3-3
> Advanced Maryland Automatic Network Disk Arc ii amanda-server
> 2.4.4p3-3 Advanced Maryland Automatic Network Disk Arc ii
> gnuplot 4.0.0-2 A command-line driven interactive
> plotting p ii gnuplot-nox 4.0.0-2 A command-line driven
> interactive plotting p ii gnuplot-x11 4.0.0-2
> X11-terminal driver for gnuplot ii gawk 3.1.4-2
> GNU awk, a pattern scanning and processing l ii libc6
> 2.3.2.ds1-22 GNU C Library: Shared libraries and Timezone ii
> libc6-dev 2.3.2.ds1-22 GNU C Library: Development Libraries
> and Hea ii libncurses5 5.4-4 Shared libraries for
> terminal handling ii libncurses5-de 5.4-4 Developer's
> libraries and docs for ncurses ii libncursesw5 5.4-4
> Shared libraries for terminal handling (wide ii mailx
> 8.1.2-0.200405 A simple mail user agent
>ii cpio 2.5-1.3 GNU cpio -- a program to manage
> archives of ii libtar 1.2.11-2 C library for
> manipulating tar archives ii tar 1.14-2 GNU tar
>ii libperl5.8 5.8.4-8 Shared Perl library
>ii libsnmp-perl 5.1.2-6.2 NET SNMP (Simple Network Management
> Protocol ii liburi-perl 1.35-1 Manipulates and accesses
> URI strings ii libwww-perl 5.803-4 WWW client/server
> library for Perl (aka LWP) ii perl 5.8.4-8 Larry
> Wall's Practical Extraction and Report ii perl-base 5.8.4-8
> The Pathologically Eclectic Rubbish Lister ii perl-modules
> 5.8.4-8 Core Perl modules
>=====================================================================
>=========
>
>-jpg
Please subscribe to the amanda-users list, it can be very helpfull, and
isn't THAT busy.
--
Cheers, Gene
People having trouble with vz bouncing email to me should use this
address: <gene.heskett@verizononline.net> which bypasses vz's
stupid bounce rules. I do use spamassassin too. :-)
Yahoo.com and AOL/TW attorneys please note, additions to the above
message by Gene Heskett are:
Copyright 2005 by Maurice Eugene Heskett, all rights reserved.
Reply to: