Bug#478904: cron script for apt needs improvement
Package: apt
Version: 0.7.11
Severity: normal
Tags: patch
I saw several issues (Bug 438803, Bug 462734, Bug 454989, Bug 341970) as
detailed below with curent cron script for apt and made fixed script.
Although diff is large, it is mostly reorder and refactoring with more
sanity checks and bug fix.
I also implemented snapshot archive backup with hardlinks. It works
like plan9 fs or apple time machine. This should allows automatic
autoclean without loosing them immediately but properly aged.
So this is now very usable addition to unstable user.
My script is attached. All you have to do is put it as
debian/apt.cron.daily file into source. (Probably for post lenny)
(I am testing this script now under sid.)
=== details ===
I think current use of unattended-upgrades package without even
suggesting in dependency nor documented clearly in script itself is bad.
Now at least checks script. I still do not see why python dependency is
needed when stable security update is needed. unattended-upgrades
should be done by simple shell script as long as objective is
to understand:
Unattended-Upgrade::Allowed-Origins:: "Debian stable";
APT::NeverAutoRemove:: "^linux-image.*";
APT::NeverAutoRemove:: "^linux-restricted-modules.*";
(It seems latter 2 are from apt package so it should be supported)
The first one can be apt-config with sed should extract it.
Then we can use sources.list and Release files in /var/lib/apt/lists
should be able to be used. .... This part, I am not so interested but
good to keep noting here.
Here are things I did on this script:
1. remove funcrion and clean script and reorder logically
2. add 3 layer backup function using hardlink so autoclean is safe
to issue frequently (I think we all use unix)
3. Use "which ... >/dev/null" test for all command existance before use
(it is cron so it should not run if these commands are removed.)
4. Preserve apt-get check -qq output since it may break with other reason
the fix you want.)
5. Made all time stamp update as condituional to success of command.
6. $now is defined once in script.
7. use -m with du to simplify script to MB units
8. use -ne instead of ! $x -eq $y
9. unattended-upgrades package is needed fot unattended part to function.
Lighter weight solution is desired. (Now at least check existance by
cron script)
--> unattended-upgrades does not seem yey to produce error code when it
fails to upgrade. cron script now is ready when it does.
In relation to curent bugs:
Bug 438803:
dbus error issue fixed. Neglect them.
Bug 462734 and Bug 454989:
slightly better patch applied to report full information
(its bug is open but patch was already applied)
Question seems to be if we should use "-f".
Bug 341970:
Only download/upgrade if previous step succeed. Done by nesting.
Here are what this alternative apt /etc/cron.daily addresses.
I hope this will be a place to discuss cron script.
-- Package-specific info:
-- apt-config dump --
APT "";
APT::Architecture "amd64";
APT::Build-Essential "";
APT::Build-Essential:: "build-essential";
APT::Install-Recommends "1";
APT::Install-Suggests "0";
APT::Acquire "";
APT::Acquire::Translation "environment";
APT::Authentication "";
APT::Authentication::TrustCDROM "true";
APT::NeverAutoRemove "";
APT::NeverAutoRemove:: "^linux-image.*";
APT::NeverAutoRemove:: "^linux-restricted-modules.*";
Dir "/";
Dir::State "var/lib/apt/";
Dir::State::lists "lists/";
Dir::State::cdroms "cdroms.list";
Dir::State::userstatus "status.user";
Dir::State::status "/var/lib/dpkg/status";
Dir::Cache "var/cache/apt/";
Dir::Cache::archives "archives/";
Dir::Cache::srcpkgcache "srcpkgcache.bin";
Dir::Cache::pkgcache "pkgcache.bin";
Dir::Etc "etc/apt/";
Dir::Etc::sourcelist "sources.list";
Dir::Etc::sourceparts "sources.list.d";
Dir::Etc::vendorlist "vendors.list";
Dir::Etc::vendorparts "vendors.list.d";
Dir::Etc::main "apt.conf";
Dir::Etc::parts "apt.conf.d";
Dir::Etc::preferences "preferences";
Dir::Bin "";
Dir::Bin::methods "/usr/lib/apt/methods";
Dir::Bin::dpkg "/usr/bin/dpkg";
Dir::Log "var/log/apt";
Dir::Log::Terminal "term.log";
DPkg "";
DPkg::Pre-Install-Pkgs "";
DPkg::Pre-Install-Pkgs:: "/usr/sbin/apt-listbugs apt || exit 10";
DPkg::Pre-Install-Pkgs:: "/usr/bin/apt-listchanges --apt || test $? -ne 10";
DPkg::Pre-Install-Pkgs:: "/usr/sbin/dpkg-preconfigure --apt || true";
DPkg::Tools "";
DPkg::Tools::Options "";
DPkg::Tools::Options::/usr/sbin/apt-listbugs "";
DPkg::Tools::Options::/usr/sbin/apt-listbugs::Version "2";
DPkg::Tools::Options::/usr/bin/apt-listchanges "";
DPkg::Tools::Options::/usr/bin/apt-listchanges::Version "2";
DPkg::Post-Invoke "";
DPkg::Post-Invoke:: "if [ -x /usr/bin/debsums ]; then /usr/bin/debsums --generate=nocheck -sp /var/cache/apt/archives; fi";
Unattended-Upgrade "";
Unattended-Upgrade::Allowed-Origins "";
Unattended-Upgrade::Allowed-Origins:: "Debian stable";
-- /etc/apt/preferences --
Explanation: Pinned by apt-listbugs at Tue Apr 15 00:24:56 +0900 2008
Explanation: #418462: mailman: Fails to upgrade from Sarge to Etch
Package: python
Pin: version 2.4.4-6
Pin-Priority: 1000
-- /etc/apt/sources.list --
deb http://cdn.debian.or.jp/debian/ sid main contrib non-free
deb-src http://cdn.debian.or.jp/debian/ sid main contrib non-free
-- System Information:
Debian Release: lenny/sid
APT prefers unstable
APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)
Kernel: Linux 2.6.24-1-amd64 (SMP w/2 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash
Versions of packages apt depends on:
ii debian-archive-keyring 2008.04.16+nmu1 GnuPG archive keys of the Debian a
ii libc6 2.7-10 GNU C Library: Shared libraries
ii libgcc1 1:4.3.0-3 GCC support library
ii libstdc++6 4.3.0-3 The GNU Standard C++ Library v3
apt recommends no packages.
-- no debconf information
#!/bin/sh
#
# vim: set sts=4 ai:
#
#
#set -x
#set -e
#
# This file understands the following apt configuration variables:
#
# "Dir='/'"
# - RootDir for all configuration files
#
# "Dir::Cache='var/apt/cache/'",
# - Set apt package cache directory
#
# "Dir::CacheArchive='archives/'",
# - Set package archive directory
#
# "APT::Periodic::BackupArchiveInterval=0",
# - Backup after n-days if archive contents changed.(0=disable)
#
# "Dir::BackupArchive=backup",
# - Set periodic package backup directory
#
# "APT::Archives::MaxAge=0",
# - Set maximum allowed age of a cache package file. If a cache
# package file is older it is deleted (0=disable)
#
# "APT::Archives::MinAge=2"
# - Set minimum age of a package file. If a file is younger it
# will not be deleted (0=disable). Usefull to prevent races
# and to keep backups of the packages for emergency.
#
# "APT::Archives::MaxSize=0",
# - Set maximum size of the cache in MB (0=disable). If the cache
# is bigger, cached package files are deleted until the size
# requirement is met (the biggest packages will be deleted
# first).
#
# "APT::Periodic::Update-Package-Lists=1"
# - Do "apt-get update" automatically every n-days (0=disable)
#
# "APT::Periodic::Download-Upgradeable-Packages=0",
# - Do "apt-get upgrade --download-only" every n-days (0=disable)
#
# "APT::Periodic::Unattended-Upgrade"
# - Run the "unattended-upgrade" security upgrade script
# every n-days (0=disabled)
# Requires the package "unattended-upgrades" and will write
# a log in /var/log/unattended-upgrades
#
# "APT::Periodic::AutocleanInterval"
# - Do "apt-get autoclean" every n-days (0=disable)
#
check_stamp()
{
stamp="$1"
interval="$2"
if [ $interval -eq 0 ]; then
return 1
fi
if [ ! -f $stamp ]; then
update_stamp $stamp
# treat as enough time has passed
return 0
fi
# compare midnight today to midnight the day the stamp was updated
stamp=$(date --date=$(date -r $stamp --iso-8601) +%s)
delta=$(($now-$stamp))
# intervall is in days,
interval=$(($interval*60*60*24))
#echo "stampfile: $1"
#echo "interval=$interval, now=$now, stamp=$stamp, delta=$delta"
if [ $delta -ge $interval ]; then
return 0
fi
return 1
}
update_stamp()
{
stamp="$1"
touch $stamp
}
debug_echo()
{
# activate by uncomment
# echo $1
:
}
# check apt-config exstance
if ! which apt-config >/dev/null; then
exit 0
fi
# laptop check, on_ac_power returns:
# 0 (true) System is on mains power
# 1 (false) System is not on mains power
# 255 (false) Power status could not be determined
# Desktop systems always return 255 it seems
if which on_ac_power >/dev/null; then
on_ac_power
if [ $? -eq 1 ]; then
exit 0
fi
fi
# check if we can lock the cache and if the cache is clean
if which apt-get >/dev/null && ! apt-get check -qq 2>/dev/null; then
echo "$0: error encountered in cron job with \"apt-get check -qq\"."
apt-get check -qq || true
exit 1
fi
# Current time in seconds since 1970-01-01 00:00:00 UTC
now=$(date +%s)
# Set default values and normalize
Dir="/"
eval $(apt-config shell Dir Dir)
Dir=${Dir%/}
CacheDir="var/cache/apt/"
eval $(apt-config shell CacheDir Dir::Cache)
CacheDir=${CacheDir%/}
if [ -z "$CacheDir" ]; then
echo "practically empty Dir::Cache=, exiting"
exit
fi
CacheArchive="archives/"
eval $(apt-config shell CacheArchive Dir::Cache::archives)
CacheArchive=${CacheArchive%/}
if [ -z "$CacheArchive" ]; then
echo "practically empty Dir::Cache::archives=, exiting"
exit
fi
BackupArchiveInterval=0
eval $(apt-config shell BackupArchiveInterval APT::Periodic::BackupArchiveInterval)
BackupArchive="backup"
eval $(apt-config shell BackupArchive Dir::BackupArchive)
BackupArchive=${BackupArchive%/}
if [ -z "$BackupArchive" ]; then
echo "empty Dir::Cache::backup, exiting"
exit
fi
MaxAge=0
eval $(apt-config shell MaxAge APT::Archives::MaxAge)
MinAge=2
eval $(apt-config shell MinAge APT::Archives::MinAge)
MaxSize=0
eval $(apt-config shell MaxSize APT::Archives::MaxSize)
UpdateInterval=0
eval $(apt-config shell UpdateInterval APT::Periodic::Update-Package-Lists)
DownloadUpgradeableInterval=0
eval $(apt-config shell DownloadUpgradeableInterval APT::Periodic::Download-Upgradeable-Packages)
UnattendedUpgradeInterval=0
eval $(apt-config shell UnattendedUpgradeInterval APT::Periodic::Unattended-Upgrade)
AutocleanInterval=0
eval $(apt-config shell AutocleanInterval APT::Periodic::AutocleanInterval)
Cache="${Dir}/${CacheDir}/${CacheArchive}/"
Back="${Dir}/${CacheDir}/${BackupArchive}/"
BackX="${Back}${CacheArchive}/"
Back0="${Back}0/"
Back1="${Back}1/"
Back2="${Back}2/"
# backup after n-days if archive contents changed.
# (This uses hardlink to save disk space)
BACKUP_ARCHIVE_STAMP=/var/lib/apt/periodic/backup-archive-stamp
if check_stamp $BACKUP_ARCHIVE_STAMP $BackupArchiveInterval && \
[ $({(cd $Cache 2>/dev/null; find . -name "*.deb"); (cd $Back0 2>/dev/null;find . -name "*.deb" ; })| sort|uniq -u|wc -l) -ne 0 ]; then
mkdir -p $Back
if [ -e $Back1 ]; then rm -rf $Back2 ; mv -f $Back1 $Back2 ; fi
if [ -e $Back0 ]; then rm -rf $Back1 ; mv -f $Back0 $Back1 ; fi
rm -rf $Back0 ; cp -la $Cache $Back ; mv -f $BackX $Back0
update_stamp $BACKUP_ARCHIVE_STAMP
else
echo not run
fi
# package archive contnts removal by package age
if [ $MaxAge -ne 0 ] && [ $MinAge -ne 0 ]; then
find $Cache -name "*.deb" \( -mtime +$MaxAge -and -ctime +$MaxAge \) -and -not \( -mtime -$MinAge -or -ctime -$MinAge \) -print0 | xargs -r -0 rm -f
elif [ $MaxAge -ne 0 ]; then
find $Cache -name "*.deb" -ctime +$MaxAge -and -mtime +$MaxAge -print0 | xargs -r -0 rm -f
fi
# package archive contnts removal down to $MaxSize
if [ $MaxSize -ne 0 ]; then
MinAge=$(($MinAge*24*60*60))
# reverse-sort by mtime
for file in $(ls -rt $Cache/*.deb 2>/dev/null); do
du=$(du -m -s $Cache)
size=${du%%/*}
# check if the cache is small enough
if [ $size -lt $MaxSize ]; then
break
fi
# check for MinAge of the file
if [ $MinAge -ne 0 ]; then
# check both ctime and mtime
mtime=$(stat -c %Y $file)
ctime=$(stat -c %Z $file)
if [ $mtime -gt $ctime ]; then
delta=$(($now-$mtime))
else
delta=$(($now-$ctime))
fi
#echo "$file ($delta), $MinAge"
if [ $delta -le $MinAge ]; then
#echo "Skiping $file (delta=$delta)"
break
fi
fi
# delete oldest file
rm -f $file
done
fi
# update package lists
UPDATE_STAMP=/var/lib/apt/periodic/update-stamp
if which apt-get >/dev/null && check_stamp $UPDATE_STAMP $UpdateInterval; then
if apt-get -qq update 2>/dev/null; then
if which dbus-send >/dev/null; then
# dbus signal is optional
dbus-send --system / app.apt.dbus.updated boolean:true || true
fi
update_stamp $UPDATE_STAMP
# download all upgradeable packages if it is requested
DOWNLOAD_UPGRADEABLE_STAMP=/var/lib/apt/periodic/download-upgradeable-stamp
if which apt-get >/dev/null && check_stamp $DOWNLOAD_UPGRADEABLE_STAMP $DownloadUpgradeableInterval; then
if apt-get -qq -d dist-upgrade 2>/dev/null; then
update_stamp $DOWNLOAD_UPGRADEABLE_STAMP
# auto upgrade all upgradeable packages
UPGRADE_STAMP=/var/lib/apt/periodic/upgrade-stamp
if which unattended-upgrade >/dev/null && check_stamp $UPGRADE_STAMP $UnattendedUpgradeInterval; then
if unattended-upgrade; then
update_stamp $UPGRADE_STAMP
fi
fi
fi
fi
fi
fi
# autoclean package archive
AUTOCLEAN_STAMP=/var/lib/apt/periodic/autoclean-stamp
if which apt-get >/dev/null && check_stamp $AUTOCLEAN_STAMP $AutocleanInterval; then
if apt-get -qq autoclean; then
update_stamp $AUTOCLEAN_STAMP
fi
fi
Reply to: