Re: cross-get script
On 2006-10-23 09:45 +0100, Martin Guy wrote:
> >One thing that strikes me straight away is to remove the need to
> >specify a mirror each time. If it's to be apt-cross it should use
> >/etc/apt/sources.list IMHO along with the usual version handling.
>
> By default, maybe, but I currently need to fetch binary packages for
> the host system from a private repository which has its own modified
> source packages, fetch sources for cross-compilation from a different
> one, the main one (though this may also need to become a local source
> repo), and *also* fetch arch-independent binaries from the main repo.
I agree that you may well want to get your cross packages from
somewhere other than where you get the main system packages (which may
simply not have your cross-arch packages).
After using my script for an hour or so last night I found that a) the
once_a_day funciton was just broken (odd, I stole it from somewhere
where it seemed to be working!) and b) it fails to get
arch-independent packages at all. The latter particularly needs
attention. Other than that I was enormously pleased with the
efficiency improvement it represents - a couple of hours very well
spent!
> I had been doing the first two by specifying deb lines to the private
> binary repo and deb-src lines to the main source repo in sources.list
> but I can't say I'm very happy with that as a general solution, and I
> still had to download the standard arch-indep binaries from the main
> repo manually with copy&paste. I'm hoping this tool will solve both
> of these nasty cases.
Not that last bit, sadly. But i don't think that's very hard to fix.
> The case that Wookey solves (downloading binaries to feed to
> dpkg-cross) is worse still, since it handles binaries that may well
> come from two different binary repos, and there's no clean way to
> solve that with a single sources.list
Can you expand on that a little? Do you mean that (for example) you
might getting some packages from main and some from backports and
ideally you'd like this script to get the equivalent cross-packages?
Because it uses apt to get the lists and search them I'd hope it would
be easy to combine repos in this way. You would depend on apt prefences
or version numbers to choose between repos/packages.
> So until it keeps separate sources.list and apt and dpkg databases for
> the cross-arches it handles, this seems the best for the moment;
It already does this. That's how it works. Well actually this version
has a completely separate dpkg database and config, and a partly
separate apt-database. Separating the apt database out fully too is
not particularly difficult.
Ah, or do you mean as well as separating out its list for the host it
should separate out all its config by arch as well? That is possible.
Probably not even hard.
What is really needed is for apt to be aware that one system may have
binary packages for more than one arch on it. Currently it simply
doesn't understand that at all (beyond the special case of
binary-indep and arch packages, which don't overlap). I understand
that the apt codebase is difficult to grok. And Simon Richter has been
working on an alternative that can do multiple arches. One of those is
probably a longer-term solution, but also much more work than my
simple hack plus a bit of fettling to deal with the above points.
This sort of use of apt (changing all its paths) does reveal some
bugs. I first noted them about a year ago but have not sat down to
write a proper detailed bug-rep. One is noted in the script.
Updated version (with working once_a_day funciton) attached.
Wookey
--
Aleph One Ltd, Bottisham, CAMBRIDGE, CB5 9BA, UK Tel +44 (0) 1223 811679
work: http://www.aleph1.co.uk/ play: http://wookware.org/
#!/bin/sh -e
#maybe this would be better as a perl script?
if [ "" = "$1" ]; then
echo -e "you must suply a package name\napt-cross package [arch] [suite] [mirror]"
exit 1
else
package=$1
fi
#defaults should be derived from dpkg-cross config
#mirror=http://us.arm.mirror.debian.net/debian/
mirror=http://ftp.uk.debian.org/debian/
arch=arm
suite=unstable
dpkg_dir=/var/lib/dpkg-cross
apt_dir=/var/lib/apt-cross
if [ "" != "$2" ]; then
arch=$2
fi
if [ "" != "$3" ]; then
suite=$2
fi
if [ "" != "$4" ]; then
mirror=$2
fi
function do_once_a_day
{
# $1 is touchfile $2 is command
# find finds touchfile if created in this 24hrs.
if [ -z `find $apt_dir -name $1 -daystart -mtime 0` ] ; then
if $2; then touch $1; fi
fi
}
function download_arch_packages()
{
archivename=`cat $apt_dir/sources.$suite | grep "deb " | cut -d " " -f 2`
# for package in $packagelist
# do
filename=`fakeroot apt-cache -o Apt::Architecture=$arch -c \
$apt_dir/apt.conf-$suite show $package | \
grep ^Filename: | sed -e "s/Filename: //"`
wget -nd -N -nv ${archivename}${filename}
# done
}
function update_sources()
{
#start by downloading sources, but only if not already done this 24hrs
do_once_a_day ".aptupdate-stamp" "fakeroot apt-get -o Apt::Architecture=$arch -c $apt_dir/apt.conf-$suite update"
#this doesn't work - but should - needs bug filing
# (cd $native_dir && sudo aptitude -o Apt::Architecture=$arch \
# -o Apt::Dir="/var/emdebian/tools/apt/" \
# -o Apt::Dir::Etc="/var/emdebian/tools/apt/" \
# -o Apt::Dir::Etc::SourceList="sources.$suite" \
# -o Apt::Dir::State=$suite/ \
# -o Apt::Dir::Cache=$suite/ \
# download $packages
}
function setup_config()
{
#set up necessary dirs for cross-dpkg database
for dir in $suite/lists/partial $suite/archives/partial; do
mkdir -p $apt_dir/$dir
done
for file in alternatives info parts updates; do
mkdir -p $dpkg_dir/$file
done
for file in diversion statoverride status lock; do
if [ ! -e $dpkg_dir/$file ] ; then touch $dpkg_dir/$file; fi
done
#make apt-conf file and sources files
if [ ! -e $apt_dir/apt.conf-$suite ]; then
cat > $apt_dir/apt.conf-$suite << **END**
Dir "$apt_dir/"
{
Etc "$apt_dir/"
{
SourceList "sources.$suite";
};
State "$suite/";
Cache "$suite/";
};
**END**
fi
#should check if $mirror is diff to what's in this file and if so re-write.
if [ ! -e $apt_dir/sources.$suite ]; then
cat > $apt_dir/sources.$suite << **END**
deb $mirror $suite main
deb-src $mirror $suite main
**END**
fi
}
#-----------------
#here is the program proper
setup_config
update_sources
download_arch_packages
Reply to: