Bug#737085: apt: Apt downloads arch all packages from wrong repo/checks wrong checksum
On Thu, Jan 30, 2014 at 12:27:21PM +0000, Wookey wrote:
> +++ Julian Andres Klode [2014-01-30 08:12 +0100]:
> > On Thu, Jan 30, 2014 at 03:13:16AM +0000, Wookey wrote:
> > > Package: apt
> > > Version: 0.9.15
> > > Severity: important
> > >
> > > In the sources I have my own bootstrap repository containing a lot of
> > > (unstable) packages built for arm64, and plain debian unstable and saucy repos
> > >
> > > apt-get install <arch-all-package> (that is available in all 3 repos)
> > > results in a size mismatch error. It seems that apt is using the
> > > checksum from one repo but downloading the package from another.
> > >
> > > The packages used is just an example it seems to be the same for any arch all package
> > >
> > > (debian-arm64)# apt-cache policy x11proto-scrnsaver-dev
> > > x11proto-scrnsaver-dev:
> > > Installed: (none)
> > > Candidate: 1.2.2-1
> > > Version table:
> > > 1.2.2-1 0
> > > 500 http://people.debian.org/~wookey/bootstrap/debianrepo2/ debianstrap/main arm64 Packages
> > > 500 http://ftp.uk.debian.org/debian/ unstable/main amd64 Packages
> > > 500 http://ports.ubuntu.com/ubuntu-ports/ saucy/main arm64 Packages
> > >
> >
> > Right, and that's a problem, as having two different packages with the
> > same version is not really supported. APT differentiates packages with the
> > same version by CRC-16 hashing the fields
> > Installed-Size
> > Depends
> > Pre-Depends
> > Conflicts
> > Breaks
> > Replaces
> > in order to handle packages where those are the same APT would need to hash
> > size or SHA hash as well, but this fails for installed packages, as this
> > information is not provided in /var/lib/dpkg/status.
>
> OK. That makes sense. I see what's going on now.
>
> Which of course if why we do -B builds for other architectures and
> carefully ensure there is only one copy of the arch all packages.
>
>
> The problem is that in order to debootstrap you need all the packages in
> one repo so leaving the arch all packages in ftp.uk.debian.org means you
> can't debootstrap if you only uploaded the new-arch 'any' packages to
> the 'bootstrap' repo. It's also important to test that the arch-all
> build actually works, and not just the arch-any part so doing those
> builds and testing the results can be good.
A work around might be to reorder sources.list entries. The order of
those entries determines from which source a package is retrieved, I
believe the first match takes precedence.
>
> It's fine for apt to consider these packages to be functionally
> equivalent, but it does need to check the correct checksum on download.
> It seems to me that this can be fixed by either adding size/hash to the
> hash as you suggest(making them 'different packages', or just separately
> ensuring that the checksum for the repo/file that was downloaded is
> used. Apt knows that there is more than one repo source for this
> package, but doesn't record that there might be more than one checksum?
> The fact that it can end up choosing one checksum and another source
> does seem wrong. Perhaps the code/object structure makes it hard to fix
> this this way and your fix is the only one that makes sense?
It seems right to me in this case, because otherwise functional aspects
like dependencies could differ as well. And if APT uses the dependencies
from one source and then fetches the package from another source, but that
one has different dependencies, installing it would produce an error.
>
> > An alternative would be to change the cache-building algorithms to look
> > at SHA hashes and/or size and create different version entries in the cache
> > if they are present in both versions, but different. SHA Hashes would require
> > all repositories to use the same best checksum algorithm.
>
> I think just adding size to the hash would be cheap and easy and would
> largely solve this problem. Adding the hash would cover a few extra
> cases where the size came out the same too, but if it's difficult I'd be
> happy to have this mostly-solved, as it's a situation we normally try to
> avoid anyway.
Adding the size to the hash is not possible, as dpkg does not store the
size for installed packages. This would mean that an installed package
always has a different hash than an available package, causing APT to
go crazy (it would try to "upgrade" all installed packages...).
David or Michael probably have some more ideas.
--
Julian Andres Klode - Debian Developer, Ubuntu Member
See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.
Please do not top-post if possible.
Reply to: