Re: apt-get update hashsum mismatch prevent

[Michael Vogt <mvo@debian.org>]

On Tue, May 27, 2014 at 07:46:11PM +0200, Peter Palfrader wrote:
> On Mon, 26 May 2014, Michael Vogt wrote:
> > (1) apt fetches the InRelease file and if during that fetching the
> > server updates its indexfiles the subsequent GET for the indexfiles
> > will fail with a hashsum mismatch because the InRelease file has the
> > hashes of the previous generation of the indexfiles.
> > 
> > (2) apt fetches a new InRelease file but the new indexfiles are not
> > updated/mirrored yet. A hashsum mismatch error is found because the
> > new InRelease file hashes do not match the old indexfiles.
> > 
> > 
> > Problem (2) is of less relevance right now because AIUI our mirror
> > scripts updates in multiple steps, i.e. sync pool, sync indexes, sync
> > release. But its still worth thinking if this could be simplified.
> 
> It'd be nice if this could also work in cases where
>  a) an archive behind a given name is provided my more than one host,
>     and they are not necessarily strictly in sync.  That is, apt-get
>     updates should sill work, even if the servers' mirror runs don't
>     switch over individual files such as InRelease at the same time.
>  b) apt-get update should work if it faces an archive behind a caching
>     infrastructure that, for instance, keeps all files for at least 5
>     minutes.
> 
> In both of these instances problem (2) can easily appear.

Thanks for your feedback and sorry for my slow reply (I was
traveling).

Indeed, those two cases you outlined are worth fixing too. 

The case (a) is somewhat mitigated by apt using the same IP for the
entire apt-get update. So it should not jump to a different host if
e.g. round-robin dns is used.

A more generic solution could be to keep the hashes of the previous
generation(s) in the signed Release file. Combined with the by-hash
schema outlned for (1) this should cover the following cases:

Servers {A, B} are used by apt:
-------------------------------
A old InRelease, B old Packages: ok
A new InRelease, B new Packages: ok
A old InRelease, B new Packages: ok if by-hash is used
A new InRelease, B old Packages: ok if apt knows about the
                                 prev. hashes and can fallback to
                                 fetching them by-hash

One downside of this approach is that apt would have to "probe"
again. So in case it gets a 404 on the new hash it would have to look
for the previous hash and do a GET on that hoping to get the right
file. This is kind of "speculation" is something we would like to get
rid of in apt, i.e. only download files that we know exist from
examing the Release file and fail if its not there.

But maybe someone has a alternative idea? In any case, it seems like
implementing the hash approach is something we want for this as well.

Thanks!
 Michael