[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: partial snapshot mirror amd64/bullseye/bookworm


Le 2/27/21 à 3:09 PM, Holger Levsen a écrit :

snapshot.debian.org is an awesome service for the wider free software community
and especially for those working on reproducible builds. Sadly accessing *many*
packages from it is limited and troublesome (see below for bug numbers), thus
we (mostly Frédéric Pierret and myself) came up with the idea of setting up a
partial mirror, covering only the years 2017 until now and arch:amd64 and arch:all
only as well. (for a start, maybe we need 2015+2016 too and maybe we can afford
to also host arm64 or some other architecture...)

Background: several projects (at NYU, from Qubes, from Debian, some independent
researchers) want to setup rebuilders of Debian (bullseye) rebuilding all 30000
source packages which need thousands of different snapshots and thus are regularily
hit/hurt by these bugs:

#977653 Please document rate limits on snapshots.debian.org
#960304 snapshot.debian.org: Snapshot repo repeatedly cutting off connection, returning partial content
#969906 snapshot.debian.org: error 500 internal server error after some requests via Python

It's possible to work around them partly/sometimes, but it seems clear by
now that using snapshot.d.o *as it is* doesn't scale for our use cases. Hence
the idea of a partial snapshot mirror which is only suitable to rebuild bullseye
am64 (but not previous release or other archs.) - and which can also be mirrored
more easily than the whole of snapshot.d.o if the need arises.

Does that sound feasable/sensible from the snapshot.d.o team perspective?

The idea would be to ask Debian for money to buy four 16tb discs (one currently
costs around 300€) and then have two shipped to Frédéric (for local development)
and two shipped to OSUOSL.org, which probably could host them for us. (We've got
an informal offer we need to formalize once/if this idea is deemed good.)

misc notes:

this is ment as prototype / temporary solution for the next 2-3 years to that we can
continue to develop (several) tools for the verification of Reproducible Builds of
amd64/bullseye and probably amd64/bookworm, but it's clear that in the long
term we want solutions for other supported archs and releases after bookworm.

I'd very much hope that snapshot.d.o can be scaled, just this will take time, probably
a lot, and so I think it makes sense to spend <1.5k€ to have an interim solution *now*.

In theory the machine offered to us informally at osuosl can take 6 sata drives...

feedback very much welcome!

A quick update: with respect to the original message and the RT ticket [rt.debian.org #8547] for having a one shot exported database to find a mirroring solution, I've started working on mirroring in another way without the help of the DB. Currently, a POC is running for 'amd64' and 'unstable only. For example a little tree output: https://gist.githubusercontent.com/fepitre/27e18815712fd1df96cb72993a78c4b7/raw/dacf81d2334305a8db5e9ea338d55ef31695b950/fragment.

Once I've got several timestamps synced for 'unstable', 'bullseye' ('amd64 and 'all' arches) I will properly check that I can use those local mirrors for package rebuild.

Best regards,

Attachment: OpenPGP_signature
Description: OpenPGP digital signature

Reply to: