Hi, Le 2/27/21 à 3:09 PM, Holger Levsen a écrit :
Hi, snapshot.debian.org is an awesome service for the wider free software community and especially for those working on reproducible builds. Sadly accessing *many* packages from it is limited and troublesome (see below for bug numbers), thus we (mostly Frédéric Pierret and myself) came up with the idea of setting up a partial mirror, covering only the years 2017 until now and arch:amd64 and arch:all only as well. (for a start, maybe we need 2015+2016 too and maybe we can afford to also host arm64 or some other architecture...) Background: several projects (at NYU, from Qubes, from Debian, some independent researchers) want to setup rebuilders of Debian (bullseye) rebuilding all 30000 source packages which need thousands of different snapshots and thus are regularily hit/hurt by these bugs: #977653 Please document rate limits on snapshots.debian.org #960304 snapshot.debian.org: Snapshot repo repeatedly cutting off connection, returning partial content #969906 snapshot.debian.org: error 500 internal server error after some requests via Python It's possible to work around them partly/sometimes, but it seems clear by now that using snapshot.d.o *as it is* doesn't scale for our use cases. Hence the idea of a partial snapshot mirror which is only suitable to rebuild bullseye am64 (but not previous release or other archs.) - and which can also be mirrored more easily than the whole of snapshot.d.o if the need arises. Does that sound feasable/sensible from the snapshot.d.o team perspective? The idea would be to ask Debian for money to buy four 16tb discs (one currently costs around 300€) and then have two shipped to Frédéric (for local development) and two shipped to OSUOSL.org, which probably could host them for us. (We've got an informal offer we need to formalize once/if this idea is deemed good.) misc notes: this is ment as prototype / temporary solution for the next 2-3 years to that we can continue to develop (several) tools for the verification of Reproducible Builds of amd64/bullseye and probably amd64/bookworm, but it's clear that in the long term we want solutions for other supported archs and releases after bookworm. I'd very much hope that snapshot.d.o can be scaled, just this will take time, probably a lot, and so I think it makes sense to spend <1.5k€ to have an interim solution *now*. In theory the machine offered to us informally at osuosl can take 6 sata drives... feedback very much welcome!
A quick update: with respect to the original message and the RT ticket [rt.debian.org #8547] for having a one shot exported database to find a mirroring solution, I've started working on mirroring in another way without the help of the DB. Currently, a POC is running for 'amd64' and 'unstable only. For example a little tree output: https://gist.githubusercontent.com/fepitre/27e18815712fd1df96cb72993a78c4b7/raw/dacf81d2334305a8db5e9ea338d55ef31695b950/fragment. Once I've got several timestamps synced for 'unstable', 'bullseye' ('amd64 and 'all' arches) I will properly check that I can use those local mirrors for package rebuild. Best regards, Frédéric
Description: OpenPGP digital signature