Bug#1020217: IRC log from #debian-admin
Hi,
I'm copying here, for reference, an IRC log from #debian-admin:
19:38 <@jcristau> lucas: right, so the importer would indeed somehow need to learn about copying stuff to other locations/backends.
the python/web bit in practice doesn't matter, i think, because it's not used for /file/... requests, those are
serviced by apache directly
(https://salsa.debian.org/dsa-team/mirror/dsa-puppet/-/blob/production/modules/roles/templates/snapshot/snapshot.debian.org.conf.erb#L76)
19:42 <@jcristau> lucas: the way the copying of stuff to the replica happens currently is very async
(https://salsa.debian.org/snapshot-team/snapshot/-/blob/master/master/import-run#L47-L52) which means we just hope
things are eventually consistent, but there's a window where the replica would 404 for new files. that might be ok,
because in practice users probably aren't requesting the very latest mirror run, but
19:42 <@jcristau> it might be nice to somehow keep track of what's synced where and what's new, also to reduce how much time we're
spending walking directories.
21:38 < lucas> ah right, apache could just redirect to S3 or act as a proxy, or the web app could point to S3 directly
21:40 < lucas> re replica, I wonder if we shouldn't instead aim for several parallel instances of snapshot (at least 2), each living
its own life (not in a master/slave setup)
03:46 <@pabs> lucas: then you would get different snapshot timestamps?
07:47 < lucas> pabs: do we care?
07:52 <@pabs> having multiple sets of timestamps served from different servers seems confusing for end-users
08:38 < lucas> ah, my idea was more that they would be exposed as different services, with one being used for snapshot.debian.org
08:40 < lucas> the snapshot services could actually share timestamps afterwards (for example to cover a period of downtime)
15:26 <@jcristau> lucas: my idea would be to add the s3 bucket as a backend in
https://salsa.debian.org/dsa-team/mirror/cdn-fastly/-/blob/master/services/snapshot.yaml and use that for /file/
21:14 < lucas> jcristau: I'm not familiar with how CDNs work. could you tell fastly to fallback to other backends if the first one
replied with 404?
21:57 <@Mithrandir> we can do that
21:57 <@Mithrandir> it might not be wise
Reply to: