[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Bug#1020217: S3-backed snapshot implementation

Lucas Nussbaum <lucas@debian.org> writes:

> - I got the OK to host a S3-backed snapshot mirror using the Debian AWS
>   account (see thread in #1020217)

Is this s3 bucket public, or will it be?

I have been worried about the state of snapshot and I am mirroring its
data into local Git LFS.  Since snapshot.debian.org doesn't support
rsync and don't make the postgres database dumps available (so that I
can identify SHA1 objects and speed up downloads), I am using HTML web
scraping to find out what files exists to snapshot.d.o.

My goal has been to put all the Git LFS objects in a publicly-accessible
S3 bucket too.  While imports were running I didn't work on the bucket
side, and I suspect my download will take months to complete at current
speeds.  I publish Git LFS versions of archive.debian.org,
ftp.debian.org and ftp.ports.debian.org already, though, so perhaps I
could start on the bucket publishing part for them and see about adding
an incremental snapshot.d.o copy while it is still working.


Attachment: signature.asc
Description: PGP signature

Reply to: