Re: wishlist script: “hegemon”: set up bare git mirror farm for all debian sources
Rough stats for “hegemon emerge” now available courtesy Gentoo, see
the bottom of the following article, last section titled "Git versus
rsync":
https://lwn.net/Articles/759467/
and here's the money quote:
Matt Turner said that he has set aside a 1GB partition for the
tree, which works fine for the roughly 600MB needed by rsync, but
not for Git. A shallow clone of the Git repository is roughly the
same (around 660MB), but each pull adds to that, so without some
kind of "auto-trimming", Git will grow quickly, Freeman said
There is also a security issue considered:
The GitHub mirror compromise has clearly led to some thinking (and
rethinking) within the project about its practices and how they
might be improved. It is not clear that there are any real
conclusions that have been reached, much less plans made, but
considering the various parts of the problem is certainly to the
good.
which leads to the obvious thought regarding the hegemon utility -
the marginal repo size increase for say a Debian source builder to
also include Gentoo's git branches for those packages which overlap
(and vice versa of course) should not be too significant, but more
importantly, if one entity's server is compromised (be it Github,
Debian or etc), then this ought be simpler to auto-detect when your
source repo has multiple git-repo upstreams to test against.
Sounds like we're talking a few TiB of disk space here, including
source checkouts and build artifacts, so nothing that cannot fit in a
10TiB spinning rust bucket…
“To hegemon,
and beyond‼”
(And really, who wants Gentoo to beat Debian to the hegemonic finish
line?)
On Sun, Jul 01, 2018 at 07:55:01PM +1000, Zenaan Harkness wrote:
> git at alioth is coming along very nicely and was a great step
> forward for Debian (thank you Ian).
>
> What would be nice is a script e.g. called say "hegemon" which could
> be used to greate a Debian git repo farm locally.
>
> Why?
>
> Well, I hold that this would be preferable to adding sources to my
> Debian mirror - these days, who wants tar balls?
>
> Seriously, what you want is a source repo, with tags for the various
> releases of a package, so you can e.g. compare the current sid
> release with stable or testing, view the Debian specific patches,
> etc.
>
> This is what "sources" used to be used for.
>
> Nowadays sources is kinda moot.
>
> I note from here https://wiki.debian.org/Alioth/Git ^B
> https://wiki.debian.org/PackagingWithGit it says "If upstream is
> using git to manage their source, the debianization repository can
> live in a branch off of the main upstream tree. Clearly this can work
> only with some upstreams, but the big upside is that the relationship
> between the debianization and the upstream code is VERY clear." which
> is immediately followed with this "Since the Debian APT repositories
> still use tarballs you still have to manage those with this setup,
> but the pristine-tar exists for that purpose."
>
> There are possibly still advantages to tarballs, I don't know, since
> I haven't used one in something like a decade or more. Consistency of
> distribution of sources? Compliance with Debian packaging guidelines?
>
> Debian sources is currently a poor man's hegemon, providing little
> more than a point in time snapshot (which admittedly 'matches'
> (hopefully) the current respective binaries).
>
> The true hegemon (maniacal laughter echoes in the background) wants
> unified git repos, for every package.
>
> I imagine that some "canonical vcs" datum would need to be collected/
> stored for each package, and hegemon would cycle through them running
> some default cmd like “git fetch”
>
> hegemon would also know to use git plugins to hungrily grab upstreams
> of other persuasions such as hg, bzr or (ughh! svn), for that
> equanimous local uniformity and hegemonic experience that we all
> crave so.
>
> Then, who would ever download 55GiBsMeDats again? Especially when a
> simple
>
> git clone /my/hegemon/p/package-source-repo /my/tmp/package-work-dir
> cd /my/tmp/package-work-dir
> git reset --hard v1.02
> deb-buildpackage --pristine-tar
>
> will do that job?
>
> THEN, finally, a weekly
>
> hegemon build-tar git-deltas ^1week
>
> could be usefully generated and distributed (and downloaded and
> applied to local hegemon repo farms, by those who would otherwise be
> updating their debian source repos) and might ultimately supplant the
> tar source distribution.
>
> …
Reply to: