Re: GIT for pdiff generation
On Sun, 27 Mar 2011, Joerg Jaspert wrote:
> Right now the source contents of unstable has, unpacked, 220MB. (Packed
> gzip its 28MB, while the binary contents per have each have 18MB
> packed).
That should not be a problem in any non-joke box. Unless you'll run it
in a memory-constrained vm or something.
> Lets add a safety margin: 350MB is a good guess for the largest.
> A packages file nearly doesnt count compared to them, unpacked its just
> some 34mb
I.e. something very easy to keep in RAM on a "server class" or "desktop
class" box.
> > There is an alternative: git can rewrite the entire history
> > (invalidating all commit IDs from the start pointing up to all the
> > branch heads in the process). You can use that facility to drop old
> > commits. Given the indented use, where you don't seem to need the
> > commit ids to be constant across runs and you will rewrite the history
> > of the entire repo at once and drop everything that was not rewritten,
> > this is likely the less ugly way of doing what you want. Refer to git
> > filter-branch.
>
> Its the one and only thing I ever seen where "history rewrite" is
> actually something one wants to do.
Indeed.
> > Other than that, git loads entire objects to memory to manipulate them,
> > which AFAIK CAN cause problems in datasets with very large files (the
> > problem is not usually the size of the repository, but rather the size
> > of the largest object). You probably want to test your use case with
> > several worst-case files AND a large safety margin to ensure it won't
> > break on us anytime soon, using something to track git memory usage.
>
> Well, yes.
At the sizes you explained now (I thought it would deal with objects 7GB
in size, not 7GB worth of objects at most 0.5GB in size), it should not
be a problem in any box with a reasonable ammount of free RAM and vm
space (say, 1GB).
> Some NM:
> > FTBFS=Fails to Build from Start
> Err, yes? How do you start in the middle?
You screw up debian/rules clean, and try two builds in sequence ;-)
--
"One disk to rule them all, One disk to find them. One disk to bring
them all and in the darkness grind them. In the Land of Redmond
where the shadows lie." -- The Silicon Valley Tarot
Henrique Holschuh
Reply to: