Re: The annual git/svn discussion (was: Re: Minutes of the pkg-perl BoF at DebConf 10)

[Jeremiah Foster <jeremiah@jeremiahfoster.com>]

On Aug 7, 2010, at 23:35, Tim Retout wrote:

>> Tim says he has given up on trying to get git to work. Not saying it
>> is impossible, just saying someone has to care about it.
> 
> I should add more detail of what I found last year, most of which I've
> already communicated to Jonas:
> 
> I assumed we would want one git repository per package.  Now, 1700 git
> repositories turns out to be quite difficult to make perform as
> quickly as a single svn trunk checkout.

I'm not convinced of this. In fact, I'm pretty sure that any update will be a lot faster than svn, because git just transmits the delta off the bat while svn goes through the entire repo looking into every .svn file.
> 
> Assumptions: we want to support the workflow where all packages are
> checked out at once, so that mass changes can be made.  We do not care
> so much about the initial 'clone' speed, but the equivalent of 'svn
> up' on trunk should run in a reasonable time.

Here I think we should do a benchmark test because this is in fact a fairly common scenario.
> 
> For starters, it is really slow if 'git pull' is run in serial on each
> repository (as, say, the 'mr' tool would do, last I checked?) - you
> have to parallelize the operations to get the maximum out of your
> bandwidth, updating around 20 repositories at a time.

The model of a single git repo with git submodules per package ought to make this significantly faster.

> Also, the single biggest speed improvement iirc was to share the ssh
> connections - see the '-M' option to ssh for details on 'master mode'.
> The overhead of setting up one ssh connection per package is too
> high.
> 
> I then got bogged down in trying to share objects across the
> repositories, so that I could run a single 'git pull' to fetch all
> changes.  At that point I decided it was all too much effort.
> 
> Caveats: I did this testing one year ago (so not with git 1.6), and I
> found out afterwards that I was using really very slow disks (SSDs
> rather than HDDs).  The disk speed could really affect this.

I'm confused. If SSD you mean solid state drive, that should be significantly faster.

Jeremiah