Re: RFC: DEP-14: Recommended layout for Git packaging repositories

[Ron <ron@debian.org>]

On Fri, Nov 14, 2014 at 11:25:36AM +0100, Raphael Hertzog wrote:
> On Fri, 14 Nov 2014, Ron wrote:
> > > So I am a Kali Linux contributor. We use git repos to maintain all our
> > > packages and we use git-buildpackage.
> > 
> > I guess the first question there is what were the arguments put forward
> > for deciding to 'standardise' on gbp?  If there wasn't one, maybe that's
> > an argument you should have (and if there was, maybe it's one to revisit :)
> 
> 1/ It's the tool I was most familiar with
> 2/ It was rather well documented
> 3/ It seemed to be the most popular tool within Debian
> 4/ It forces the user to follow some conventions and this is good when
>    you have newcomers
> 5/ It did the job
> 
> You can try to sell me gitpkg but I'm not sure it will bring this
> discussion forward.

I'm not trying to "sell" you anything in particular here.  I'm trying
to share a little of the view from a perspective where most of the
problems you seem to be worried about are in fact either already solved
or not actually problems at all.

If your answer to "Doctor it hurts when I bang my head against the wall"
is "I know, let's change the building codes to mandate walls must be
made of nerf bricks!", there's still going to be a whole world full of
walls that are going to hurt your head.

You can't solve problems that are inherently chaotic by pretending
that they are not.  If you could, we'd have all been replaced by
very small shell scripts years ago.  Ask ubuntu how well that worked
for them ;)


If you don't want to learn about how the other tools that you don't
really know much about yet work, you're going to have a really hard
time trying to "unify" them in any actually meaningful way.



> > If you have a clearer idea now of the problems you are facing, it might
> > make properly evaluating things that avoid those problems easier.
> 
> My problems are not so much technical, my problem is having a general
> recipe that work well enough for most packages so that I have to be
> involved for the few special cases.

Have you considered starting with a solution to something a bit simpler
first?  Like, say, world peace ;?

What you're asking for there is not only a technical problem, it's a
technically Hard problem.  There are ways you might be able to make
it easier, but a naming convention really isn't one of them.

That might make the people who like to have their socks strictly sorted
by colour feel a little more at home, but it doesn't actually address
what is actually hard about this.


> Changing for another tool will not improve the situation if that other
> tool requires an evaluation of the layout of each repository before being
> able to execute the right incantation.

How can you possibly know what you want to export from it without
looking at it?  You can't even know if the repo is currently in a
coherent state suitable to be exported from if you don't look at
it and exercise some amount of judgement on what you see.


> When Kali forks a package it's usually because we need a new upstream
> version, or because we want to apply a pre-existing patch. It's not rocket
> science and it ought to be doable rather easily, whatever helper tool
> Debian uses.

Well that sounds a lot less like something you'd be pushing back to
Debian.  And not even something you particularly need a (D)VCS for,
or where you'd get a whole lot of value from one.


> git-buildpackage has not been a major obstacle for us, on the contrary.
> Thanks to git-import-dsc we can hack an any package by always following
> the same workflow.
> 
> But now if I want to go one step further and build on the existing
> repositories while still trying to keep the same level of simplicity,
> we have to go one step forward. And that's the step that I'm trying
> to push for with this DEP.

Again, I'm not arguing with the goal.  Just trying to point out that
simply formalising the actual source of many of your current problems
isn't really a way to solve them.


> > Even if everyone did stick to the conventions already discussed
> > (which in reality, they won't) there's still far too many degrees of
> > (quite necessary) freedom to really approach a "just follow these
> > three easy steps" kind of uniformity.  And even if you got close to
> > achieving that for the "debian branches", the (again necessary)
> > variability between upstream repos is going to be even greater.
> 
> I don't want 100% uniformity, I just want as much uniformity as possible
> when there's no good reason to diverge from the recommendations.

And I'm not particularly arguing against the value of conventions
where conventions are valuable.  But they aren't going to solve
the problem(s) that you seem to have convinced yourself that they
will - and they especially aren't going to solve them if you go
making up new conventions that are different to what everyone else
in the world not using gbp (ie. 99% of it) are using.

Upstream developers aren't going to read and follow your
recommendations.  If you don't adapt to being able to cope with
the infinite variations they might have reason to use, your
problems are going to remain the same problems they are for you
today.


> > If it really is the 'real' debian package, you have no problem
> > and don't need a kali tag for it anyway.  If it's a kali special,
> > then the package already ought to have a x.y.x-1kali2 type version
> > anyway.  So this part already naturally avoids conflicts too.
> 
> Using namespaces has other possible benefits: think of building Debian
> packages out of a git push. The explicit tag prefix avoids building the
> Debian version of the package when I want to build the Kali one (I might
> be the maintainer on both Debian and Kali, using the same GPG signature).

And what do you do when I package an upstream project named Kali,
where upstream tags their releases "Kali/x.y.z" ?

If you're going to trigger a build from a git push, there are so
many ways that can go horribly wrong that I don't even know where
to start.

It's fine for c-i infrastructure where you always want to test build
the HEAD of some branch, but for making actual releases??


> > > And we could also build on work in progress that has not yet been
> > > released as a source package. Right now, the only packages where we
> > > build on top of the Debian git repositories are some native packages
> > > (like debian-installer).
> > 
> > I can't really tell you how kali ought to work, but if you can solve
> > it for this case, you should be able to do the same for non-native
> > packages too.  And I would recommend that you try to, even if you
> > can't get the "uniformity" that you'd thing of as ideal.
> > 
> > What is the exact difference that makes native packages work for this
> > but others be harder?  In the case of gitpkg, this is really just
> > the difference between passing one ref and two to export it.
> 
> In the case of debian-installer, I am the one taking care of it on
> Kali and I have the required knowledge. And also we needed some unreleased
> fixes for newer kernel versions.
> 
> Still, native packages are easier because they look like any upstream repo
> with a master branch (there's no need to have multiple branches around).

Native packages never have branches for multiple derivatives or
backports to different suites?

But really, if you think non-native packages are inherently more
complicated in any significant way, then you really ought to be
*rethinking* your workflow, not trying to formalise it as a problem
that everyone else should have too.


> > > I can't afford to document all the possible ways Debian is maintaining
> > > their package but if I can write a documentation that covers the common
> > > case and if I can tell them "when you see those branches, you can follow
> > > the instructions below", then we have made some real progress.
> > 
> > I really do think that the names of the branches are actually going to
> > be the least of your worries here, unfortunately.  Even with a naming
> > scheme that's widely adopted, things just aren't going to be that sort
> > of uniform outside of (a fairly large number of) fairly small subsets.
> > 
> > You're going to need a better solution than this unfortunately, if you
> > want it to actually work.
> 
> Yes, completely agreed. But git is rather robust and smart, and even if
> we would all prefer that derivatives continue maintaining their repository
> in the same spirit that we maintained ours, I believe that until we have
> proper tools to do just that, it's not too difficult to create some
> script that allows us to fixup common problems when we fetch a Debian
> git repository so that the standard Kali-recommended workflow works
> afterwards.
> 
> I'm certainly eager to explore all those options (with you and anyone
> else interested, the scope of the suggested sprint can include such stuff
> once we have cracked the easy fixup to harmonize the helper tools)

If you genuinely believe there actually is an "easy fixup to harmonize
the helper tools" (and I don't mean to sound like I doubt that you do)
then I really do think that the absolutely most valuable thing you can
do at this point is to take a step back from what you think you know
about this and *really* spend some time learning to use them and getting
to know how they actually work.

Like ... the kind of time that Russ put into learning about init systems.

Because the differences between them aren't just "defaults to different
names for various refs".  It's that at some level they really do
fundamentally work differently to each other.  And that's even before
you start to think about the significant numbers of people and groups
who also manage their packages in git without using *any* of them.

That's why they exist as discrete projects as they do.

Bernhard didn't write git-dpm because he didn't like the naming
conventions of gbp or the flexibility of gitpkg.  I didn't write
gitpkg because I like reinventing wheels.  They both exist because
we looked at the problems in fundamentally different ways, based on
our own experiences with the problems of the packages we maintain.

They both exist as they do because they weren't just a minor tweak
to something that already existed.

So I do think you're going to have a really hard time understanding
what can and can't be usefully "harmonised" if you're coming at this
from only understanding one tool.  One that neither of us thought
was up to the job of doing the things we needed to be able to do.

> but I would still like to complete this DEP first because I'm convinced
> that it is useful nevertheless.

...  which is why I think this is approaching the real problem
completely backwards.  If you want to produce a "how to work with
gbp" DEP, I don't have a problem with that at all.  But you should
be quite clear about what it really is and call it that.

If you want to do something broader than that, I don't think you're
going to be able to do it without a much better understanding of
the other tools than you currently have.


That's what I've been trying to give you a crash course on for
gitpkg.  I'm sorry if you saw that as a sales pitch.  I'm always
happy to explain it to anyone who is interested in solving the
problems that it was written to solve, but I have no interest at
all in shoving it down people's throats if they can't see that
for themselves.  It's git.  There really *is* more than one way
to do things.  And that's an enormously good thing for software
developers who want the right tool for *their* job.


> > Really the only way this can possibly work is if all the tools that
> > you consider viable candidates to use *don't* require some special
> > structure of their own in the repo to work (or knowing the right
> > magic incantation specific to each repo for them to work correctly).
> 
> git-buildpackage doesn't have many hard requirements. git-dpm a bit
> less so IIRC.
> 
> > Otherwise, the tools that do are simply never going to work with a
> > repo that doesn't have that.  There really isn't any way around that,
> > except to replace those tools with better ones ...
> 
> Or improve them, or merge them. Which is the outcome that I'm looking
> forward in this DEP.

As I already said before, if some tool authors learn more about the
fundamental mistakes they've made, that would be awesome.  But they
aren't going to learn that from "here's how naming conventions might
work better with gbp".


> Right now teams are imposing one specific tool/workflow because there
> are so many differences even on basic things like the branch names. When
> the differences start to fade away, people look more closely at ways
> to allow everyone to use their tool of choices.

I want vim modelines and fold markers.  You want whatever emacs
calls their thing.  Someone else absolutely hates that crap in
their source code.

When you solve that problem, you'll understand exactly how to
solve the "git workflow of choice" one too! :)


> > Unlike several of them, gitpkg didn't try to be a "framework" for
> > package management.  It simply focussed on the job of exporting a
> > source package, in any valid source package format, from an existing
> > git repo.  It provides hooks to let you tweak that, and to perform
> > automated tasks after export (like shipping the package off to be
> > built), but it explicitly doesn't provide high level functions for
> > repo management tasks *prior* to export.
> 
> Why couldn't gitpkg be an option within git-buildpackage?

Again, I invite you to experiment with it and explain how you
think that might work.


> I believe it's important to provide a global solution (i.e. from
> creating/cloning the repo) and importing the history, to building
> the package and tagging the releases, etc.

Again, I invite you to experiment with it and tell me exactly what
of those things you think is missing.  After nearly a decade of use
with nobody complaining that any of those things are missing, I'll
certainly be interested if you see something that is.

You might want to clarify whether you want to build the package
in the current workspace, or with pbuilder, or with sbuild, or
with qemu, or with ...   though!


> > Whether the other tools can be fixed ...  yeah.  After all, there
> > was a reason gitpkg was born as a separate thing in the first place :)
> 
> I'm much more positive than you. At leas Guido is very welcome to enhance
> git-buildpackage with support of more workflows and I'm convinced that
> we could include everything that makes gitpkg/git-dpm interesting in
> git-buildpackage.

You still haven't convinced everyone that format 3 is the one-size
fits all solution they've always been looking for yet :)

You might have convinced yourself of this, but you'd have a much
better chance of convincing me and maybe some others if you can
show that you actually understand the *real* differences between
them rather than just insisting it's "a simple matter of renaming
some branches".  It really isn't.

It's not just a matter of "adding more workflows" to gbp either.
You can't make a fundamentally inflexible thing become flexible
just by special casing it even harder.

It's fine if you don't want to take my word for it, but if you're
not going to do that then you really do need to put in the time
to learn more about the other systems and how real people really
use them yourself.

If you don't do that, then you really are just going to yak shave
the least of the problems without getting anywhere near the real
things that might be able to be improved.

And again, you can take this as "negativity" if you like, but I
really hope you don't.  Good engineering is all about identifying
problems and trying to find real solutions to them.  I'm just
trying to point out the problems that mean a fresh coat of paint
won't fix this in the way that you seem to be hoping very hard
that it will.


> > ... less than 8 hours later, *all* of those patches had been cherry
> > picked and merged into the mainline upstream repo.
> 
> This is just a proof that storing the patches as real commits is useful.
> And that's the point of the patch-queue tag. Instead of having the patches
> only as real commits in the local repository, they get pushed to the
> public repository too under that tag (instead of appearing only within
> debian/patches/).

How is that different to "instead of having the orig tarball only
as a p-t delta in the repository, you also push the whole orig
tarball under some special tag too"?

Or did you just not understand that the "real commits" also get pushed
to the public repository too?  That is after all what makes the upstream,
developer able to cherry pick them.

I'm actually not quite sure what you're not understanding about
this at all, no matter how many times I re-read that ...

It's not about "storing the patches as real commits", the real commits
*are* the patches.  The only question is whether or not you are one
of the people who believe that also exporting them in the package
as debian/patches is a useful thing to do.  (which arguably it isn't
if 8 hours later you can upload a new package with them properly
merged upstream :)

You make the commits by "working in git like you normally would",
not by going through some convoluted dance with quilt and then
extracting them from that to commit them.  The export tools we
created are what makes it possible to have that choice without
needing to change your workflow or your repo content *at all*.

Using this you could even export a package with a debian/patches
from an ancient version in the history that existed before there
was ever a format 3, or before quilt even existed.  The kind of
package you want exported does *not* depend on having a magic
layout or tool specific configuration files in the repository.


But as I say, the best way you can really realise all this is to
just go play with it yourself.  I don't care if you switch to it
long term or not.  But if you don't understand it, and can only
imagine things about how it works based on what you know about
just one quite different tool, this is going to be a really hard
conversation to have in any sensible way.

I'm trying to explain a platypus to you, and you're saying,
"yeah, yeah.  ducks.  we have those too!" :)

  Cheers,
  Ron