Re: RFC: DEP-14: Recommended layout for Git packaging repositories
On Fri, 14 Nov 2014, Ron wrote:
> > So I am a Kali Linux contributor. We use git repos to maintain all our
> > packages and we use git-buildpackage.
>
> I guess the first question there is what were the arguments put forward
> for deciding to 'standardise' on gbp? If there wasn't one, maybe that's
> an argument you should have (and if there was, maybe it's one to revisit :)
1/ It's the tool I was most familiar with
2/ It was rather well documented
3/ It seemed to be the most popular tool within Debian
4/ It forces the user to follow some conventions and this is good when
you have newcomers
5/ It did the job
You can try to sell me gitpkg but I'm not sure it will bring this
discussion forward.
> If you have a clearer idea now of the problems you are facing, it might
> make properly evaluating things that avoid those problems easier.
My problems are not so much technical, my problem is having a general
recipe that work well enough for most packages so that I have to be
involved for the few special cases.
Changing for another tool will not improve the situation if that other
tool requires an evaluation of the layout of each repository before being
able to execute the right incantation.
When Kali forks a package it's usually because we need a new upstream
version, or because we want to apply a pre-existing patch. It's not rocket
science and it ought to be doable rather easily, whatever helper tool
Debian uses.
git-buildpackage has not been a major obstacle for us, on the contrary.
Thanks to git-import-dsc we can hack an any package by always following
the same workflow.
But now if I want to go one step further and build on the existing
repositories while still trying to keep the same level of simplicity,
we have to go one step forward. And that's the step that I'm trying
to push for with this DEP.
> Even if everyone did stick to the conventions already discussed
> (which in reality, they won't) there's still far too many degrees of
> (quite necessary) freedom to really approach a "just follow these
> three easy steps" kind of uniformity. And even if you got close to
> achieving that for the "debian branches", the (again necessary)
> variability between upstream repos is going to be even greater.
I don't want 100% uniformity, I just want as much uniformity as possible
when there's no good reason to diverge from the recommendations.
For the other cases, we can stick to the current model (building on top of
imported sources packages). It's not perfect but it's a lot better
if most of our forked packages are based on Debian's git repo.
> I think at some point you're going to have to rely on real human
> intelligence to be able to look at a repo and form their own
> understanding of its structure. Most really aren't all that
> complicated (however much they vary between each other), and in
> the worst case you can always actually ask the 'upstream'
> maintainer to explain anything that is unclear.
Kali contributors are humans with intelligence but I'd rather
not require too much of their intelligence when it comes to trivial
changes like adding a patch or importing a new upstream version.
Also when you're 2-3 persons maintaining 300+ packages, you really
don't want to have to remember what's special about each package.
This is inavoidable for some packages (say "linux") but for the vast
majority one should not need to ask himself too many questions.
> git is already going to namespace them so when you add the remotes
> the branch refs will be (for remotes named upstream, debian, kali):
I know all this, but this requires again a deeper knowledge of git's
features whereas "git checkout <branch>" will create a new local branch
based on the remote branch of the same name. If you have <branch> in
multiple remotes, there is ambiguity.
> If it really is the 'real' debian package, you have no problem
> and don't need a kali tag for it anyway. If it's a kali special,
> then the package already ought to have a x.y.x-1kali2 type version
> anyway. So this part already naturally avoids conflicts too.
Using namespaces has other possible benefits: think of building Debian
packages out of a git push. The explicit tag prefix avoids building the
Debian version of the package when I want to build the Kali one (I might
be the maintainer on both Debian and Kali, using the same GPG signature).
> > And we could also build on work in progress that has not yet been
> > released as a source package. Right now, the only packages where we
> > build on top of the Debian git repositories are some native packages
> > (like debian-installer).
>
> I can't really tell you how kali ought to work, but if you can solve
> it for this case, you should be able to do the same for non-native
> packages too. And I would recommend that you try to, even if you
> can't get the "uniformity" that you'd thing of as ideal.
>
> What is the exact difference that makes native packages work for this
> but others be harder? In the case of gitpkg, this is really just
> the difference between passing one ref and two to export it.
In the case of debian-installer, I am the one taking care of it on
Kali and I have the required knowledge. And also we needed some unreleased
fixes for newer kernel versions.
Still, native packages are easier because they look like any upstream repo
with a master branch (there's no need to have multiple branches around).
And they don't pose any problem with requiring an .orig.tar.gz: the
current instructions I gave to other Kali contributors include enabling
the --pristine-tar flag to git-bp so they never have to download the
.orig.tar.gz.
Now think of the case of importing a new upstream version on a git
repository: you need to know whether the packaging branch is based on
the upstream git (if yes you probably need to find that one and add it as
a remote) or if tarballs are imported (in that case you need to know on
which branch it must be imported), it's immediately more complicated.
If you consider the case of adding a patch, it gets worse. While you can
rely on quilt with a freshly imported "3.0 (quilt)" source package, that's
no longer the case with some of the git helper tools (i.e. those where
patches are applied but have no .pc).
I know that only a tiny part of this will be improved by the current DEP,
but it's a start and an important one IMO.
> > I can't afford to document all the possible ways Debian is maintaining
> > their package but if I can write a documentation that covers the common
> > case and if I can tell them "when you see those branches, you can follow
> > the instructions below", then we have made some real progress.
>
> I really do think that the names of the branches are actually going to
> be the least of your worries here, unfortunately. Even with a naming
> scheme that's widely adopted, things just aren't going to be that sort
> of uniform outside of (a fairly large number of) fairly small subsets.
>
> You're going to need a better solution than this unfortunately, if you
> want it to actually work.
Yes, completely agreed. But git is rather robust and smart, and even if
we would all prefer that derivatives continue maintaining their repository
in the same spirit that we maintained ours, I believe that until we have
proper tools to do just that, it's not too difficult to create some
script that allows us to fixup common problems when we fetch a Debian
git repository so that the standard Kali-recommended workflow works
afterwards.
I'm certainly eager to explore all those options (with you and anyone
else interested, the scope of the suggested sprint can include such stuff
once we have cracked the easy fixup to harmonize the helper tools)
but I would still like to complete this DEP first because I'm convinced
that it is useful nevertheless.
> Really the only way this can possibly work is if all the tools that
> you consider viable candidates to use *don't* require some special
> structure of their own in the repo to work (or knowing the right
> magic incantation specific to each repo for them to work correctly).
git-buildpackage doesn't have many hard requirements. git-dpm a bit
less so IIRC.
> Otherwise, the tools that do are simply never going to work with a
> repo that doesn't have that. There really isn't any way around that,
> except to replace those tools with better ones ...
Or improve them, or merge them. Which is the outcome that I'm looking
forward in this DEP.
Right now teams are imposing one specific tool/workflow because there
are so many differences even on basic things like the branch names. When
the differences start to fade away, people look more closely at ways
to allow everyone to use their tool of choices.
It's a process, it takes time, but we need to start it somewhere.
This DEP + a sprint looks like a good way to go forward to me.
> Unlike several of them, gitpkg didn't try to be a "framework" for
> package management. It simply focussed on the job of exporting a
> source package, in any valid source package format, from an existing
> git repo. It provides hooks to let you tweak that, and to perform
> automated tasks after export (like shipping the package off to be
> built), but it explicitly doesn't provide high level functions for
> repo management tasks *prior* to export.
Why couldn't gitpkg be an option within git-buildpackage?
I believe it's important to provide a global solution (i.e. from
creating/cloning the repo) and importing the history, to building
the package and tagging the releases, etc.
> Whether the other tools can be fixed ... yeah. After all, there
> was a reason gitpkg was born as a separate thing in the first place :)
I'm much more positive than you. At leas Guido is very welcome to enhance
git-buildpackage with support of more workflows and I'm convinced that
we could include everything that makes gitpkg/git-dpm interesting in
git-buildpackage.
Unfortunately Guido is lacking time lately but we should be able to
work together.
> ... less than 8 hours later, *all* of those patches had been cherry
> picked and merged into the mainline upstream repo.
This is just a proof that storing the patches as real commits is useful.
And that's the point of the patch-queue tag. Instead of having the patches
only as real commits in the local repository, they get pushed to the
public repository too under that tag (instead of appearing only within
debian/patches/).
I don't think your experience contradicts anything here :-)
Cheers,
--
Raphaël Hertzog ◈ Debian Developer
Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/
Reply to: