Basics of packaging with the new workflow
[OK this took WAY longer than expected. I am stopping here, but I would
really appreciate if others could help with comments, extra examples, etc.]
In the past couple of years, I have informally helped a few different
people adopt the new workflow[1], but that does not scale well.
So, I am putting some of my ideas in a big email to try to document this
better, to discuss some problems with the new system, and hopefully to
help more people switch to this workflow.
I will put notes prefixed with [TEAM], which would really benefit from
wider input.
[1] https://go-team.pages.debian.net/workflow-changes.html
Emanuel & Tong: excuse me for adding some extra stuff here, hopefully it
does not confuse you much. I would recommend you follow my suggestions
where this text deviates from the published doc, because it will make
things easier for you, and we can easily change that later if the team
decides something else.
===
Basics of packaging with the new workflow
=========================================
I am writing this to complement the new workflow page published in [1].
So, please read that first, and hopefully this text will get you up and
running quickly.
Note that there are some fuzzy things there, and there are also problems
and I and others have found after a few years of using this workflow.
Basic concepts
--------------
* we keep unchanged upstream git history in our git repo. Debian
packaging is a descendant of that history, merging at every new upstream
release.
* we don't use upstream tarballs or pristine-tar, those are created from
git tags.
[TEAM]
Michael noted some issues sometimes with tars created directly from git
in the document, but this has never happened to me with packages that
were not using pristine-tar. Has anybody experienced this?
[/TEAM]
* many times we need to do repackaging of upstream sources: removing
vendored libraries, upstream "debian/" directory, or non-dfsg material.
We also do that in branches descending from upstream history.
Tags
----
Tags are very important for making things simpler, as many of our tools
will use them automatically.
If we are packaging version 1.33 of a project, we should have these tags:
* upstream/1.33 must point to the exact commit in the upstream
repository we are packaging.
* upstream/1.33+ds (or +dfsg): if we need to repackage, this must point
to the commit after all the repackaging changes are applied, and becomes
the relevant tag for tools. The tag without the "+ds+ suffix becomes
irrelevant, but is good for documentation.
* debian/1.33-1 (or debian/1.33+ds-1): after all the packaging work is
done, and only after the resulting package is uploaded to the archive,
we use "debcommit -r" to generate this tag automatically.
Other tags must NOT be pushed to salsa, so be careful not to use "git
push --tags" unless you have pruned other tags first. The git config
suggested at the end helps achieve this.
Branches
--------
The usual branch structure (and hierarchy) is as follows:
- upstream
+- unvendored (also named repackaged, dfsg, etc)
+- debian/sid
+- debian/<distribution target>
Upstream branches
-----------------
The "upstream" branch follows upstream release tags or commit of the
snapshot to be packaged. Ideally there should be no commits from us.
[TEAM: problem]
In the last year we have had many instances where -with uptreams that
*do* releases- the release tags couldn't be fast-forwarded, and keeping
the "upstream" branch updated required force-pushing (which is very
complicated, and requires extra privileges in salsa) or adding merge
commits, which defeat the purpose of having a pristine upstream branch.
[TEAM: my proposal for consideration of the team]
I can think of 3 solutions for this problem:
* Drop the upstream branch altogether for upstreams that do release
tags, and rely on tags as explained in the previous section.
* Un-protect the upstream branches in Salsa and always force-fetch and
force-push.
* Only use the upstream branch locally and force-fetch as needed, only
push tags to Salsa.
For this introduction, I will ignore this problem, as it only affects a
small number of packages.
[/TEAM]
The "unvendored"/"repackaged"/etc branch contains the changes to the
upstream codebase before it is packaged.
On each new upstream release or snapshot, the upstream branch is merged
onto it, and any new modifications applied.
[TEAM: problem]
Another issue is the naming inconsistency of the repackaged branches,
and the impossibility of using upstream/foo branches when there is
already a branch called "upstream".
[TEAM: proposal]
One solution that came in discussions in #debian-prometheus is to
instead use these branch names, which allow for future new uses:
upstream/master
upstream/repackaged
etc.
Of course, "upstream/master" might be absent if we also go with the
previous proposal.
[/TEAM]
Packaging branches
------------------
These follow DEP-14 format, which is the used to automatically configure
sbuild, pbuilder , or cowbuilder so that the build is performed in the
right environment.
Packaging work happens in the "debian/<distribution>" branch, being
"debian/sid" the branch where most of the work happens.
When we need to backport packages, or produce updates to stable
releases, we use branches named like "debian/buster-backports", which
must branch off the relevant commit in "debian/sid" history.
Tooling configuration
---------------------
To make things more uniform and simple, the "debian/gbp.conf" file
should define the parameters needed for the building of the package with
no extra fuzz. This file will differ in the "debian-branch" parameter
for each packaging branch, and the "upstream-branch" will vary depending
on the package.
Normally, it should just contain this:
#####
[DEFAULT]
debian-branch = debian/sid
debian-tag = debian/%(version)s
#upstream-branch = upstream/repackaged
upstream-branch = upstream/master
upstream-tag = upstream/%(version)s
[buildpackage]
dist = DEP14
#####
Any local configuration, such as which builder to use, paths for
resulting files, etc. should be placed in $HOME/.gbp.conf.
Changelog creation
------------------
Normally, you should format your commit messages so they can be used
unchanged to generate the debian/changelog file, and never modify the
debian/changelog direclty when doing other changes to avoid merge conflicts.
There are also some tags you can add to modify the automatic generation,
see the gbp-dch man page for details.
When you start working on a new Upstream release, you need to create a
new entry in the changelog with the right version before you can attempt
building, as gbp will use that to determine the tag where the upstream
source can be found. You can leave this uncommitted until the packaging
is ready:
$ dch -v 1.33+ds-1 "New upstream release."
To fill the changelog with all the commit messages, use:
$ gbp dch debian/
This will only take into account commits since the last time the
changelog was modified in git unless you pass the "--since" option.
Recommended local configurations
--------------------------------
This is not normative, as it only affects your local environment, but
makes things easier.
Environment variables: $HOME/.bashrc or equivalent:
===
export DEBEMAIL=tina@debian.org
export DEBFULLNAME="Martina Ferrari"
===
Devscripts configuration: $HOME/.devscripts
===
DEBCHANGE_RELEASE_HEURISTIC=changelog
DEBCOMMIT_SIGN_TAGS=yes
DEBSIGN_KEYID=<your GPG key id>
DEBUILD_DPKG_BUILDPACKAGE_OPTS="-i -ICVS -I.svn"
USCAN_DOWNLOAD=no
USCAN_VERBOSE=yes
===
Git-buildpackage configuration: $HOME/.gbp.conf
===
[DEFAULT]
pristine-tar = False
sign-tags = True
color = auto
[buildpackage]
export = WC
ignore-new = True
notify = False
export-dir = ../build-area/
# Make sure source-only packages are generated too (options for pbuilder and
# sbuilder).
#builder = sbuild --source-only-changes -s -v -A
pbuilder = True
pbuilder-options = --source-only-changes
# Run a full lintian check and sign both the source-only as well as the
# arch-specific changes files.
postbuild = echo Running lintian and debsign: && \
cd "$GBP_BUILD_DIR" && \
lintian -I --pedantic --no-tag-display-limit && \
debsign --debs-dir `dirname "$GBP_CHANGES_FILE"` && \
debsign --debs-dir `dirname "$GBP_CHANGES_FILE"` -S --no-re-sign
[dch]
full = True
meta = True
customizations = /usr/share/doc/git-buildpackage/examples/wrap_cl.py
[clone]
repo-user = DEBIAN
repo-email = DEBIAN
debian-branch = debian/sid
# Rename remote to "debian" and set-up automatic fetch & push rules.
postclone = git remote rename origin debian && \
git config --add remote.debian.push 'refs/heads/debian/*' && \
git config --add remote.debian.push 'refs/heads/upstream' && \
git config --add remote.debian.push 'refs/tags/debian/*' && \
git config --add remote.debian.push 'refs/tags/upstream/*' && \
git config --add remote.debian.fetch 'refs/tags/*:refs/tags/*'
===
Workflow examples
-----------------
These examples assume you are using the recommended configurations. I am
not yet using the experimental new features of dh-make-golang, which I
only learn today they existed :)
1. Create a repository from scratch.
# Use dh-make-golang to create a basic template (but we will discard the
git repo).
$ dh-make-golang make -dep14 -wrap-and-sort at -type library \
golang.org/x/arch
$ mv golang-golang-x-arch/debian generated-debian
$ rm -rf golang-golang-x-arch
# Create salsa project.
$ PKG=new-workflow-demo
$ dh-make-golang create-salsa-project $PKG
# Set up local git repository, and add configuration normally added by
gbp clone hooks.
$ git clone -o debian \
git@salsa.debian.org:go-team/packages/$PKG.git
$ git config --add remote.debian.push 'refs/heads/debian/*'
$ git config --add remote.debian.push 'refs/heads/upstream/*'
$ git config --add remote.debian.push 'refs/tags/debian/*'
$ git config --add remote.debian.push 'refs/tags/upstream/*'
$ git config --add remote.debian.fetch 'refs/tags/*:refs/tags/*'
# Add upstream remote, using info extracted by dh-make-golang
$ SRC=$(sed -n 's/Source:\s*\(\S\+\)/\1/p' \
../generated-debian/copyright)
$ git remote add --fetch --no-tags upstream $SRC
# Decide the commit where the packaging takes place: either a snapshot
in upstream's master branch, or a specific tag.
# Use git-log and git-ls-remote to find them.
$ git ls-remote --tags upstream
$ git log remotes/upstream/master
# Define variables based on those findings.
$ COMMIT=368ea8f
$ VERSION=0.0~git20191126.368ea8f
$ GIT_VERSION=$(echo $VERSION | sed 's/[~:]/_/g')
# Create upstream/master at that commit and tag.
$ git checkout -b upstream/master $COMMIT
$ git tag upstream/$GIT_VERSION
# If needed, do repackaging, and tag the new source.
$ git checkout -b upstream/repackaged refs/heads/upstream/master
$ git rm -rf vendor
$ git commit -m "Remove vendor/ directory."
$ git tag upstream/${GIT_VERSION}+ds
# Create the packaging branch (either from upstream/master or
upstream/repackaged).
$ git checkout -b debian/sid
# Push initial branches to salsa; order is important, as it determines
the default branch.
$ git push --set-upstream debian debian/sid
# And all the others, according to the push configuration.
$ git push --set-upstream debian
# Use auto-generated packaging as starting point.
$ mv ../generated-debian/ debian/
$ git add debian/
$ git commit -m "Initial packaging prepared by dh-make-golang."
$ git push debian
And now you are ready to go. I actually ran these commands, and you can
see the result in
https://salsa.debian.org/go-team/packages/new-workflow-demo/
2. Clone an existing repository from salsa.
# The postclone hook in gbp-clone configures the debian remote.
$ PKG=prometheus-node-exporter
$ gbp clone git@salsa.debian.org:go-team/packages/$PKG
# Add the upstream remote, fetch commits, but ignore tags.
$ SRC=$(sed -n 's/Source:\s*\(\S\+\)/\1/p' debian/copyright)
$ git remote add --fetch --no-tags upstream $SRC
Reply to: