Re: Basics of packaging with the new workflow
Oh thank you Martina very much for such detailed instructions. I'll be
packing a new package soon, and will follow this when trying.
And I'd like to especially thank you for trying it out -- actually ran
those commands through and through.
Very appreciated.
Thanks again
On Fri, Feb 21, 2020 at 4:15 AM Martina Ferrari <tina@debian.org> wrote:
>
> [OK this took WAY longer than expected. I am stopping here, but I would
> really appreciate if others could help with comments, extra examples, etc.]
>
> In the past couple of years, I have informally helped a few different
> people adopt the new workflow[1], but that does not scale well.
>
> So, I am putting some of my ideas in a big email to try to document this
> better, to discuss some problems with the new system, and hopefully to
> help more people switch to this workflow.
>
> I will put notes prefixed with [TEAM], which would really benefit from
> wider input.
>
> [1] https://go-team.pages.debian.net/workflow-changes.html
>
> Emanuel & Tong: excuse me for adding some extra stuff here, hopefully it
> does not confuse you much. I would recommend you follow my suggestions
> where this text deviates from the published doc, because it will make
> things easier for you, and we can easily change that later if the team
> decides something else.
>
> ===
>
> Basics of packaging with the new workflow
> =========================================
>
> I am writing this to complement the new workflow page published in [1].
> So, please read that first, and hopefully this text will get you up and
> running quickly.
>
> Note that there are some fuzzy things there, and there are also problems
> and I and others have found after a few years of using this workflow.
>
>
>
> Basic concepts
> --------------
>
> * we keep unchanged upstream git history in our git repo. Debian
> packaging is a descendant of that history, merging at every new upstream
> release.
>
> * we don't use upstream tarballs or pristine-tar, those are created from
> git tags.
>
> [TEAM]
> Michael noted some issues sometimes with tars created directly from git
> in the document, but this has never happened to me with packages that
> were not using pristine-tar. Has anybody experienced this?
> [/TEAM]
>
> * many times we need to do repackaging of upstream sources: removing
> vendored libraries, upstream "debian/" directory, or non-dfsg material.
> We also do that in branches descending from upstream history.
>
>
>
> Tags
> ----
>
> Tags are very important for making things simpler, as many of our tools
> will use them automatically.
>
> If we are packaging version 1.33 of a project, we should have these tags:
>
> * upstream/1.33 must point to the exact commit in the upstream
> repository we are packaging.
>
> * upstream/1.33+ds (or +dfsg): if we need to repackage, this must point
> to the commit after all the repackaging changes are applied, and becomes
> the relevant tag for tools. The tag without the "+ds+ suffix becomes
> irrelevant, but is good for documentation.
>
> * debian/1.33-1 (or debian/1.33+ds-1): after all the packaging work is
> done, and only after the resulting package is uploaded to the archive,
> we use "debcommit -r" to generate this tag automatically.
>
> Other tags must NOT be pushed to salsa, so be careful not to use "git
> push --tags" unless you have pruned other tags first. The git config
> suggested at the end helps achieve this.
>
>
> Branches
> --------
>
> The usual branch structure (and hierarchy) is as follows:
>
> - upstream
> +- unvendored (also named repackaged, dfsg, etc)
> +- debian/sid
> +- debian/<distribution target>
>
>
> Upstream branches
> -----------------
>
> The "upstream" branch follows upstream release tags or commit of the
> snapshot to be packaged. Ideally there should be no commits from us.
>
> [TEAM: problem]
>
> In the last year we have had many instances where -with uptreams that
> *do* releases- the release tags couldn't be fast-forwarded, and keeping
> the "upstream" branch updated required force-pushing (which is very
> complicated, and requires extra privileges in salsa) or adding merge
> commits, which defeat the purpose of having a pristine upstream branch.
>
> [TEAM: my proposal for consideration of the team]
>
> I can think of 3 solutions for this problem:
>
> * Drop the upstream branch altogether for upstreams that do release
> tags, and rely on tags as explained in the previous section.
>
> * Un-protect the upstream branches in Salsa and always force-fetch and
> force-push.
>
> * Only use the upstream branch locally and force-fetch as needed, only
> push tags to Salsa.
>
> For this introduction, I will ignore this problem, as it only affects a
> small number of packages.
>
> [/TEAM]
>
> The "unvendored"/"repackaged"/etc branch contains the changes to the
> upstream codebase before it is packaged.
>
> On each new upstream release or snapshot, the upstream branch is merged
> onto it, and any new modifications applied.
>
> [TEAM: problem]
>
> Another issue is the naming inconsistency of the repackaged branches,
> and the impossibility of using upstream/foo branches when there is
> already a branch called "upstream".
>
> [TEAM: proposal]
>
> One solution that came in discussions in #debian-prometheus is to
> instead use these branch names, which allow for future new uses:
>
> upstream/master
> upstream/repackaged
> etc.
>
> Of course, "upstream/master" might be absent if we also go with the
> previous proposal.
>
> [/TEAM]
>
>
> Packaging branches
> ------------------
>
> These follow DEP-14 format, which is the used to automatically configure
> sbuild, pbuilder , or cowbuilder so that the build is performed in the
> right environment.
>
> Packaging work happens in the "debian/<distribution>" branch, being
> "debian/sid" the branch where most of the work happens.
>
> When we need to backport packages, or produce updates to stable
> releases, we use branches named like "debian/buster-backports", which
> must branch off the relevant commit in "debian/sid" history.
>
>
> Tooling configuration
> ---------------------
>
> To make things more uniform and simple, the "debian/gbp.conf" file
> should define the parameters needed for the building of the package with
> no extra fuzz. This file will differ in the "debian-branch" parameter
> for each packaging branch, and the "upstream-branch" will vary depending
> on the package.
>
> Normally, it should just contain this:
>
> #####
> [DEFAULT]
> debian-branch = debian/sid
> debian-tag = debian/%(version)s
> #upstream-branch = upstream/repackaged
> upstream-branch = upstream/master
> upstream-tag = upstream/%(version)s
>
> [buildpackage]
> dist = DEP14
> #####
>
> Any local configuration, such as which builder to use, paths for
> resulting files, etc. should be placed in $HOME/.gbp.conf.
>
>
> Changelog creation
> ------------------
>
> Normally, you should format your commit messages so they can be used
> unchanged to generate the debian/changelog file, and never modify the
> debian/changelog direclty when doing other changes to avoid merge conflicts.
>
> There are also some tags you can add to modify the automatic generation,
> see the gbp-dch man page for details.
>
> When you start working on a new Upstream release, you need to create a
> new entry in the changelog with the right version before you can attempt
> building, as gbp will use that to determine the tag where the upstream
> source can be found. You can leave this uncommitted until the packaging
> is ready:
>
> $ dch -v 1.33+ds-1 "New upstream release."
>
> To fill the changelog with all the commit messages, use:
>
> $ gbp dch debian/
>
> This will only take into account commits since the last time the
> changelog was modified in git unless you pass the "--since" option.
>
>
> Recommended local configurations
> --------------------------------
>
> This is not normative, as it only affects your local environment, but
> makes things easier.
>
> Environment variables: $HOME/.bashrc or equivalent:
>
> ===
> export DEBEMAIL=tina@debian.org
> export DEBFULLNAME="Martina Ferrari"
> ===
>
> Devscripts configuration: $HOME/.devscripts
>
> ===
> DEBCHANGE_RELEASE_HEURISTIC=changelog
> DEBCOMMIT_SIGN_TAGS=yes
> DEBSIGN_KEYID=<your GPG key id>
> DEBUILD_DPKG_BUILDPACKAGE_OPTS="-i -ICVS -I.svn"
> USCAN_DOWNLOAD=no
> USCAN_VERBOSE=yes
> ===
>
> Git-buildpackage configuration: $HOME/.gbp.conf
>
> ===
> [DEFAULT]
> pristine-tar = False
> sign-tags = True
> color = auto
>
> [buildpackage]
> export = WC
> ignore-new = True
> notify = False
> export-dir = ../build-area/
>
> # Make sure source-only packages are generated too (options for pbuilder and
> # sbuilder).
> #builder = sbuild --source-only-changes -s -v -A
> pbuilder = True
> pbuilder-options = --source-only-changes
>
> # Run a full lintian check and sign both the source-only as well as the
> # arch-specific changes files.
> postbuild = echo Running lintian and debsign: && \
> cd "$GBP_BUILD_DIR" && \
> lintian -I --pedantic --no-tag-display-limit && \
> debsign --debs-dir `dirname "$GBP_CHANGES_FILE"` && \
> debsign --debs-dir `dirname "$GBP_CHANGES_FILE"` -S --no-re-sign
>
> [dch]
> full = True
> meta = True
> customizations = /usr/share/doc/git-buildpackage/examples/wrap_cl.py
>
> [clone]
> repo-user = DEBIAN
> repo-email = DEBIAN
> debian-branch = debian/sid
> # Rename remote to "debian" and set-up automatic fetch & push rules.
> postclone = git remote rename origin debian && \
> git config --add remote.debian.push 'refs/heads/debian/*' && \
> git config --add remote.debian.push 'refs/heads/upstream' && \
> git config --add remote.debian.push 'refs/tags/debian/*' && \
> git config --add remote.debian.push 'refs/tags/upstream/*' && \
> git config --add remote.debian.fetch 'refs/tags/*:refs/tags/*'
> ===
>
>
> Workflow examples
> -----------------
>
> These examples assume you are using the recommended configurations. I am
> not yet using the experimental new features of dh-make-golang, which I
> only learn today they existed :)
>
> 1. Create a repository from scratch.
>
> # Use dh-make-golang to create a basic template (but we will discard the
> git repo).
>
> $ dh-make-golang make -dep14 -wrap-and-sort at -type library \
> golang.org/x/arch
> $ mv golang-golang-x-arch/debian generated-debian
> $ rm -rf golang-golang-x-arch
>
> # Create salsa project.
>
> $ PKG=new-workflow-demo
> $ dh-make-golang create-salsa-project $PKG
>
> # Set up local git repository, and add configuration normally added by
> gbp clone hooks.
>
> $ git clone -o debian \
> git@salsa.debian.org:go-team/packages/$PKG.git
> $ git config --add remote.debian.push 'refs/heads/debian/*'
> $ git config --add remote.debian.push 'refs/heads/upstream/*'
> $ git config --add remote.debian.push 'refs/tags/debian/*'
> $ git config --add remote.debian.push 'refs/tags/upstream/*'
> $ git config --add remote.debian.fetch 'refs/tags/*:refs/tags/*'
>
> # Add upstream remote, using info extracted by dh-make-golang
>
> $ SRC=$(sed -n 's/Source:\s*\(\S\+\)/\1/p' \
> ../generated-debian/copyright)
> $ git remote add --fetch --no-tags upstream $SRC
>
> # Decide the commit where the packaging takes place: either a snapshot
> in upstream's master branch, or a specific tag.
> # Use git-log and git-ls-remote to find them.
>
> $ git ls-remote --tags upstream
> $ git log remotes/upstream/master
>
> # Define variables based on those findings.
>
> $ COMMIT=368ea8f
> $ VERSION=0.0~git20191126.368ea8f
> $ GIT_VERSION=$(echo $VERSION | sed 's/[~:]/_/g')
>
> # Create upstream/master at that commit and tag.
>
> $ git checkout -b upstream/master $COMMIT
> $ git tag upstream/$GIT_VERSION
>
> # If needed, do repackaging, and tag the new source.
>
> $ git checkout -b upstream/repackaged refs/heads/upstream/master
> $ git rm -rf vendor
> $ git commit -m "Remove vendor/ directory."
> $ git tag upstream/${GIT_VERSION}+ds
>
> # Create the packaging branch (either from upstream/master or
> upstream/repackaged).
>
> $ git checkout -b debian/sid
>
> # Push initial branches to salsa; order is important, as it determines
> the default branch.
>
> $ git push --set-upstream debian debian/sid
>
> # And all the others, according to the push configuration.
>
> $ git push --set-upstream debian
>
> # Use auto-generated packaging as starting point.
>
> $ mv ../generated-debian/ debian/
> $ git add debian/
> $ git commit -m "Initial packaging prepared by dh-make-golang."
> $ git push debian
>
>
> And now you are ready to go. I actually ran these commands, and you can
> see the result in
> https://salsa.debian.org/go-team/packages/new-workflow-demo/
>
>
> 2. Clone an existing repository from salsa.
>
> # The postclone hook in gbp-clone configures the debian remote.
>
> $ PKG=prometheus-node-exporter
> $ gbp clone git@salsa.debian.org:go-team/packages/$PKG
>
> # Add the upstream remote, fetch commits, but ignore tags.
>
> $ SRC=$(sed -n 's/Source:\s*\(\S\+\)/\1/p' debian/copyright)
> $ git remote add --fetch --no-tags upstream $SRC
>
>
Reply to: