Re: Bug#429577: #429577: ITP: emacs-snapshot -- The GNU Emacs editor (development snapshot)
[Dropping the bug tracker. I suspect that the first part of this
message may be of more general interest, and the latter (excepting
perhaps the "how we want to handle epochs" bit that mentions tag
encoding) will be primarily of interest to Arnaud, where it goes into
detail regarding the current packaging process.]
Arnaud Fontaine <arnau@debian.org> writes:
> I have almost finished preparing emacs-snapshot[0] and as I said to Rob,
> my plan is to keep emacs-snapshot package as close as possible to
> emacsXY package. I know that you've been working on unversioning emacs
> package (BTW Where is that repository?).
I haven't published any of the changes yet, but I'll push them to the
current salsa repository once they're ready (likely soon):
https://salsa.debian.org/rlb/deb-emacs
> * bin_priority (for update-alternatives): I think it would make sense
> to have an higher one for emacs-snapshot,
I don't have a strong opinion about this at the moment, but I wondered
if it might also be possible to argue the opposite, i.e. that
emacs-snapshot's priority should be low, so that if you had both emacs
and emacs-snapshot installed, you'd have to explicitly request
emacs-snapshot be the default, either globally by update-alternatives,
or locally with a suitable ~/bin symlink.
That way you'd only run snapshot in cases where you knew you wanted it,
and other things (particularly anything that's automated) would still
use the "stable" emacs. On multi-user systems, such an arrangement
might be even more desirable -- not as sure about single-user systems.
> what about a number big enough so it doesn't clash with future stable
> release (such as 999)?
Let's set this aside until we know what what we want to do with respect
to the ordering question above, but if we end up needing it, I was
pondering some kind of encoding like:
1 four-major-digits 0-or-1-indicating-snapshot-status
i.e.
100250
or something else that lets us sort the way we want (in this case
assuming the upstream doesn't produce emacs 1000.0 in a timeframe we
care about).
> * I have made several changes to emacs25 branch, feel free to merge
> them if they look fine to you (mostly debian/copyright work and
> fixing lintian warnings):
> https://salsa.debian.org/arnau/deb-emacs/commits/deb/emacs25/d/sid/master
Let's plan to work on this once I get the unversioning finished -- it's
imminent (most of it was finished just after debconf last year, but then
I got side-tracked). I'll be happy to help figure out how to handle any
conflicts, etc.
> And some packaging questions I will document in debian/README.source:
>
> * Do you plan to keep using git-dpm? (I'm asking because Dima, the
> current packager of emacs-snapshot, is using gbp-pq. I have no
> preference/opinion on this subject, just asking)
Yes, I do for now -- I've been using it for a good while, and I use it
in my other packages, so while I'm not opposed to switching if/when
something else is available that handles the main things I find helpful
about git-dpm, I'd like for it to appear worth the cost, and it'll take
some time.
> I ran the following command after importing patches from emacs25 and
> merging them with the ones from emacs-snapshot from Dima:
>
> $ git-dpm init --record-patch-name ../emacs-snapshot_20180414-1+git836dce6.orig.tar.xz deb/emacs-snapshot/d/sid/upstream
>
> However, I'm not so familiar with git-dpm, so would you mind
> explaining how you use it for emacs25/unversioned emacs?
Happy to, and note that I'm just describing what I currently do, not
arguing that any particular bit has to be done this way...
When beginning work on a new package revision, I make sure I'm on
deb/PACKAGE/d/sid/master and:
* gbp dch --git-author --git-log=--first-parent -aS --meta --full
(yes, I use gbp-dch because it handles --meta, etc.)
* git commit -am 'Adjust debian/changelog for UNRELEASED 25.2+1-6 development'
then if I only need to change debian/* I just use git normally, making
commits as appropriate, and (generally) writing the commit messages with
an eye toward the summary line and some or all of the body becoming an
entry in debian/changelog. I also include
Closes: #NNNN
lines at the end of any relevant commit messages, and I always try to
remember to include appropriate thanks to those who have helped with the
changes.
Then when I'm ready to release:
* gbp dch --git-author --git-log=--first-parent -aR --meta --full
* Edit debian/changelog to clean up the entries to add punctuation to
the summary line if dch didn't, avoid splitting Closes entries
across lines, fill lines (M-q), and add blank lines (C-j) between
entries (because it's been that way for a long time, and because I'm
still inclined to prefer reading it that way -- though I don't feel
all that strongly about it).
* git commit -am 'Update debian/changelog for 25.2+1-6 release'
Then I prepare the packages. I often use fakeroot debian/rules binary,
etc. for non-release builds, but for the releases:
* sbuild-update -udcar unstable-amd64-sbuild # If it's stale
* sbuild -d unstable -sA --build-failed-commands='%SBUILD_SHELL'
* Poke at the packages: install them locally, see if notmuch still
works, etc.
* If all is well, sign them, "git tag -s deb/PACKAGE/v/VERSION -s
deb/PACKAGE/v/VERSION", and upload.
Whenever I need to edit existing upstream patches, I do the following
(most often separately, once for each patch that needs changing):
* git-dpm checkout-patched
* git rebase -i HASH_PRECEDING_PATCH_THAT_NEEDS_EDITING
# Set 'e' for the patch I want to change
* Make the changes.
* git rebase --continue
* git-dpm update-patches
* git commit --amend
I do that last amend to change the merge commit message to something
suitable for the debian/changelog (via gbp-dch), which is why I handle
each patch separately in most cases. Of course it's possible there
might be a coordinated change that affects multiple patches that should
be handled with a single merge (all withing the same
checkout-patched/update-patches sequence).
Whenever I need to create a new upstream patch:
* git-dpm checkout-patched
* git cherry-pick UPSTREAM # or git-am, or ...
* Edit commit message(s) to look like those in the other patches,
i.e. write them in NEWS style given that we automatically generate
NEWS entries from the patch headers. I also try to remember to add
appropriate DEP-? pseudo-headers.
* git-dpm update-patches
* git commit --amend
During the amend, as for non-debian/ changes above, I change the merge
commit message to something suitable for incorporation into
debian/changelog via gbp-dch (not NEWS style). For example:
Block remote code execution via enriched text
Add 0012-A-remote-execution-exploit-via-enriched-text-has-bee.patch to
fix the problem. Thanks to David Bremner for the alert and Salvatore
Bonaccorso for reporting the problem to Debian.
Closes: 875447
All of the above approximately captures "steady state" work, but as you
might imagine, given the DFSG split, incorporating new upstream changes
is more difficult. I'll discuss that process in answer to your other
question below.
> * I followed deb-emacs25[1] naming scheme for branches and tags, thus:
> + Branches:
> deb/emacs-snapshot/d/sid/master
> deb/emacs-snapshot/d/sid/upstream
> + Tags:
> deb/emacs-snapshot/v/upstream/20180414-1+git836dce6
Looks good -- though we'll have to ponder how we want to handle epochs
(now that the unversioned package needs to add one). Sean said that
there was a DEP suggesting they be included in versions via "%",
e.g. 1%20180414-1+git836dce6.
I'm not sure what I think about that yet -- locally I've just created
two tags VERSION and 1%VERSION. If there's no harm in that, then maybe
it'd be the most flexible...
> I guess unversioned emacs Git repository follows this scheme too,
> right?
Yep - the unversioned work very closely matches the existing work, just
with a flavor of emacs instead of emacsXY.
> * Could you please explain how you remove non-DFSG documentation (such
> as emacs.texi) from the Git repository?
Yep. Unsurprisingly I imagine, this is one of the less entertaining,
and more complicated bits. Though the incremental work is notably less
than the original split was.
Below I'll describe my current git-dpm based process. Some of this is
work that would have to be done regardless, and some of it is git-dpm
specific. And note that when I new release comes out, we completely
ignore the upstream archive since we're going to have to repackage it
anyway. The packages are built from the upstream tags.
Also, I've created all of the instructions in this mail from notes
and/or memory, so be somewhat cautious, e.g. double-check that the
commands seem sensible if you try them.
To handle a new major release Y, given previous major X (minor releases
are simpler, but handled similarly):
* Validate upstream tag if there's a sig. Since there doesn't
currently appear to be a solid trust chain between my keyring and
theirs, I rely on the key I've seen in the previous release(s):
GNUPGHOME=some/where/release-gnupg git tag -v emacs-Y.1
* Update .git/config to add git-dpm entries for new branches, i.e.:
[branch "deb/emacsY/d/sid/master"]
dpmUpstreamBranch = deb/emacsY/d/sid/upstream
dpmPatchedBranch = deb/emacsY/d/sid/patched
* git checkout -b deb/emacsY/d/sid/master deb/emacsX/d/sid/master
* gbp dch --git-author --git-log=--first-parent -aS --meta --full -N Y.1+1-1
* git commit -am 'Adjust debian/changelog for UNRELEASED Y.1+1-1 development'
At this point, in general, we just merge the new upstream into the
existing upstream, but there could be cases where we'd want to "start
over", i.e. branch .../d/sid/upstream directly from the upstream tag.
For the typical merging case:
* git checkout -b deb/emacsY/d/sid/upstream deb/emacsX/d/sid/upstream
* git merge emacs-Y
* Use various commands (git, etc.) to figure out how to handle the
dfsg split. Try to determine what new or existing files have to be
removed, and whether any previously removed files can be re-added.
I imagine we could automate some of this, but especially in the
past, and unless we were planning to generalize it for "other
packages", this work was fairly rare and so I've just handled it in
an ad-hoc manner.
During this process, keep track of any corresponding changes that
will need to be made to the non-dfsg package.
Examples from my rough notes (untested):
- Check additions/deletions/renames and add/remove them from each
repo as needed (probably a way to have git handle this directly):
diff -u <(git ls-files --with-tree emacs-X) \
<(git ls-files --with-tree emacs-Y) | less
...rm new files non-dfsg files, i.e. anything with gfdl invariants
(front/back/sections), etc.
- Check path changes (i.e. new/deleted paths, etc.). Check
deleted-by-us to make sure license hasn't changed, etc. and git-rm
any that should still be gone, etc.
git diff --diff-filter=ACRB(maybe D) --name-status emacs-X emacs-Y
- Of course, we can also compare the previous sid/upstream upstream
against the new tag's tree.
* Commit the final tree with a suitable commit message - perhaps even
just starting with an existing one (cf. git commit -c REF), e.g:
commit 53dad1e8f5cd98546a215217b1f6dcd74fe03981
Author: Rob Browning <rlb@defaultvalue.org>
Date: Mon Apr 4 22:46:24 2011 -0500
Remove files that appear to be incompatible with the DFSG
Files that appear to be incompatible with the DFSG have been removed.
A number of files have been removed from this package because their
licenses are not compatible with the Debian Free Software Guidelines
(DFSG), or because it wasn't completely clear that their licenses are
compatible.
...
Once we have the upstream tree we want:
* git archive --prefix=emacsY-Y.1+1.orig/ deb/emacsY/v/Y.1+1 | xz -9v > \
../emacsY_Y.1+1.orig.tar.xz
* pristine-tar commit ../emacsY_Y.1+1.orig.tar.xz deb/emacsY/v/Y.1+1
* chmod a-w ../emacsY_Y.1+1.orig.tar.xz
* git tag -sm deb/emacsY/v/Y.1+1 deb/emacsY/v/Y.1+1
* git-dpm record-new-upstream ../emacsY_Y.1+1.orig.tar.xz deb/emacsY/d/sid/upstream
* git-dpm rebase-patched
* git-dpm update-patches
* git commit --amend --no-edit -m 'Merge upstream version Y.1'
* Fix debian/rules, etc. for major version change (bin_priority, etc.).
* debian/rules debian-sync
* git grep -F OLDVERSION debian # (and fixup and oversights)
* git commit -am 'Update debian/ for Y.1'
* Read NEWS; maybe add commits to adjust dependencies, ./configure, etc.
* gbp dch --ignore-branch --git-author --git-log=--first-parent -aR --meta --full
* git commit -am 'Update debian/changelog for Y.1+1-1 release'
* Examine/test packages, etc.
* If all is well, sign them, "git tag -s deb/PACKAGE/v/VERSION -s
deb/PACKAGE/v/VERSION", and upload.
For what it's worth, I often make a copy of my whole git tree before
beginning work I'm not certain about, so that it's trivial to "revert"
if I make a mistake, or if some command doesn't do what I expected,
(particularly for major updates).
Though of course it'd usually be possible to just note the relevant tags
and refs before you start so that you can roll them back manually if you
need to -- but "cp -a" and "mv" are simpler.
Hope this helps.
--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4
Reply to: