Packaging-Workflow paper [was: Re: ruby-cmdparse]
Am / On Sat, 14 Nov 2020 16:36:04 +0100
schrieb / wrote Klaumi Klingsporn <klaumikli@gmx.de>:
> Am / On Sat, 14 Nov 2020 19:32:40 +0530
> schrieb / wrote Utkarsh Gupta <utkarsh@debian.org>:
>
> > On Sat, Nov 14, 2020 at 7:18 PM Klaumi Klingsporn
> > <klaumikli@gmx.de> wrote:
> > > And again thanks for your help. I made a cheat sheet
> > > for the workflow and all the tips after all. So
> > > hopefully next time it will become easier ;-)
> >
> > That's great! Do you think you can put together this
> > thing in a markdown style so we can put it on the Ruby
> > Team Pages[1]?
>
> I can try but it'll take a while. At the moment it's a
> two-language-thingy, which needs a little bit more
> structure as well.
I finally managed to write it all down (in kramdown). It has
become more a manual than a cheat sheet but I learned a lot
writing it. And it now also includes the topic of building
a new package (as far as I understand this).
So, feel free to comment! Or do whatever you
want to do with it .
Klaumi
-----
Klaumi Klingsporn
mail: klaumikli@gmx.de
web: www.klaumikli.de
# How to help the Ruby-Team Maintaining Debian Packages for beginners
Helping the ruby-team upgrading packages maintained by the team or building new ones is always welcome. This document will try to give a short description of the general workflow for beginners.
Communication in the ruby-team mainly happens via mailinglist: [debian-ruby@lists.debian.org](https://lists.debian.org/debian-ruby/){:target="_blank"}. So it would be a good start to subscribe this list.
We also assume that you are a little bit familiar with the Debian package-management and have built one or another debian package for personal use before. You should have had installed at least the packages
* `build-essential`
* `maint-guide` and
* `developers-reference`
and you should have had taken a closer look into the installed Documents
* [New Maintainers' Guide](file:/usr/share/doc/maint-guide/html/index.en.html){:target="_blank"} and
* [Developer's Reference](file:/usr/share/developers-reference/index.html){:target="_blank"}
as well as into the [Ruby pages of the Debian Wiki](https://wiki.debian.org/Teams/Ruby/){:target="_blank"}
It will also usefull to set your name and the email-address you want to use developing packages in your `~/bashrc` (if you have not already done so):
DEBFULLNAME="your name"
DEBEMAIL=youremail@provider.com
All done? Then let's get started!
## Preparations Part I: Get ready to work with Git and Salsa
The packages maintained by the ruby team are developed using GitHub repositories at salsa.debian.org.
####1.
So first you need to have installed Git itself and a suite to help with Debian packages in Git repositories:
sudo apt install git git-buildpackage
> __Note:__ If you never have worked with git before the small and free ebook [Learning GIT](https://riptutorial.com/ebook/git){:target="_blank"} from [riptutorial.com](https://riptutorial.com){:target="_blank"} may give you an impression how this version-management-system generally works. But of course there are other tutorials as well.
####2.
Next you have to get yourself an account at [salsa.debian.org](https://salsa.debian.org){:target="_blank"}. With this account you get personal webspace for your projects which you can access in Salsa's web-interface at: `https://salsa.debian.org/<your-user-name>`.
> __Note:__ When you are logged in at *salsa.debian.org* you will find a dashboard menu on the left side of the page to interact with Salsa's GitHub.
Normally you don't want to work with the web-interface of Salsa but clone a repository to your local machine, work there and push your changes back to Salsa after you are done.
Especially when are pushing code Salsa needs to be sure that the machine you are working at is really yours (even if you after days of work may think it might be the other way round). Therefore you need to create a ssh-key (or use an existing one from `~/ssh/id_rsa.pub`) and add the public-key to your Salsa account. Run:
ssh-keygen
to create an ssh-key, and upload the public key, which ends with `.pub` via Salsa's web-interface: Go to *User settings* and choose *ssh-keys* in the left dash-bar.
If you have picked a custom username for Salsa which differs from your username on your local machine you can add the following to `~/.ssh/config` (if the file does not exist yet just create it):
host salsa.debian.org
User <your-salsa-username>
PreferredAuthentications publickey
IdentityFile ~/.ssh/<your_custom_name_git_rsa>
####3.
To have the possibility to interact with Salsa's GitLap API directly (what maybe usefull at some point) you need to install the package libgitlab-api-v4-perl:
sudo apt install libgitlab-api-v4-perl
To make the command-line-client work with Salsa you have to create a personal access token on salsa via web-interface: Go to *User settings*, choose *Access tokens* in the left dash-bar, then choose *with scope: api* and copy the created token. Run:
gitlab-api-v4 configure
and add the token.
>__Note:__ You will NOT see your input! Use copy and paste! The configured token is stored in the file `~/.gitlab-api-v4-config`.
## Preparations Part II: Setting up the building an testing environment
If you built debian packages for personal use before you probably had used something like `dpkg-buildpackage` or `debuild` in the source directory on your local machine to build the source and binary packages.
But your daily used machine normally runs a stable Debian system, maybe testing or a mixture of both with some unstable packages. To build and test an official Debian package this needs to happen in an clean unstable environment. So you will have to build and test using chroots or lxc-containers.
###1. ruby-meta
To make setting up such an environment for building and testing a little bit easier for you the ruby-team offers a set of scripts called *meta*, which is located in its own Salsa-repository: [salsa.debian.org/ruby-team/meta](https://salsa.debian.org/ruby-team/meta){:target="_blank"}.
Clone this repository to the local computer where you want to build and test packages:
git clone git@salsa.debian.org:ruby-team/meta.git meta
Enter the newly created directory *meta*, read the *README.md* and then create your building and testing environment by running the *setup*-script (which utilizes sbuild, autopkgtest and lxc to do so):
./setup
The script will install missing packages and therefore ask for your user password. You may have to install some missing packages by yourself too to help. At the end the script informs you:
Finished testbed setup for unstable: <Date>
From now on you can use the *build*-script in *meta* repository from within your package's local git repository to build and test in a clean environment.
###2. lintian options
For testing the packages lintian should be called with some extra options to make its report more verbose and pedantic. For using the *build* script of ruby-team's meta-repository you should make this options standard for sbuild by adding them to your `~/.sbuildrc`:
# this enables all lintian tags:
$lintian_opts = ['-i', '-I', '-E', '--pedantic'];
***
And after all this preparations: You are finally ready to start your first project!
## Case 1: Help upgrading an existing package
First of all: When you want to help upgrading an existing package make the debian-ruby team know about it by posting your intention at the debian-ruby mailinglist! You may also want to file a bugreport for the package about not being upgraded.
###1. Fork existing GIT repository of the package
The git repositories of the packages maintained by the ruby-team are stored in `https://salsa.debian.org/ruby-team/<package-name>`.
As the ruby-team don't know you and you don't know them the easiest way to start your upgrade attempt will be to fork the ruby-teams repository of the package to your own namespace at Salsa. To do so log in at salsa.debian.org, go to the ruby-teams package repository `https://salsa.debian.org/ruby-team/<package-name>` and click on the small *Fork*-Button on the upper right side of the page (beside the package name). Choose your own namespace as the one to fork to.
You can now work with this fork in your own namespace at `https://salsa.debian.org/<yourname>/<package-name>` without any fear to destroy something. When you are done you will make a merge request for the teams repository to get your changes back there.
###2. Next clone (download) your Salsa Git repository to your local computer.
A debian package repository normally has a lot of tags and at least 3 branches:
* _master_ : thats where you want to work
* _pristine-tar_ : where the source-packages are stored
* _upstream_ : where the upstream source tree is stored
To get all this tags and branches cloned to your computer it is the easiest way to use *gbp* instead of *git*. Go to the location where you want to store the directory with your local git-tree of the package and run:
gbp clone --pristine-tar git@salsa.debian.org:<yourname>/<package-name>.git <package-name>
Then change to the newly created directory
cd package-name
and start your changes.
###3. Import the new upstream version
This should be easy if the information in the debian/watch file still works. You can check if the new upstream version is found with:
uscan --no-download --verbose
If all works run:
gbp import-orig --pristine-tar --uscan
This will download the upstream source package and create a debian source-package in the parent directory with uscan and then extract this source-package into the current directory and import it in the master branch. You can also run this two steps seperately:
uscan --verbose
gbp import-orig --pristine-tar ../<package>_<new.version>.orig.tar.gz
Now update the debian changelog file to the new version:
gbp dch -a
and open the file `debian/changelog` with an text-editor of your choice. In the first line replace the word `testing` with `UNRELEASED` and as first entry of changes add `* Team Upload`. Then make git aware of this change by running:
git commit -a -m "set package as unreleased and team-upload"
Last, to update your remote Salsa repository you want to push all changes in all branches using gbp:
gbp push
or using git:
git push -u --all --follow-tags
###4. Make all necessary changes in the debian-directory
__Remember:__ All changes are only done in the debian-directory! Do not touch the upstream code! If you have to do so make patches using quilt. See:
[Debian Wiki: Using Quilt](https://wiki.debian.org/UsingQuilt){:target="_blank"}
and
[New Maintainers' Guide: Modifying the source](https://www.debian.org/doc/manuals/maint-guide/modify.html){:target="_blank"}
All changes should be done in the master branch of the repository. To check which branch the directory-tree in front of you represents run:
git branch
the master should be highlighted. If it's not change to master with
git checkout master
Git will then rebuild the tree to match the master branch.
When upgrading the package you have to check all files in the debian-directory, adjust at least the dates in the copyright-file and eventually existing patches. Also consult the upstream changelog what else might have changed affecting the packaging.
Whenever you have done all changes concerning a specific topic you should document your changes in the debian/changelog file and make git aware of your changes by running:
git commit -a -m "short description of your changes"
from within the main-directory of your git repository.
>__Note:__ You can also run the `git`-command without the `-m` option. In this case an editor (*vi* if not configured otherwise) will pop up where you have to type in your description of the changes. If there is no description, the changes are not recorded by git.
(To remind you of the essential `vi` commands: `<I>` or `<INS>` will put the program into input-mode, `<ESC>` will return to command-mode, `:wq` while in command-mode will save the changes and exit the program.)
Whenever you create a new file you have to make git aware of this file:
git add debian/<newfile>
When you think you are done with all your changes or when you want to have a longer break you can push your local changes in the actual (master-)branch back to your remote Salsa repository with:
gbp push
To only see what will happen you may try first:
gbp push --dry-run
When you are working at more than one local machines or when you are working together with other people on the same remote repository every time you start working you have to make sure that the local git tree matches the remote one by running:
gbp pull --redo-pq
and you should push back your changes whenever you want to change the local machine or pause working.
###5. Building and testing the package:
The moment of truth: When you finally think you are done with all necessary changes in the debian directory you try to build the package and test ist. If you had done the preparations part II above you only need to run the build-script included in the ruby-teams meta repository from within the local git repository of the package you want to have built;
~/<path-where-you-cloned-meta>/build
It will run the necessary commands to build (sbuild) and test (autopkgtest, lintian) the package and also will rebuild and test all packages (build-)dependening on the package you just built. The final part is essential when trying to build and upload a major upgrade of a package.
This build process will leave your local git repository unchanged, its products can be found in in a directory `build-area` in the parent folder of your repository.
And of course something will go wrong! Maybe the package-building itself doesn't work at all, some tests may fail or at least lintian will grumble about something. In all this cases you need to find the reason for the problems, go back to work on the files in the debian-directory and try to fix the problems. If you are not sure how to do so: Never say die! Ask the debian-ruby mailing-list!
When you think you fixed all problems run the build script again.
###6. Merging to ruby-teams repository, fetching and merging back changes
When building and testing the package finally works fine don't forget to push your latest changes to your remote repository:
gbp push
Then you should brief the ruby-team that you have finished upgrading the package and create a *merge request* for your (remote) repository into the ruby-teams repository via Salsa's web-interface: Log in, go to the teams repository of the package and choose *Merge Requests* on the left dashbar.
You have to create one merge requests for every branch you worked on since you forked the teams repository, which were: _upstream_, _pristine-tar_ and _master_.
The team members will inspect the changes you made and may accept them, may change things by themself or may want you to change things.
In any case before you start working on your repository again you have to make sure that it still matches the teams repository. Therefore you need to add the teams repository to yours:
git remote add team git@salsa.debian.org:ruby-team/<package-name>.git
so that is available for git under the name team. Check with:
git remote -v
You can now update your local repository and get the teams content by running:
git fetch --all
>__Note:__ This will not pull any changes to your repository, fetch only downloads objects and refs from another repository.
You can check the changes between your master branch and the one in the
team's repository:
git diff master team/master
When there are changes in the teams's repository you should merge this repository into your master branch with:
git merge team/master
If something seems to go wrong you can allways abort with:
git merge --abort
When the (back) merging went fine you can push the changes to your remote repository:
gbp push
and start working on your repository again.
##Case II: Preparing a new Package
The workflow in this case is very similar to the one described above for upgrading a package but the beginning is a little bit different: There is no remote repository to clone and to start with.
So you have to create a new git repository for your package on your local machine first.
###1. Debian package from a *rubygems* Package
In the case you want to build a Debian package for a piece of software that is available at rubygems you can let gem2deb do most of the work. Go to the directory where you want your new git repository to be created an run:
gem2deb -p <debian-package-name> -g --purge <gem-Package>
This will run `gem fetch <gem-Package>` to download the package from rubygems, build a git-tree for the package, build the debian-source- and binary-packages, import the prestine-tar to git-tre and then delete all crap outside the git-tree. As result you have a git-tree with 3 branches:
* `master`
* `pristine-tar`
* `upstream`
and 2 tags:
* `debian/<upstream-version>-1`
* `upstream/<upstream-version>`
Because on Salsa the debian tag is only set when a package is released you should remove this tag for now. Look up the name of the tag:
git tag
and remove it:
git tag -d debian/upstream-version>-1
###2. Debian Package from an upstream Archive-File
In case the software you want to package is only available upstream as some gzip, bzip2, lzma or xz compressed tar archive or as zip archive you have to do a little more.
First check if the package `python3-requests` is installed on your system:
dpkg -l python3-requests
If it is not install it:
sudo apt install python3-requests
Then go to the directory where you want to store your local git-repository of the package and create a directory with the name of the debian package you want to build and change into it:
mkdir <package-name> && $_
Now place this directory under Git's version control, i.e. make an empty Git repository:
git init
Next point your Browser to the site where the upstream archive can be downloaded and copy the link for the archive you want to use.
Then go back to your terminal and use *gbp* to download and import this archive to your empty git repository:
gbp import-orig --interactive --upstream-version=<upstream.program.version> --pristine-tar <https://place-your-copied-archive-url.here>
This lets *gbp* download the upstream archive, import it into your git directory, create and populate 3 branches *master*, *upstream* and *pristine-tar* and tag the upstream version.
__Note:__ If the url you gave *gbp* to import is an url-redirect the above *gbp*-command will not work. In such a case you have to download the archive yourself, place it in the parent directory of you repository an run from within your repository:
gbp import-orig --interactive --upstream-version=<upstream.program.version> --pristine-tar ../<downloaded-archive.tar.xzy>
After this *gbp*-import you will notice that the *debian* directory and all the files in it are still missing in your repository.
So let's create it:
mkdir debian
and let `dh_make` populate it with templates:
dh_make -p <package-name>_<version> --addmissing
`dh_make` will ask you some questions about the package, choose the templates to copy and write a first entry into the copyright file.
>__Note__: To avoid questions you can give `dh_make` more options. Namely `-c <copyright-typ>` might be interesting if you know the license typ of the program because `dh_make` does not ask for it.
After `dh_make` has populated the debian directory you have to make git aware of all the new files:
git add debian/.
and commit all changes:
git commit -a -m "create and fill debian directory"
###3. Creating the remote Salsa repository
Wether you have created your local git repository from a rubygem package or from an upstream archive before making any further changes in the debian directory you should setup your remote Salsa repository.
Point your browser to `https://salsa.debian.org/<your-username>`, log in and create a new project with the name of your new debian package via Salsa's web-interface.
Then go back to your terminal and tell git that the created new and empty git repository at Salsa is your remote origin:
git remote add origin https://salsa.debian.org/<your-username>/projectname
and push your local repository to the remote:
gbp push
or:
git push -u --all --follow-tags
Now you can start ...
###4. working on the debian directory and all further tasks
Especially when you created your repository from an upstream archive, adjusting the files in the debian directory will be much more work as when upgrading an existing package. You have to touch every single file, work on it or decide to delete it. You have to set dependencies and build dependencies for the package, the description and so on. Check the upstream docs to get some information. Sadly `dh_make` has no template files for ruby packages. You may want to take a look at other ruby repositories' debian directories to get some ideas.
Nevertheless: Concerning the working process with git and gbp it is all the same as upgrading a package: You have to commit your changes with comment:
git commit -a -m "short description of your changes"
You have to inform git of newly created files:
git add debian/<newfile>
and at the end of your work you have to push your changes to your remote repository:
gbp push
The same applies for building and testing the package: It is the same workflow as upgrading an existing package.
Only getting your work finally into a ruby team's repository will differ slightly: You cannot make a merge request, the team has to fork your repository into their namespace.
##Troubleshooting
To be continued ;-)
Reply to: