[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Packaging crowdsec and some other packages



Hello Cyril,

just answering a few points quickly.

> [...] “Plugins” can wait a little.

If I understand correctly, you want to focus on crowdsec the program first. In this case, starting with 'dh-make-golang make -type program' is the right way to go. The source package will be named 'crowdsec', and produce a binary package named 'crowdsec'. Later on when you want to provide a library package, you will just add a new binary package to debian/control, named 'golang-github-crowdsecurity-crowdsec-dev', then add the file `debian/golang-github-crowdsecurity-crowdsec-dev.install`, and that's pretty much done already.

Many packages do that, for example nomad, consul, containerd.


>    Maintainer: Debian Go Packaging Team <team+pkg-go@tracker.debian.org>
>    Uploaders: Cyril Brulebois <cyril@debamax.com>
>    [...]

I also think you're right here. All the build depends, ie. packages you don't necessarily care about, and that might also be used by other programs, are better maintained within the team. So that there's no friction to work on it, and everyone from the team can fix or update them when needed. All these small packages are "common ground" more or less.

As for the main package crowdsec, that you care about, I guess it also makes sense to put yourself as the maintainer, or to create a "Corwdsec Packaging Team" as you suggest. Right now I maintain the docker.io package, it's the same situation, it lives in the "Docker Packaging Team".

> Does everything make sense so far?

Yep.

> One which seemed easy enough was the following, and the rev-deps:
>  - golang-github-oschwald-maxminddb-golang
>  - golang-github-oschwald-geoip2-golang
>  - syncthing

So basically, when you update libraries, you should check that the rev build deps don't break. We have a tool for that, it's ratt (rebuild all the things). It automates rebuilding all the reverse build deps with sbuild, while injecting in the build environment the new package.

If ratt succeeds, then you can upload. If you're still unsure, better ask on the Debian Go mailing list before uploading, it won't hurt. Some packages are sensitive, or have many many reverse build deps. Better ask before upload until you're sure :)

If ratt fails (ie. some packages fail to build with the new library that you just packaged), then you should investigate why, and fix the reverse build-deps for the new package that you intend to upload.

If ratt fails and you don't care, and upload your package all the same, well, nothing will stop you, but it's not nice at all. You're putting more work on other people's shoulders. And it can be a lot of work. Please don't do that :)


> Speaking of rev-deps, do we have any specific tooling to inspect all the
> reverse dependencies of a given package, to make sure I wouldn't miss
> anything?

As mentioned above, check ratt, which relies on dose-ceve to find the rev build-deps, if I'm not mistaken.


Apart from that, a few comments from my experience.

Sometimes, a little patching is much better than a lot of blind packaging. For example, if you can patch a feature away, and save yourself packaging and maintaining a tree of 10+ dependencies, then maybe you should do that. You can see that at work in docker-registry, where support for relic and azure was removed. Nobody complained.

You should exclude the 'vendor' directory if there's one. It's done in the Files-Excluded field of debian/copyright. However, for the "big packages", it's fairly common to keep a few of the vendored libraries, either because it saves us from a complicated build failure, or because some dependencies are not packaged. You can see that at work in the debian/copyright file of "big packages", once again: nomad, consul, containerd. docker.io, ... However, 1) the Files-Excluded list is a bit of a pain to maintain, 2) You're out of luck with crowdsec, they don't have a vendor directory.

Other than that, good luck with crowdsec, and welcome to the Go Team, we have many packages and need more hands!

Cheers,

  Arnaud

--
Arnaud Rebillout -- https://arnaudr.io


On Wed, Dec 16, 2020 at 5:29 PM Cyril Brulebois <cyril@debamax.com> wrote:
Hi everyone,


Context
-------

Let's start with some words about CrowdSec for context. From its
website:

    CrowdSec is a free, open-source, security automation tool using
    local IP behavior detection & an online community-driven IP
    reputation system.

More at: https://crowdsec.net/

There's also a company behind it, who's asked my help to get the package
into Debian. You'll find my proposed plan below, and I'd be happy to get
some feedback about it since Go packaging is rather new to me, and I'd
like to make sure not to add any burden on the shoulders of the Debian
Go Packaging Team.


New packages
------------

CrowdSec would be a program-oriented package. I didn't dive too deep
into dh-make-golang, but I would expect it not to be entirely crazy to
start with a program profile, and possibly add a binary or two after a
while, if we'd like to make some “plugin mechanism” (for the lack of a
better term) available in the future. This would make it possible for
other packages to have in their build dependencies a -dev package
produced by crowdsec. I'd like to start with some kind of “minimal
viable package” anyway, since there's lot to do already, hence my
current focus on starting with the program aspect (some postinst/wizard
will require some attention as well)… “Plugins” can wait a little.


This package would depend on quite a number of new packages. Since those
packages aren't in Debian at the moment, and would then only be used as
Build-Depends for the src:crowdsec package, I've investigated their
initial packaging as library-only. I haven't tweaked metadata yet (as
lintian correctly spotted, looking at the quite awful descriptions),
tweaked some excludes and/or ignored test results to get everything
started, and I've reached a point where crowdsec's building process
finds all of the required dependencies, and builds fine.

Checking the incrementally-fed reprepro repository where I've stashed my
results, the involved packages would be:
 - golang-github-alecaivazis-survey 2.2.4-1
 - golang-github-antonmedv-expr 1.8.9-1
 - golang-github-appleboy-gin-jwt 2.6.4-1
 - golang-github-appleboy-gofight 2.1.2-1
 - golang-github-denisbrodbeck-machineid 1.0.1-1
 - golang-github-enescakir-emoji 1.0.0-1
 - golang-github-facebook-ent 0.5.1-1
 - golang-github-go-co-op-gocron 0.3.3-1
 - golang-github-goombaio-namegenerator 0.0.2-1
 - golang-github-go-openapi-inflect 0.19.0-1
 - golang-github-hinshun-vt10x 0.0~git20180809.d55458d-1
 - golang-github-jamiealquiza-tachymeter 2.0.0-1
 - golang-github-logrusorgru-grokky 0.0~git20180829.47edf01-1
 - golang-github-mohae-deepcopy 0.0~git20170929.c48cc78-1
 - golang-github-netflix-go-expect 0.0~git20201125.85d881c-1
 - golang-github-nxadm-tail 1.4.5-1
 - golang-github-prometheus-prom2json 1.3.0-2

(I haven't double-checked whether some ITPs are around for any of those
yet.)

If the “let's start with library-only” aspect looks good to you, I was
expecting to have all those packages have the recommended (dixit
<https://go-team.pages.debian.net/packaging.html>):

    Maintainer: Debian Go Packaging Team <team+pkg-go@tracker.debian.org>
    Uploaders: Cyril Brulebois <cyril@debamax.com>

while the crowdsec package itself would be using something like:

    Maintainer: Cyril Brulebois <cyril@debamax.com>
    Uploaders: Debian Go Packaging Team <team+pkg-go@tracker.debian.org>

or:

    Maintainer: CrowdSec Packaging Team <debian.package@crowdsec.net>
    Uploaders: Debian Go Packaging Team <team+pkg-go@tracker.debian.org>, Cyril Brulebois <cyril@debamax.com>

(Maybe some upstream developers might want to update the package on
their own after a while, but some reviewing/mentoring would be required
anyway.)

The idea would be that help from the team would be welcome here, even if
it'd be nicer to have me and/or upstream in the loop depending on the
scope of proposed changes.

I would be on the lookout for everything related to the packages we're
introducing, through my subscription to this very list.

Does everything make sense so far?


Updated packages
----------------

I'll have some specific questions (that I'll ask in separate threads)
for some aspects, like a possible package/gopath misdetection, how to
deal with vendoring properly, how to deal with an obsolete v8 vs. a v10,
etc.

The second biggest topic for me besides new packages would be about two
sets of packages that are a bit dated in Debian. Upstream would prefer
if we could just update the relevant packages, instead of having to
“downgrade” their code, or disable features.

One which seemed easy enough was the following, and the rev-deps:
 - golang-github-oschwald-maxminddb-golang
 - golang-github-oschwald-geoip2-golang
 - syncthing

Fetching “blindly” the latest versions for the first two makes crowdsec
happy, and syncthing (with its apparently extensive test suite) still
builds fine.

I've added Alexandre Viau in copy to see if proceeding with those
upgrades would seem appropriate, or whether I should envision a
different plan.


The second stack seems way more packed, it's about the
golang-github-gin-gonic-gin package, and a quick debtree -R shows quite
a number of reverse dependencies. I haven't investigated rebuilding them
and checking how it goes yet, I have only checked that updating this
package (and introducing in turn new dependencies/packages) makes it
possible to build crowdsec. I've added Shengjing Zhu in copy to get some
input, would updating this set of packages be welcome?

Here's the list of extra packages I needed to get the new upstream to
build:
 - golang-github-gin-gonic-gin 1.6.3-1
 - golang-github-go-playground-assert 2.0.1-1
 - golang-github-go-playground-locales 0.13.0-1
 - golang-github-go-playground-universal-translator 0.17.0-1
 - golang-github-go-playground-validator 10.4.1-1
 - golang-github-leodido-go-urn 1.2.0-1

Again, I didn't investigate the rev-deps (yet!).


Speaking of rev-deps, do we have any specific tooling to inspect all the
reverse dependencies of a given package, to make sure I wouldn't miss
anything?


I'm happily taking any questions/comments/recommendations, thanks for
reading thus far!


Cheers,
Cyril.
--
Cyril Brulebois -- Debian Consultant @ DEBAMAX -- https://debamax.com/

Reply to: