Current issues with the Go Ecosystem in Debian [Was: DebConf19 pkg-go BoF minutes (Finally!)]
On 29/01/2020 14:34, Clément Hermann wrote:
> ---
>
> # Minutes of the Debconf19 Go Packaging Team BoF
<skip irrelevant stuff>
> DECISION: nodens will get in touch with someone from the release team to
> discuss the dak issue. Further discussion to happen on the Mailing List.
> Update post-BoF: nodens talked with Ivod - Detailed email on the issue
So I've discussed the matter with Ivo during DebConf19 in Curitiba, and tried
to map the issues properly. Hopefully this gives a good overview of the issues
and we can discuss how to move forward.
TL;DR: It's a complicated matter as there are several teams involved, but by no
means unfixable. We might want to start a discussion with the involved teams
(ftp-masters, release, rust…) first to be sure we're on the same page.
If anyone is comfortable with python and wants to help on the dak issue,
it would be great! I don't really know python myself.
Anyway, here it is.
---
# Trying to map the Go ecosystem issues in Debian
Why is it an issue when it's not the first language where you end up with
statically linked binaries?
What where the problems during Buster Release cycle?
- security team: go is actually (widely) used by people, and for sensitive
packages (Docker, Prometheus).
- Haskell/OCAML have theorically the same issues, but not the same exposure.
- Rust share some of the issues, but on a smaller scale.
# unstable->testing migration
Issue: we need to know, for a given package that is updated, what needs to be
rebuilt (binNMU).
Currently, we know it by using `Built-Using:` field. This is generated by dh_golang at
build time, using `${misc:Built-Using}` substvar.
Note: this is a policy violation, we need to change this field.
The field used by Rust is `X-Cargo-Built-Using`.
DISCUSSION: check with the Rust team? It would make sense to use the same, as we
might then share tooling. Alternatively, just make our own and use
X-Go-Built-Using (and tooling can check `X-[^-]+-Built-Using`
## How it can be a problem: exemple of the issue during Buster freeze
During the freeze, people kept uploading new versions of packages to unstable,
which meant that whenever a package in testing needed a fix, it would end up
building with a more recent version of libraries than what was in testing… So
we had to upload fake new releases to make the package go back to the previous
version and respect the freeze.
OCAML & haskell also have the issue, but they also have very stricts deps,
which means they have to coordinate a lot wrt. package migration.
In Go and Rust ecosystem, individual packages can migrate.
DISCUSS: find a way to prevent this (migrating when a Build-depends is different
than in testing).
- Maybe using autopkgtest?
- Britney should prevent somehow the migration of the rdepends until the
build-depends has migrated?
- just upload non-fixes to experimental during the freeze?
# stable updates (through security update or point release)
Everything has to be rebuilt and manually uploaded (either by Security Team or
Release team for a point release).
Same tool than for testing migration can be used in this case: we should be
able to map easily what needs to be rebuilt.
Problem is, it gets particularly messy when it's a security update: dak
database is public for the main archive, but not the security archive. Indeed,
there might be embargoed issues, so this database needs to be kept private (and
thus separate).
The issue is, in case of BinNMU, you still need the orig tarball, but if it's
the first security update, you don't have it in the local database, and dak
doesn't know how to check if it exists in another database, and even less how to
get it.
It means that if you upload a new revision, you also need the orig tarball. If
you do a binnmu, source isn't uploaded, and they don't have it in the security
archive, which means it can't be uploaded (dak will screem at you)
## potential solutions:
DISCUSS:
1. make a full upload every time an upload is made on the SA for a package for
the first time (so not a binnmu), and then you can make binnmu on this
package afterwards?
2. (preferred) have some way in dak to detect that the source actually exists -
even is it's a different db.
→ IOW: When I'm the security archive, if I don't have source, check the main
archive and get the source from it if it's there.
Maybe Security and Release Teams would be OK with 1., if we provide tooling to
help ? It sure seems the easiest solution.
## Suggestion on how to move forward by ivodd:
In the case we want to fix the issue on dak side - which would benefit not only
Go, but other similar ecosystems in Debian - the suggested path forward is to
try to create 2 archives using the interactive shell to replicate the problem.
Note that we'll probably need help (info) from ftp team/security team because
the databases will be empty initially.
Ivodd is working on a test that will send a binNMU, which is part of the
failure case. ETA unknown, so please don't wait up on that.
ACTION: replicate the failure on a test dak instance.
- 2 archives (one non-public)
- upload a binNMU in the non-public one
- it will fail because source is missing
- Then, it's "just" a matter of patching dak to look for the source in other archives
### References for dak
- lives at https://salsa.debian.org/ftp-team/dak
- check https://salsa.debian.org/ftp-team/dak/tree/master/integration-tests for
interactive shell test version (will import what's needed in local database
etc)
- tests are good examples (see 2000-xxx, which generate suites etc)
- debian/control shows all you need to setup a local version
- https://salsa.debian.org/ftp-team/dak/blob/master/docs/debian-specific.rst
shows what steps are needed for security archive handling (lots of manual
work)
Reply to: