Re: grpc transition

Sorry for being late to the party, I've been spending now a few days of research to understand this issue. I'd like to share my thoughts, see below.

On Sun, Mar 24, 2024 at 4:10 AM Maytham Alsudany <maytha8thedev@gmail.com> wrote:

Hi Simon,

On Sun, 2024-02-18 at 20:42 +0100, Simon Josefsson wrote:
> Shengjing Zhu <zhsj@debian.org> writes:
>
> > My last attempt for the grpc-go transition is stuck at the protobuf things.
> > One thing that might help is gprc-go is finally going to use the new
> > protobuf library, https://github.com/grpc116:33: /grpc-go/pull/6961
> > So we could decouple the grpc-go transition from the protobuf things.
>
> Are you saying we should apply that patch to grpc-go and upload to
> experimental, and hope that will help resolve the FTBFS in reverse
> dependencies? Do we need to change some Build-Depends as well? It
> seems grpc 1.60 has these proto-related Build-Depends:
>
> golang-github-golang-protobuf-1-5-dev,
> golang-google-genproto-dev,
> golang-google-protobuf-dev,
> protobuf-compiler (>= 3.11.4~),
> protoc-gen-go,

This patch (which is now merged and included in >=1.62.0) just removes the requirement for the
legacy golang-github-golang-protobuf-1-5-dev dependency. This should fix dependency problems where
there was conflict between the 1-5 and 1-3 variations.

You are correct. However, there are more serious issues as you point out below.

> Not all FTBFS errors seems related to protobuf, any ideas on the error
> below? Or is this a protobuf related error?

This error is due to outdated golang-google-genproto package, which itself is a whole other rabbit
hole (involving cyclical dependencies).

Here is the thing: the golang-google-genproto package is essentially "pre-generated" golang code derived from protobuf .proto files that come from protobuf. For example:

Given the fundamental type definition in https://sources.debian.org/src/protobuf/3.21.12-8.2/src/google/protobuf/type.proto the golang-google-genproto package in sid ships with this:

https://sources.debian.org/src/golang-google-genproto/0.0~git20200413.b5235f6-3/protobuf/ptype/type.pb.go

Note that in sid, the 'type.pb.go' was generated with the protoc-gen-1-3 package.

However in experimental, the golang-google-genproto package ships with this:

https://sources.debian.org/src/golang-google-genproto/0.0~git20210726.e7812ac-2/protobuf/ptype/type.go

Note that in experimental, this 'type.go' is not generated at all. Instead, it is a "forwarder" package to import the definitions from the golang-google-protobuf package:

https://sources.debian.org/src/golang-google-protobuf/1.33.0-1/types/known/typepb/type.pb.go

And this one in turn is generated again from the src:protobuf (i.e., from https://sources.debian.org/src/protobuf/3.21.12-8.2/src/google/protobuf/type.proto), but this time with the protoc-gen-go compiler from the same sources.

Confused? -- it gets worse (sorry). For any of this to make sense, you have to realize that protobuf in golang has gone through a major API evolution in 2020 *sic*. The details are outlined in https://go.dev/blog/protobuf-apiv2, but in short:

- the golang-github-golang-protobuf-1-3 package implements the protobuf APIv1

- the golang-github-golang-protobuf-1-5 package implements the protobuf APIv1, but using the implementation from APIv2
- the golang-google-protobuf package implements protobuf APIv2

The situation in Debian now is extremely messy, because the src:golang-google-grpc_1.38.0+really1.33.3-1 package in sid is using code generated by the protoc-gen-go-1-3 compiler, which produces APIv1. We now have a bunch of applications that have not been ported to APIv2, some of them not even upstream. Those have no hope of actually working with golang code generated from a compiler that produces protobuf bindings using APIv2.

On a side-note, even swapping out golang-github-golang-protobuf-1-3 with golang-github-golang-protobuf-1-5 breaks some packages, albeit they are supposed to be mostly compatible.

So how do we get out of this mess? Unless we want to break (read: remove from testing/unstable) quite a large number of packages, I would suggest to introduce:

* src:golang-google-genproto-apiv1 which is essentially a rename of the package golang-google-genproto we have currently in sid. This contains the fundamental types for googleapis and protobuf compiled with golang/protobuf@v1.3 and thus implements the APIv1
* src:golang-google-grpc-apiv1 which is essentially a rename of the package golang-google-grpc we have currently in sid. This is built against the golang-google-genproto-apiv1-dev binary package (which comes from src:golang-google-genproto-apiv1) and serves as a stop-gap for existing applications that are not ready to be ported to APIv2.

With those packages in place, we can then:
* Upload src:golang-google-genproto from experimental to sid
* Update src:golang-google-grpc from experimental to sid
* Fix packages that FTBFS by either updating them to newer upstream versions that compile against protobuf APIv2, or update their build-depends to build against the APIv1 versions.

I think this gives a clear plan forward and allows us to update to newer versions of grpc while also allowing us to keep packages that are not ready to do this migration.

Please let me know what you think. Can you think of a better plan forward?

regards,
Reinhard