golang protobuf status (was Re: Bug#1033839: Packaging docker 24.0.9)
On Tue, Jun 4, 2024 at 9:59 PM Reinhard Tartler <siretart@gmail.com> wrote:
>
> Adding debian-go@, I think this topic deserves a wider audience
>
> sorry for the duplicate mail, I messed up the copy for debian-go@, so this is a re-send.
>
>
> On Mon, Jun 3, 2024 at 2:31 PM Reinhard Tartler <siretart@gmail.com> wrote:
>>
>> Basically notary insists on golang-github-golang-protobuf-1-3-dev, whereas golang-github-grpc-ecosystem-grpc-gateway.v2-dev (which is coming via the opentelemetry deps) is expecting golang-github-golang-protobuf-1-5-dev.
>>
>> How can we resolve this situation?
>
>
> so, I noticed that we have at least 12 packages in Debian that have alternative dependency declaration in the form of `golang-github-golang-protobuf-1-3-dev | golang-github-golang-protobuf-1-5-dev`. Most (if not all of them) contain generated *.pb.go files. I think I may have been under the false assumption that code generated with protoc-gen-go-1-3 would not be compatible with golang-github-golang-protobuf-1-5-dev. But is code generated with protoc-gen-go-1-5 actually compatible with golang-github-golang-protobuf-1-3-dev, that is, can it be linked with golang/protobuf @v1.3?
>
> However, in that case the alternative "Depends" on those 12+ packages still confuses me. Let me explain.
>
> As far as I can tell, there are several implementations of protobuf, each coming with their own code generator and runtime library. Let me put this down the four implementations of protobuf in Debian in the following table:
>
> source package | generator | notes
> -------------- | --------- | -----
> golang-gogoprotobuf | gogoprotobuf | more performant, deprecated upstream
> golang-github-golang-protobuf-1-3 | protoc-gen-go-1-3 | implements protobuf APIv1
> golang-github-golang-protobuf-1-5 | protoc-gen-go-1-5 | is a version of APIv1 implemented "in terms of APIv2"
> golang-google-protobuf | protoc-gen-go | APIv2, supersedes the above
>
> The above information is mostly taken out of my understanding from https://go.dev/blog/protobuf-apiv2
>
> The alternative dependencies in the 12+ package seem to imply that the generated code in those packages can be freely used with the runtimes that come with either golang/protobuf@v1.3 or @v1.5. But is that actually a safe assumption? I'm particularly concerned that code generated with protoc-gen-go-1-5 is unlikely to work with the library that comes with golang-github-golang-protobuf-1-3. I can see the other way around (code generated with protoc-gen-go-1-3 working with golang/protobuf@v1.3), but that sounds like something we should set a policy for.
>
I'm not sure about that either. But I only heard the runtime break
with gogoprobuf, not the google ones (as your next paragraph says). I
assume if the packages can be compiled, then they are safe.
> I totally see how code generated with gogoprotobuf not being compatible with anything else, leading to issues such as #975431 / https://github.com/containerd/ttrpc/issues/62#issuecomment-686768932. So this is effectively prevented by the "Conflicts" relationship in golang-github-golang-protobuf-1-5-dev:
>
Yes they are not compatible. But for the packaging, there are no
conflicts around gogoprotobuf packages. You probably misread
golang-goprotobuf-dev as golang-gogoprotobuf-dev. (These names are
very similar...). golang-goprotobuf-dev is a transitional package
which ties the runtime dev package and the code generator together.
> Package: golang-github-golang-protobuf-1-5-dev
> Source: golang-github-golang-protobuf-1-5
> Version: 1.5.4-1
> Conflicts: golang-github-golang-protobuf-1-3-dev, golang-goprotobuf-dev
>
> Similarly, we have another relationship here:
>
> Package: golang-github-golang-protobuf-1-3-dev
> Source: golang-github-golang-protobuf-1-3
> Version: 1.3.5-4
> Conflicts: golang-github-golang-protobuf-1-5-dev
>
> (Isn't there a missing Conflicts relationship on golang-goprotobuf-dev? -- is that intentional or an oversight?)
>
> In a similar vein, I can see that code generated by protoc-gen-go might cause similar crashes when used with the runtime from golang-github-golang-protobuf-1-3 (or possibly golang-github-golang-protobuf-1-5, but I'm not sure). Note that neither package declares any conflicts to golang-google-protobuf-dev (which comes from google/protobuf). Is that intentional or an oversight?
If the pb.go files are generated by protoc-gen-go (from
src:golang-google-protobuf), they can't be consumed by
golang-github-golang-protobuf-1-3-dev or
golang-github-gogo-protobuf-dev. The application will fail at compile
time, so it would just be noticed as FTBFS.
>
>
> Does my rambling above make sense? What conclusions do we want to take from here?
>
> - It seems to me that we probably want to move packages away from src:golang-gogoprotobuf as much as possible, in particular because of the interoperability concerns. Is that agreeable?
>
Yes. This package is dead, and we also should convince upstream to
move away from it.
> - It also seems to me that we probably want to avoid having both golang-github-golang-protobuf-1-3 and golang-github-golang-protobuf-1-5 at the same time. Both implement APIv1, and as such should be source-code compatible. We currently seem to have 19 packages build-depending on golang-github-golang-protobuf-1-3-dev. Can we lift-and-shift all of them in one transition to @v1.5?
>
I think so. The runtime library golang-github-golang-protobuf-1-5-dev
should be compatible with golang-github-golang-protobuf-1-3-dev.
Current approach of listing them as alternative dependency seems to be
working more gracefully. I probably should not introduce
golang-github-golang-protobuf-1-3-dev at first, but only keep the
protoc-gen-go-1-3 (the pb.go generator) around.
> - As for packaging docker 2.4 (cf. #1033839, which spun off this conversation), which of the four library implementations above should it be linking against? -- We can savely rule out gogoprotobuf (it's deprecated), and I guess google-protobuf as well, because I don't think all dependencies have moved to the APIv2. That leaves two options: golang-protobuf @v1.3 and @v1.5. Given the compatibility concerns above, I guess @v1.5 is the better choice. That means that at least the 'notary' package needs to be updated (as Arnaud points separately) to pick up protobuf v1.5.
>
I think the notary code doesn't need to change. Just switching the
golang-github-golang-protobuf-1-*-dev in Depends could work. As the
1.5 should be compatible with 1.3.
> - Currently, packages that use grpc (that is, build-depend on golang-google-grpc-dev or protoc-gen-go-grpc), contain code that is generated with golang/protobuf @v1.5. Wouldn't we expect applications that use grpc to (possibly silently) break when linked against any other library than protobuf @v1.5? Would it help to reflect this in package relationships (i.e., add appropriate Conflicts or Build-Conflicts relationships)?
>
protoc-gen-go-grpc is a new binary package, which is not used by any
package yet.
In theory, if the pb.go files are generated by protoc-gen-go-1-5 or
protoc-gen-go (the generator without version suffix is from
src:golang-google-protobuf), and the application still uses
golang-github-gogo-protobuf-dev as runtime library, than it will
break.
>From my observation for some upstream projects over these years, I
think the breakage only happens when the pb.go files from
golang-google-genproto-dev are generated by the new generators.
For example, the containerd project pins the version at
https://github.com/containerd/containerd/blob/74a86c30b3fcbb06802f4e3d95f49d0a127d4298/go.mod#L143
but they have bumped protobuf runtime as well as grpc runtime.
>
> Please do share your thoughts and opinions, I'd find the whole topic rather confusing and would like to help solve it, and not make it worse.
--
Shengjing Zhu
Reply to: