On 2021-06-03 02:05 +0100, Wookey wrote: > On 2021-04-23 02:46 +0100, Wookey wrote: So, I got back to having another look at this. [Background for those new to this conversation: I'm working on packaging tensorflow and that has googleapis as a build-dep. Looking at this package it seems like we have a coresponding package for golang (which to be fair is separate upstream now) and subset packages for ruby and java, but not the other supported-by-upstream languages). Perhaps it makes sense to tidy this up to be more consistent in debian and build the full googleapis set of headers/classes/libraries for all the supported languages from one consistent source?] > > There is a package in debian built from > > https://github.com/googleapis/googleapis > > ruby-googleapis-common-protos-types but its 'source' is the _output_ > > of processing upstream with the protocol buffer compiler. Arguably > > that's not the 'preferred form of modification' and that package should be fixed? > > it's also a version from 2018 so probably quite a lot has changed. > > > > It seems to me that the right way to package this is to generate all > > the language outputs from one source package (ruby, java, gp, python, > > nodejs, php, csharp) maybe there are more?, > > I did try just building the package from upstream and had some success > > after installing protobuf-compiler and protobuf-compiler-grpc and > > python3-grpc-tools, ruby-google-protobuf and protobuf-compiler-grpc-java-plugin > > > > 3 out of 7 languages build. I don't know which language bindings tensorflow actually wants. Subsequent research suggests that it wants C/C++ headers for 'google/monitoring/v3', (defined in third_party/googleapis/BUILD) although I've still not worked out where they get used. > I've committed these experiments to a salsa repo here: > https://salsa.debian.org/deeplearning-team/googleapis > > Anyone is welcome to see if they can get some of the broken languages > building, or otherwise fill out the debian dir. So, I've now got 6 languages working (cpp, java, php, python, csharp, ruby). node, and objectivec still fail, and remain to be grokked. cpp was failing due to a bug in the compiler which fell over when '$field' was mentioned in a C++ style comment. Upstream's fix for this for the time being is to remove that $ in the comment. Not very impressive, but it'll do for now. https://github.com/GoogleCloudPlatform/cpp-samples/issues/113 It was nominally fixed in grpc back in March but hasn't been incorporated or released yet, so we get to work around it. https://github.com/grpc/grpc/pull/25538 ruby and php were both failing because their grpc compilers/parsers only support proto3 format, not proto2 format, and there are a couple of proto2 format files, notably in the grpc compiler files itself. I don't know if those are just out of date or if there is a good reason that grpc still has proto2 files? I modified the makefile to check for the 'proto2' descriptor and if it was present, to remove those files from the list to be processed. ruby is more picky than php as the code generator parsed dependencies and then checked those for being proto2 type. So I had to just resort to specifically excluding google/cloud/bigquery/storage/v1alpha2/protobuf.pb.cc from the processed files list (as writing a recursive parser to do the checks myself in the makefile seemed excessive.) node: gives this error from make OUTPUT=./debian/node LANGUAGE=node "protoc-gen-node: program not found or is not executable" Now there is a /usr/bin/grpc_node_plugin and a /usr/bin/protoc-gen-go (in golang-goprotobuf-dev), and a /usr/bin/protoc-gen-rust (in protobuf-codegen), but I can't find a /usr/bin/protoc-gen-go. Anyone know where this should come from? objectivec: has the same issue as node: protoc-gen-objectivec: program not found or is not executable (again there is a /usr/bin/grpc_objective_c_plugin) Where to put these files? This is presumably different or each language. I know how things work in C/C++ (and perl), and have some idea of java, but everything else is a bit of a mystery to me. Existing ruby package (ruby-googleapis-common-protos-types) puts ruby stuff in: /usr/lib/ruby/vendor_ruby/google Existing golang package (golang-google-genproto-dev) puts them in: /usr/share/gocode/src/google.golang.org/genproto/googleapis/ (and also has some extra stuff in /usr/share/gocode/src/google.golang.org/genproto/protobuf - is that important I wonder?) There is a java package libgoogle-common-protos-java which contains proto-google-common-protos-1.16.0.jar, which has a subset of the output from make OUTPUT=./debian/java LANGUAGE=java compiled into classes So we have this 'common' subset packaged for both ruby and java. The java one comes from https://github.com/googleapis/java-common-protos. Is there a good reason to package these subsets but not the whole googleapis repo, or was it just expedient? Are they both being kept in sync upstream? Trying to build and install the full set I have these packages and install directories: Do they make sense to people? googleapis-dev (C/C++) /usr/include/google/ libgoogleapis-java Where should this go? Java policy suggests libgoogleapi-0.jar in /usr/share/java/? (I have no idea what the version number of this package is - googleapis repo doesn't seem to go in for versions/releases). python3-googleapis /usr/lib/python3/dist-packages/google? ruby-googleapis /usr/lib/ruby/vendor_ruby/google php-googleapis Where does php look for libraries/headers/definitions? csharp-googleapis And csharp? So, this seems to be quite a large rabbithole I have fallen down. I'm still not sure that this is the right approach, but it seems like this supplies the general case, rather than various subsets, so is probably worth doing. But I could be confused as I don't actually know how this stuff is used in practice. I've cc:ed maintainers for some of the packages mentioned above as they presumably know more about what is what. The salsa repo is out of date as send this. I'll update it to match my state tomorrow. Wookey -- Principal hats: Linaro, Debian, Wookware, ARM http://wookware.org/
Attachment:
signature.asc
Description: PGP signature