[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Interested in Debian Bazel efforts



Hi Yun and Olek,

> That's a good point. The libc version change isn't detected by Bazel, either. Which does require users to run bazel clean sometimes. I guess we can start with the approach of using -L, -I flags first, then try to improve hermeticity later.

I totally agree. I will see what I can do.

I noticed that the current version of `bazel-bootstrap` is 3.5.1. Yun, may I ask how long will it be supported? If I remember correctly, Bazel follows the "live at head" approach, and only provides extended support for a few months. As a result, unfortunately, the current version would not be supported for the entire bullseye cycle, and, at this stage, I don't think Debian has enough resources or expertise to support a Bazel version that is deprecated by upstream.

However, I noticed that Bazel now provides LTS releases, starting with 4.0 [1]. 4.0 is supported until Dec 2023, and we can work on it for years to come. Olek, would it be possible to bump Bazel to 4.0? By the way, I tried to make a 4.0 package [2], and it appears to work, can you review it when you have time?

Thanks,

Jesse.

[1]: https://blog.bazel.build/2020/11/10/long-term-support-release.html

[2]: https://salsa.debian.org/jc/bazel-bootstrap

On 4/9/2021, Yun Peng wrote:
Hi Jesse,

> Maybe Olek can give more insights.  Though, in my opinion, it would be challenging to implement this for all kinds of package managers. Or we can simply track the files used by the compiler? Don't Bazel projects use libc of system? How would Bazel handle a libc/libstdc++ update right now?

That's a good point. The libc version change isn't detected by Bazel, either. Which does require users to run bazel clean sometimes. I guess we can start with the approach of using -L, -I flags first, then try to improve hermeticity later.

Cheers,
Yun


On Fri, Apr 9, 2021 at 2:47 AM Jesse Chan <jc@linux.com> wrote:

Hi Yun,


I agree.

I understand that it would not be ideal, but it should work fine with CI. To be safe, maybe we can simply track the entire `/usr/lib` and `/usr/include` directories. After all, contents of those directories are not changed unless the user does something.

To grab the list of related files:

```

$ dpkg -S /usr/lib/x86_64-linux-gnu/pkgconfig/zlib.pc

zlib1g-dev:amd64: /usr/lib/x86_64-linux-gnu/pkgconfig/zlib.pc

$ dpkg -L zlib1g-dev:amd64

... (https://packages.debian.org/sid/amd64/zlib1g-dev/filelist)

```

Maybe Olek can give more insights.  Though, in my opinion, it would be challenging to implement this for all kinds of package managers. Or we can simply track the files used by the compiler? Don't Bazel projects use libc of system? How would Bazel handle a libc/libstdc++ update right now?


Thanks,

Jesse.

On 4/8/2021, Yun Peng wrote:
Hi Jesse,

In my view, distrib_* macros are only used to switch between building from source and building against system libraries. I expect many Bazel users would still like to have their dependencies built from source by default, because that makes the build hermetic and reproducible. However, we should provide an easy way to switch to using system libraries. The proposed rule that parses the .pc file indeed will be very helpful, we can maybe call it something like sys_cc_library.

I just checked the zlib.pc, it only contains the directories of the headers and libs, not the list of files. This should work, but isn't ideal for Bazel. Because if we only pass -L -I options, Bazel won't notice any change and rebuild when zlib gets updated. It's better to declare the headers and libs explicitly.
Is there any way we can get the full list of files, something like https://packages.debian.org/sid/amd64/zlib1g-dev/filelist

> The new dependency system looks very promising. IMO it deals with the
most annoying problems of Bazel. I hope that it can be implemented soon.

Working hard on that!

Cheers,
Yun


On Thu, Apr 8, 2021 at 1:58 AM Jesse Chan <jc@linux.com> wrote:
Hi Yun,

pkgconfig is supported by autotools and CMake. Ideally, we can have a
`distrib_cc_library` that has only one required argument, `name`. For
instance, when it is `zlib`, Bazel can parse
`/usr/lib/x86_64-linux-gnu/pkgconfig/zlib.pc` to get necessary include
directories, cflags and lib paths. We have to write Bazel rules that
generate pc file, and we would want to have rules that install library
targets along with generated pc file (something like "make install", but
for Bazel).

The new dependency system looks very promising. IMO it deals with the
most annoying problems of Bazel. I hope that it can be implemented soon.

Thanks,

Jesse.

> Hi Jesse,
>
> I'm from the Bazel team and working with Olek on the Bazel Debian
> effort. I'll be very glad to have your help!
>
> > Yeah. However, in practice, it is difficult to set it to `False`. I
> got something like this:
>
> linkstatic = False will force your cc_binary to dynamically link to
> all dependencies whenever possible. If your dependencies are built
> from source, then the binary won't work outside of the Bazel
> working directory.
>
> > However, I noticed that they are kinda tailored/hacked for Debian,
> and they are not really simpler than manual `cc_import` or
> `cc_library` at the moment.
>
> Yes, in fact, the purpose of the distrib_* macros are just to redirect
> targets to the ones in the repo (@debian_cc_deps) for the Debian
> libraries, because we have to switch between building Bazel from
> checked-in source and building Bazel with Debian system libraries.
>
> > I think we can definitely make things easier. Perhaps we can have
> some rules that generate something like `*.pc (pkgconfig)` to install
> to system, so they can be parsed and turned into Bazel targets later.
> Then we can override targets of a project with system-provided ones,
> via naming conventions or mappings or patches.
>
> I like this idea. If you can write a rule to automatically generate
> BUILD files for the system libraries, that already solves half of the
> problem. The other half, as you mentioned, is mapping the targets to
> the Debian ones. The problem is how to make it effortless for users.
>
> Currently, users have to write their own BUILD files for building
> non-Bazel dependencies from source. Check TensorFlow's third party
> directory:
> https://github.com/tensorflow/tensorflow/tree/master/third_party
> <https://github.com/tensorflow/tensorflow/tree/master/third_party>
> We can put something like the distrib_* macros in those BUILD files,
> and users can switch to build with system libraries with just one flag
> (eg. --define=distribution=debian).
>
> But it still requires users to write those BUILD files by hand while
> they can actually be shared. This is where Bazel's new dependency
> management system
> <https://docs.google.com/document/d/1moQfNcEIttsk6vYanNKIy3ZuK53hQUFq1b1r0rmsYVg/edit#heading=h.e6t527rxhw5i>
> could help. We plan to have a Bazel Central Registry that can host
> those BUILD files for non-Bazel projects, so that users don't have to
> write their own. And this is probably a good place to do the target
> mappings. However, the new system is a big design that also aims to
> solve other external dependency problems in Bazel, it will still take
> some time to be available.
>
> Cheers,
> Yun
>
>
>
>
>

Attachment: OpenPGP_0xA102C2F15053B4F7.asc
Description: OpenPGP public key

Attachment: OpenPGP_signature
Description: OpenPGP digital signature


Reply to: