[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Fwd: Packaging TensorFlow for Debian



On 2021-05-28 11:29 +0000, M. Zhou wrote:
> Hi Wookey,
> 
> Thanks for your work and the updates. I think I can commont
> on some of the details.
> As I had ever written a hacky build system for tensorflow,
> I still remember some details of it.
> 
> On Fri, 2021-05-28 at 02:07 +0100, Wookey wrote:
> > 
> > * Collected .h files into -dev package (this is done horribly with
> >   rsync because tensorflow/bazel doesn't have a 'make install' I can
> >   just use - but it does know the list of headers so I'm sure there
> > is
> >   a better way).
> 
> IIRC the list of header files that should be installed varies
> according to the configuration. For example, if we enable the
> component A, but disable the component B when compiling the shared
> objects, then headers for component B should be filtered out.
> 
> Bazel is able to resolve the file-level dependency for a given
> target, where the "files" surely includes headers.

Yes, there should be a list of headers specific to the build made. I've
just taken the blunderbuss aproach of copying in all the .h files,
so they should all be available, but some are not relevant.

I don't _think_ it will actually break anything (because the headers are additive), BICBW.

> We can dump that dependency using bazel, and grep the headers out.
> I've forgotten the concrete command to achieve that. Oops.

That sounds like a sensible plan.

What we really want is a generic 'list the files/locations you would
have installed', then we can use that for all install targets. Bazel
can't actually just do 'make install' because of its philosophy of
being sandboxed. It can make tarballs though. And maybe we can just
unpack those in debian/tmp and then proceed as normal? That would be a
generic solution to all debian builds with bazel.

> > What guarantees does upstream make about backwards/forwards
> > compatibility? They are putting SONAMEs in and managing major, minor,
> > patch versioning, which is better than many projects these days.
> > 
> > I'm wondering what the right strategy is for abi/api versioning. I
> > presume we will have quite a lot of packages using this so we should
> > try and do it right.
> 
> If the upstream tightly stick to the semantics versioning, we should
> probably directly use the upstream SONAMES.

I am using the major SONAME version in the library names (indeed
upstream buld does this for us, so v2.3.1 builds libtensorflow_cc2 and
libtensorflow_framework2. So that's fine, and allows co-existing
library binary packages, in exchange for having to update the major
version number in various places in the packaging and go through NEW
on major version bumps. It would be nice if it was easier to
parameterize the major version in the control and install files so
that updates were reliable (it's too easy to miss one when updating in
my experience). Maybe debhelper could provide a variable for this as
loads of people must need this functionality?

I guess I'm wondering about the more detailed versioning you get from
symbol files tracking aditions/deletions on minor versions. I've never been
quite sure how useful this was in practice.

> > However then this question of ABIs gets sidetracked by something I
> > noticed whilst looking at the symbols situation: The symbols file for
> > libtensorflow_cc2 is 24MB (that's really quite fat) Is it worth
> > putting that in the package? I'm not sure anyone is going to actually
> > 'maintain' it beyond autogenerating a new one each version. Symbols
> > files work OK for C but are bloated and awkward for C++. Even so 24MB
> > seems huge.
> 
> C++ symbols are known to be hard to track. As currently we don't
> expect many reverse dependencies of libtensorflow, maybe we should
> not track it manually, at least for now.

Any differing opinions in the crowd?

> > lintian only complained about an embedded libpng, but now
> > I look I am pretty sure there is a still a range of embedded
> > statically-linked libs hiding in there.
> > 
> > We have lots of symbols like:
> > ZN6google8protobuf3MapINSt7__*
> > _ZN4absl14lts_2020_02_*
> > AES_decrypt@Base
> > BORINGSSL_self_test@Base
> > _ZN3Aws22AmazonWebService*
> 
> Maybe it's using static libraries for some of them?

I'm pretty sure it is. 

> > what is our view on uploading sooner vs expunging all embedded
> > libs?
 
> My recommendation is to pass the NEW queue first. Because it is
> expected to stay in experimental for a while, and the first upload
> could enhance everybody's morale.
> 
> BTW, please make sure to separate the tensorflow shared objects
> into separate binary packages, e.g.
> 
> bin:libtensorflow-framework.*
> bin:libtensorflow.*
> bin:libtensorflow_cc.*
> bin:libtensorflow-dev

Yep. That is already done.

> This is because some customized Ops/Kernels only NEED
> libtensorflow_framework.so.*

What exactly does the framework library do? (As opposed to the
libtensorflow and libtensorflow_cc libraries?)

Wookey
-- 
Principal hats:  Linaro, Debian, Wookware, ARM
http://wookware.org/

Attachment: signature.asc
Description: PGP signature


Reply to: