[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Doing clean builds

So bazel has a fancy caching and dependency mechanism so that rebuilds
are quick.  Very nice for developers, but less handy for debian
packaging where we want a full, clean, determinstic, reproducible,
build every time.

The existing (tensorflow) packaging pattern, (which I am just
re-using, assuming that it is a sensible plan, rather than fully
understanding and designing from scratch) works like this:

export HOME=/tmp
(with the comment # not $(CURDIR), that leads to recursive symlinks)
       bazel build \
                -k $(BAZELOPTS)\
                --verbose_failures \
                --repository_cache= \
                --config=opt \
                --distdir=./debian/dist \
                --repo_env=TF_SYSTEM_LIBS=nsync,curl,double_conversion,snappy,gif,zlib,com_google_protobuf,com_github_grpc_grpc,jsoncpp_git,libjpeg_turbo,nasm \
                --override_repository=bazel_skylib=$(CURDIR)/debian/mock_repos/bazel_skylib \
                --override_repository=rules_cc=$(CURDIR)/debian/mock_repos/rules_cc \
                --override_repository=rules_java=$(CURDIR)/debian/mock_repos/rules_java \

$(BASELOPTS) is there to enabled or disable -s for verbose logs,
according to DEB_BUILD_OPTIONS=terse or not, for example.

So the general concept is to use /tmp as the home dir, disable the
repository cache, set up a local set of repositories for basic cc and
java and skylib (what is that?) rules and a list of libraries to come
from system libs.

Perhaps we should have a --nohome_rc in there to stop local config
changing the build?  But maybe that's convenient for maintainers, and
we can reply on it being empty in a clean sbuild, so no need for this.

This is all fine and works. (I've just realised that this may well be
why the cc stuff is building but not the c stuff - no set of base

However whilst this all works once, it does not work a second
time. Subsequent builds just find everything is in the cache and build
nothing. So we need the 'clean' rule to actually clean things up for a
clean, determinstic, build.

bazel clean
doesn't seem to do much, and is intended to leave the cache intact.
the docs say that
bazel clean --expunge
should remove the cache

However putting that in the clean rule:
        bazel clean --expunge
        rm -rf /tmp/

Does not result in the cache actually being removed.
I have to do
rm -rf /tmp/.cache/bazel
so _really_ clean.

So what am I doing wrong? Am I misunderstanding or is there a mistake
preventing bazel clean --expunge from working?

And are we happy with this overall approach to doing bazel builds?  Is
every package going to need a set of boilerplate language rules in a
local (CURDIR)/debian/mock_repos/ tree with corresponding
--override_repository config?

Perhaps that is specific to tensorflow? I am just using it as an
example here because I don't think we have any others yet (do we? I
should probably be looking at them if we do).

Also, at some point we are going to have to teach debhelper about a
--bazel mode, which may obviate the need for some of this. Perhaps
some of you have thought about this already?

Principal hats:  Linaro, Debian, Wookware, ARM

Attachment: signature.asc
Description: PGP signature

Reply to: