Re: Extending the build profile namespace
On Mon, Apr 25, 2016 at 01:44:01PM +0100, Ian Jackson wrote:
> Helmut Grohne writes ("Extending the build profile namespace"):
> > * The nodoc profile is a bit strange. It is supposed to drop
> > documentation from packages or to drop documentation packages. The
> > former leads to packages whose content varies with profiles (which
> > generally is bad)
>
> I think there is nothing wrong with a build profile producing the same
> packages with different content.
That can be ok, but it can also break dependency contracts. The ensuing
breakage is usually hard to debug. Now documentation is special here,
because the policy says that nothing must rely on /usr/share/doc/$pkg,
but when did the last archive rebuild validate that property?
> IMO someone (human or software) who specifies the use of a build
> profile is responsible for knowing the semantic effect of the profile.
That's what we currently do. Some profiles are recorded at
https://wiki.debian.org/BuildProfileSpec. Yet even that specification is
violated in a fair number of places already. This is an attempt at
gaining more precision.
> > and the latter mostly drops Arch:all packages, so
> > in many cases simply doing an arch-only build achieves the same
> > effect. If the only Architecture: all package from a source package
> > is a -doc package, then generally the nodoc profile is not necessary
> > (e.g. cargo and rustc). Just populate Build-Depends-Indep properly.
> > Maybe we should revise rules for this profile?
>
> I don't agree with this analysis at all. One thing you might want to
> do is deliberately run builds of many packages in a situation where
> you know you don't care about generating documentation (or would like
> to save the space consumed by documentation).
>
> For this to work, there has to be a way to specify such a build
> without knowing the details of the package.
The question here is where you put that effort. Either you "pollute"
many packages with nodoc options/profiles or you put that logic into the
builder. From my pov (as the one building many packages), putting a rule
"if all arch:all packages are in section doc, skip indep" is much easier
than patching tens or even hundreds of packages.
> ISTM that the nodoc build option should be phased out in favour of a
> build profile. For compatibility, the specification for the nodoc
> profile should say that nodoc builds should always be done with the
> nodoc build option too. dpkg-buildpackage could handle this.
I certainly wouldn't object such a transition. Still I think that we
need to improve the tooling before starting it. Adding support for a
nodoc profile currently involves a fair number of steps touching both
d/control and d/rules and potentially switching to dh-exec for using
profiles in d/*.install. Instead, generic helpers (e.g. dh_installdocs)
should support that use case.
For these reasons, I believe that the maintenance cost of nodoc
currently outweighs the benefits for many packages. Of course, I welcome
efforts to lower that cost. From a bootstrap perspective, this has
rather low priority though, so I won't be doing that.
> The stage* system avoids doing a lot of work to determine and describe
> precisely the situation in an abstract way; rather, it is an explicit
> and manually maintained bootstrap sequence (with a cross-archive
> representation).
Can you point me to that manually maintained bootstrap sequence? Who is
maintaining it? Where are those stages described?
Instead, I see that bootstrapping requires intimate knowledge of which
existing stages are useful and which ones are best ignored (either
because they are unneeded or broken). It also requires knowledge of
which necessary stages only exist as patches in the BTS.
> The benefit of the current stage profiles is
> that it is easy for a cross bootstrap orchestration tool to know what
> to do.
My experience with maintaining a cross bootstrap orchestration tool is
exactly the opposite and the reason for writing the mail you replied to.
> Your suggestion, if implemented, would:
>
> 1. Complicate the metadata for what is a pretty minority feature.
> This imposes more work on package maintainers in general and more
> work on those trying to get the boostrap to work.
No, it's about naming the profile. You can call it "stage1" or
"nopython". It's just a name. The only thing that really changes is that
instead of being a vague "stage1", it now comes with meaning and follows
a specification.
> 2. Require the implementation of a new bootstrap planner which would
> be able to mine the profile-specific build-dependency information
> to construct a bootstrap plan.
There is another reason to require a bootstrap planner: prevail sanity.
The days of manually ordering architecture bootstrap have long passed.
Everyone uses at least partially automatic ordering in one way or
another.
For instance, rebootstrap orders builds and decides which packages to
build by running dose-builddebcheck. That tooling allows selecting 30%
and ordering 70% of the packages (increasing as more maintainers apply
patches) even though it doesn't do any profile selection yet. The major
problem with adding profile selection currently is the lack of metadata
and well-defined semantics.
So in a way, I agree: It requires a bootstrap planner. Johannes Schauer
spent a gsoc and much more to lay the groundwork for it. Though it can
only start working once we clean the mess created by inconsistent stage
profiles with no meaning.
The hard part is not the writing of the planner. The hard part is
getting sufficiently correct and complete metadata for it to operate.
> 3. Still not address the `problem' you identify that the stage*
> dependencies implicitly violate dependency contracts. The only way
> to address this problem fully would be to provide explicit
> representation in the dependency system for interfaces like
> `fontforge with embedded python support' (or conversely for
> `fontforge without embedded python support'). I think this would
> be very complicated.
It practically addresses the problem of violating dependency contracts.
The profiles for disabling language bindings generally remove entire
packages. So these are "safe" profiles (i.e. not breaking dependency
contracts) most of the time.
Further down the road, we will need finer grained dependencies in some
places. For instance glibc will at some point "Build-Depends:
hurd-headers-dev [hurd-i386]" (this is already an alternative). It is
provided by hurd-dev in a normal system and built as a real package
during bootstrap.
I see this problem as fully addressed, because metadata can express the
underlying dependency contracts.
> 4. Items 2 and 3 above iteract very badly. I think it may be
> impossible to do 2 without also doing 3.
Agreed.
> > Furthermore, having a way to distinguish "safe" (package set changing)
> > from "unsafe" (content changing) profiles would be very helpful.
>
> I agree that this is a useful distinction (although I don't
> necessarily agree with your terminology). Given that you propose to
> allow package maintainers to make up random shit in their bit of the
> profile namespace, how do you imagine such a package maintainer
> publishing the information about the semantics of their build
> profiles, so that tools may find it ?
Technically, you could just say "XS-Safe-Profiles: stage1" in
debian/control. Maybe I should also start using just that as an
incentive to fix terminology. ;)
Note, that I generally consider cross, nocheck, and noudeb "safe".
Helmut
Reply to: