[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: RFC: Standardizing source package artifacts build paths



On Mon, 09 Mar 2020 at 10:09:46 +0100, Guillem Jover wrote:
> We'd like to standardize on a new set of artifact build pathnames
> for our deb toolchain. These would have the following form:
> 
>   - debian/.build/upstream*
> 
>     These could be used for out-of-tree builds replacing current
>     build-tree/ and similar directories.

Presumably the '*' means it's considered OK for packages that do separate
production/debug/udeb builds to use multiple directories matching that
glob?

For example, a simple debhelper build might eventually (in a future compat
level) default to "dh $@ --builddirectory=debian/.build/upstream",
but because src:glib2.0 does separate deb and udeb builds
(with non-essential features disabled in the udeb), it
would use "--builddirectory=debian/.build/upstream-deb"
and "--builddirectory=debian/.build/upstream-udeb" instead
of its current "--builddirectory=debian/build/deb" and
"--builddirectory=debian/build/udeb"?

>   - We could just (say) .gitignore a single pathname.
>   - Cleaning the Debian artifacts would imply doing just
>     «rm -rf debian/.build/» (even though this could be abstracted via a
>     new dpkg-<something>clean<something> command).
>   - Namespacing would avoid any collisions.
>   - Would make parallelization way easier and faster.
>   - dpkg-dev tools could start defaulting to use package specfific
>     directories, instead of requiring the user/tool to specify them.

This all seems good to have.

The Salsa-CI pipeline (which runs in a git clone of a package, and
currently uses debian/output/ for intermediate build stuff) could maybe
also move its scratch directory to debian/.build/salsa-ci.

Have you seen <https://bugs.launchpad.net/launchpad/+bug/1845159> and
related discussions in the context of sbuild and pbuilder? It seems
like a related topic. Briefly, there's a desire to be able to have
"artifacts" that are not part of the main build result, are produced by
both successful and failing builds, are kept for a while by the buildd
infrastructure, and are probably not uploaded to the Debian archive,
as a generalization of the build log.

For example, Autotools packages have ${builddir}/config.log, which is not
a useful thing to publish in a .deb (not at all reproducible!), but can
be useful to diff between successful and failing builds to debug why a
build failed. At the moment there's a special case in debhelper to cat
config.log if configure fails, but it is forgotten about after the build
if configure succeeds. Meson has ${builddir}/meson-logs/meson-log.txt,
which is analogous.

Similarly, Autotools packages normally output a test summary on success
or a more detailed log on failure (${builddir}/**/test-suite.log
describes all tests that failed, were partially or entirely skipped,
or were otherwise not 100% successful, and there are also detailed logs
that capture diagnostics from even the completely successful tests),
and Meson ${builddir}/meson-logs/testlog.txt is analogous.

At the moment the only artifact, other than the .changes file and the
files it references when the build is successful, is the build log itself,
which means the build process must either cat these other artifacts to
get them into the build log (resulting in large build logs where it's
difficult to find what you're looking for), or lose them forever.

Some packages have machine-readable test output that is used to track
regressions and intermittent failures between builds, for which cat'ing
the test output into the build log is less than ideal: to compare test
results between runs it's necessary to screen-scrape the test output
from among the unstructured/human-readable messages that make up most
of the build log. A few packages work around this by packaging their
test results, which usually makes them non-reproducible.

autopkgtest's ${AUTOPKGTEST_ARTIFACTS} (a directory into which tests are
encouraged to write or copy more detailed results than the ones shown
on stdout) and Gitlab-CI's artifacts:paths: parameter (a list of globs
matching paths in the built directory) are conceptually similar ideas.

For example, dpkg-buildpackage could perhaps read one glob per
line from debian/artifacts and hardlink matched files (if any) into
debian/.build/artifacts for collection by a "larger" framework like
sbuild or pbuilder, or individual packages could copy/link files into there
as they go, or debhelper build-system classes like Autotools and Meson
could know the names of common log files from their build system, or
some combination of those.

    smcv


Reply to: