Bug#842301: autopkgtest: please capture the testbed's systemd Journal if present

[mpitt@debian.org (Martin Pitt)]

Control: tag -1 wontfix

Hello Simon,

Simon McVittie [2016-10-27 21:20 +0100]:
> Tags: patch

You forgot to actually attach the patch; but..

> When testing system-level things, the test's stdout/stderr don't always
> tell the full story: after a failure, it's often useful to read the
> logs of the service under test (and related services).

Being the test infra maintainer in Ubuntu I look at a loooot of
failures; for very few of them the journal would indeed be helpful,
but for the vast majority it's not.

My main reservation about this is that this comes with a price: The
journal is fairly big -- it can easily be (and usually is) 10 to
10,000 times the size of the default stdout/err artifacts, and with
hundreds of thousands of test runs and their results this would amount
to a significant increase of storage space that
ci.debian.net/autopkgtest.ubuntu.com had to deal with.

IMHO, for the few cases where a journal is helpful, I rather propose:

 * Tests that fail often and don't have useful output can do something
   like "journalctl -b > $AUTOPKGTEST_ARTIFACTS/journal.txt" on
   failures themselves. This avoids needlessly collecting the big
   journal on the thousands of perl, python, GUI, KDE, or PostgreSQL
   tests which provide enough information by themselves and having the
   journal doesn't help at all with debugging them.

 * For interactively investigating a failure we already have
   --shell-fail/-s. For tests of the above "close to the OS/brittle"
   kind even the journal often isn't enough, and one needs to poke
   sysfs, network status, or run a single test in a loop to debug a
   race condition.

 * It was requested some time ago to add a --teardown-commands which
   would run right before closing a testbed [1]. CI environments could
   then use that to opt into collecting journals, core dumps, sysfs
   dumps, and anything else that they like.

So for now I tag this as "wontfix". Sorry if that's disappointing!

Thanks,

Martin

[1] https://launchpad.net/bugs/1288529

-- 
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)