[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: package testing, autopkgtest, and all that

Lucas Nussbaum writes ("Re: package testing, autopkgtest, and all that"):
> If there's a packaged tool to run the test suite on a given package,
> then it's quite easy to integrate it into my infrastructure. But I
> clearly do not have the time to get autopkgtest's code back in shape
> first.

Yes, there is such a packaged tool, "adt-run".  It worked last time I
tried it.

The options are a bit complicated because you have to specify exactly
what you want to do; the script "adt-testreport-onepackage" is what
I was using to invoke adt-run, and it runs it in broadly speaking two

  adt-run --source <some source package> ---...
     to check that the source builds with dependencies satisfied
     from the archive, and if it contains tests, run those tests
     with the resulting binary packages

  adt-run --binaries=install --binary <some binary package> ---...
     to check that the package in question is installable

But another useful one would probably be:

  adt-run --built-binaries-filter=_ --source <some source package> ---...
     to check that the package's tests pass with binary packages
     currently in the archive (which almost other things tests that
     the package's dependencies haven't changed so as to break it)

I've enclosed a copy of the formatted manpage for adt-run, below.  It
seems to be missing the --instantiate option, whose usage message is:
     instantiate testbed now (during testing phase) and install
     packages selected for automatic installation, even if this might
     apparently not be required otherwise

> Well, it has rarely worked like that. Most of the time, I just do the
> log analysis + bug filing alone. That means that the tool to run the test
> suite must be built with filing bugs in mind: it should provide all the
> needed info in the logfile, so that developers can easily reproduce the
> failure without asking the bug reporter.

It's possible that autopkgtest could do with some work in this area;
currently the relevant output might be in more than one logfile.


adt-run(1)                 Linux Programmer's Manual                adt-run(1)

       adt-run - test an installed binary package using the package's tests

       adt-run options...  --- virt-server [virt-server-arg...]

       adt-run  is  the  program  for invoking the autopkgtest package testing

       autopkgtest is a facility for testing binary packages, as installed  on
       a  system  (such as a testbed system).  The tests are those supplied in
       the source package.

       adt-run runs each test supplied by a particular package and reports the
       results.  It drives the specified virtualisation regime as appropriate,
       and parses the test description metadata, and arranges for data  to  be
       copied to and from the testbed as required.

       adt-run should be invoked (unless options to the contrary are supplied)
       in the top level directory of the built source tree, on the host.   The
       package should be installed on the testbed.

       --built-tree directory
              Specifies that tests from the built source tree directory should
              be run.  Note that the packages that would normally be installed
              as a result of * in the tests' Depends field (which includes the
              case  where  the  Depends  field  is  not  specified)  are   not
              installed.   The  caller  must  explicitly  instruct  adt-run to
              install any relevant packages.

       --source dsc
              Builds dsc.  The resulting binaries will (by default) be used to
              satisfy  dependencies.  The tests from that built tree will also
              be run (by default).  The ordering is significant: each --source
              option  should precede options whose dependencies are to be sat-
              isfied by the binaries it produces.

       --unbuilt-tree directory
              Specifies that tests from  the  unbuilt  source  tree  directory
              should  be  run.   This  is  very  similar to specifing --source
              except that a directory tree (which should be pristine) is  sup-
              plied, instead of a source package.

       --binary deb
              Specifies  that  deb should be used.  By default it will be used
              to satisfy dependencies, both during building and  testing,  but
              not  necessarily installed.  The ordering is significant, as for

              Bare  filename  arguments  are  processed  as  if  --built-tree,
              --source,  --unbuilt-tree  or --binary was specified; the nature
              of the argument is guessed from the form of  the  filename.   In
              the  case  of --built-tree, either the option must be specified,
              or the filename must end in a slash; two slashes at the end  are
              taken to mean --unbuilt-tree.

       These  affect modify processing instructions.  Unless stated otherwise,
       they affect all subsequent options.

              Specifies that subsequent pathnames  in  command-line  arguments
              refer  to  files  on  the testbed, or on the host, respectively.
              The default is --paths-host.

              Specifies that the tests in subsequent --source  and  --unbuilt-
              tree arguments should (or should not) be run.

              Specifies  that  only  binaries whose package names match one of
              the specified patterns should be used; others will  be  ignored.
              This  option  applies  to subsequent --source and --unbuilt-tree

              Specifies that all built binaries should be ignored  completely;
              equivalent  to  --built-binaries-filter=_ (since no package name
              ever contains _).

       --binaries=ignore | --binaries=auto | --binaries=install
              Specifies  that  binary  package  (in   subsequently   specified
              --binary  arguments,  or  resulting  from subsequently specified
              --source or  --unbuilt-tree  arguments  and  not  filtered  out)
              should  be  ignored,  used  only  to  satisfy  dependencies,  or
              installed unconditionally, respectively.  Equivalent to specify-
              ing both --binaries-forbuilds and --binaries-fortests.

              Like  --binaries=  but  only changes the handling during package
              building: packages will be ignored, used  for  dependencies,  or
              unconditionally installed, when a source package is built.

              Like  --binaries=  but only changes the handling during testing:
              packages will be ignored, used for  dependencies  (including  as
              the  package  under  test),  or  unconditionally installed, when
              tests  are  run  (as  a  result  of  --source,  --built-tree  or

       --output-dir output-dir
              Specifies that stderr and stdout from the tests should be placed
              in output-dir.  These  files  are  named  argid-test-stderr  and
              argid-test-stdout  for each test test, and log for the log tran-
              script.  If no output-dir is specified, or the path is specified
              to  be  on  the  testbed  (ie,  if --output-dir follows --paths-
              testbed), then the log file is instead written to the  temporary
              directory  tmpdir if one was specified, or otherwise no separate
              copy is made.  Note that the log transcript output will also  be
              sent to adt-run's stderr unless --quiet is specified.

              Run builds and tests as user on the testbed.  This needs root on
              the testbed; if root on the testbed is not available then builds
              and tests run as whatever user is provided.

              Prefixes  debian/rules  binary  with  gain-root.Thedefaultisnot-
              touseanything,exceptthatif --user is supplied  or  root  on  the
              testbed is not available the default is fakeroot.

              Specifies  that  tmpdir should be used instead of a fresh tempo-
              rary directory on the host.  tmpdir will be  created  if  neces-
              sary,  and emptied of all of its contents before adt-run starts,
              and it will not  be  cleaned  out  afterwards.   tmpdir  is  not
              affected  by  --paths-testbed.   NOTE again that all of the con-
              tents of tmpdir will be deleted.

              Specifies that the  trace  log  should  be  written  to  logfile
              instead  of  to  log  in  output-dir or tmpdir.  log-file is not
              affected by --paths-testbed.

              Specifies that a summary of the outcome  should  be  written  to
              summary.   The  events  in the summary are written to the log in
              any case.  summary is not affected by --paths-testbed.

              Use a different timeout for operations on or with  the  testbed.
              There are four timeouts affected by four values of which: short:
              supposedly short operations like setting up  the  testbed's  apt
              and checking the state (default: 100s); install: installation of
              packages including dependencies (default: 3ks); test: test  runs
              (default:  10ks); and build: builds (default: 100ks).  The value
              must be specified as an integer number of seconds.

              Multiply all of the default timeouts  by  the  specified  factor
              (see  --timeout-which  above).   Only the defaults are affected;
              explicit timeout settings are used exactly as specified.

              Include additional debugging information in the trace log.  Each
              additional -d increases the debugging level; the current maximum
              is -ddd.  If you like to see what's going on, -d or -dd is  rec-

              Uses  dir  as  the GNUPGHOME for local apt archive signing.  The
              specified directory should not contain keyrings containing other
              unrelated keys, since adt-run does not specify to gpg which keys
              to use.  The default is $HOME/.autopkgtest.  --paths-testbed has
              no effect on this option.

              Use  a  fresh  temporary  directory and generate fresh keys each
              run.  This can be very slow and depends on the  availability  of
              sufficient quantities of high-quality entropy.

       -q | --quiet
              Do not send a copy of adt-run's trace logstream to stderr.  This
              option does not affect the copy sent to logfile,  output-dir  or
              tmpdir.   Note  that  without the trace logstream it can be very
              hard to diagnose problems.

       --- virt-server virt-server-arg...
              Specifies the virtualisation regime server,  as  a  command  and
              arguments  to  invoke.   All the remaining arguments and options
              after --- are passed to the virtualisation server program.

              When running commands on the testbed, sets the LANG  environment
              variable  to langval.  The default in adt-run is to set it to C.

              Suppresses the setting by adt-run of LANG on the testbed.   This
              results  in tests and builds using the testbed's own normal LANG
              value setting.

       During a normal test run, one line is printed for each test.  This con-
       sists  of  a short string identifying the test, some horizontal whites-
       pace, and either PASS or FAIL reason or SKIP reason where the pass/fail
       indication is separated by any reason by some horizontal whitespace.

       The string to identify the test consists of a short alphanumeric string
       invented by adt-run to distinguish  different  command-line  arguments,
       the argid, followed by a hyphen and the test name.

       Sometimes  a  SKIP  will  be  reported when the name of the test is not
       known or not applicable: for example, when there are no  tests  in  the
       package, or a there is a test stanza which contains features not under-
       stood by this version of adt-run.  In this case * will appear where the
       name of the test should be.

       If  adt-run  detects  that  erroneous  package(s) are involved, it will
       print the two lines blame: blamed-thing...  and badpkg: message.   Here
       each  whitespace-separated  blamed-thing is one of arg:argument (repre-
       senting a pathname found in a command line  argument),  dsc:package  (a
       source  package  name), deb:package (a binary package name) or possibly
       other strings to be determined.  This indicates which arguments  and/or
       packages  might  have  contributed  to the problem; the ones which were
       processed most recently and which are therefore most likely to  be  the
       cause of a problem are listed last.

       0    all tests passed
       1    unexpected  failure (the python interpreter invents this exit sta-
       2    at least one test skipped
       4    at least one test failed
       6    at least one test failed and at least one test skipped
       8    no tests in this package
       12   erroneous package
       16   testbed failure
       20   other unexpected failures including bad usage

       adt-virt-chroot(1), adt-virt-xenlvm(1)

       This tool still lacks some important features and  is  not  very  well-

       This  manpage  is part of autopkgtest, a tool for testing Debian binary
       packages.  autopkgtest is Copyright (C)  2006-2007  Canonical  Ltd  and

       See /usr/share/doc/autopkgtest/CREDITS for the list of contributors and
       full copying conditions.

autopkgtest                          2007                           adt-run(1)

Reply to: