Bug#41902: marked as done ([OLD PROPOSAL] Test suite proposal)
Your message dated Wed, 13 Jun 2001 13:16:54 -0500 (CDT)
with message-id <20010613181654.B82BC4711@speedy.private>
and subject line Bug #41902: Test suite proposal
has caused the attached Bug report to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what I am
talking about this indicates a serious mail system misconfiguration
somewhere. Please contact me immediately.)
Darren Benham
(administrator, Debian Bugs database)
--------------------------------------
Received: (at submit) by bugs.debian.org; 25 Jul 1999 19:27:19 +0000
Received: (qmail 13912 invoked from network); 25 Jul 1999 19:27:19 -0000
Received: from chiark.greenend.org.uk (mail@195.224.76.132)
by master.debian.org with SMTP; 25 Jul 1999 19:27:19 -0000
Received: from 379b6508.ea9d1.2207.0.bsmtp.davenant.greenend.org.uk by chiark.greenend.org.uk with local-bsmtp (Exim 2.02 #1)
id 118Tva-0002Gu-00 (Debian); Sun, 25 Jul 1999 20:27:18 +0100
Received: from anarres.relativity.greenend.org.uk [172.18.45.2] (mail)
by davenant.greenend.org.uk with esmtp (Exim 2.125 #2)
id 118SRu-0002CK-00 (Debian); Sun, 25 Jul 1999 18:52:34 +0100
Received: from ian by anarres.relativity.greenend.org.uk with local (Exim 2.125 #2)
id 118SRu-0002Ot-00 (Debian); Sun, 25 Jul 1999 18:52:34 +0100
From: Ian Jackson <ian@davenant.greenend.org.uk>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Message-ID: <14235.20194.85470.740192@anarres.relativity.greenend.org.uk>
Date: Sun, 25 Jul 1999 18:52:34 +0100 (BST)
To: submit@bugs.debian.org
Subject: Test suite proposal
X-Mailer: VM 6.62 under Emacs 19.34.1
Package: debian-policy
Version: 3.0.0.0
I think most of us will agree that we need some kind of way of
distributing and automatically using regression test suites. We need
to do this in a way that allows people to download just the .deb and
the regression tests, without the full source archive, and we need to
do it in a way that support automatic tests of various kinds.
Regression tests for many of our programs will be hard to execute
under some circumstances, because they may want to be root, change
configurations of things, etc.
I suggest that we come up with a standard scheme for
(a) distributing the tests
and
(b) expressing which ones are appropriate to run.
Regarding (a), I think a separate part of the source package is
appropriate. I suggest
<package>_<version>.tests.tar.gz
which contains a directory
<package>-<version>-tests/
This is supposed to work when unpacked anywhere.
Inside <package>-<version>-tests there should be some indication of
what tests are available. I suggest a directory `controls' or some
such, containing one file per test.
Each file would contain some basic information about the test:
* Whether it needs to run as any particular user, and if so which.
* How much change it will make to the system, chosen from perhaps
- none visible
- reconfigure or change systemwide data belonging to the package in
question or some closely related package (scorefiles, logfiles
etc. do not count unless they're deleted or stuffed full of junk)
- reconfigure or change systemwide configuration belonging to other
packages, or shared between packages. This includes randomly
installing other packages (except test packages if they are
harmless and are removed again afterwards). It also includes
rebooting the machine or anything like that.
* Whether the test is supposed to work given only the package and its
dependencies, or whether it would need the Recommends as well.
* Whether the test requires a tty and/or an X display, and whether it
interacts with the user. (NB a package may be noninteractive but
require X and/or a tty).
* Other package(s) which must be installed for the test to be
meaningful, or which must not be installed. (Depends/Conflicts)
* Any other things we think are relevant here. The format must be
extensible.
* A shell script to run. The current directory will be set to the
base of the test suite. The test script is allowed to use any file in
the test suite, and is allowed to create files etc. inside the test
suite. The test is considered to have succeeded if this script exits
with a zero exit status and without printing anything on stderr.
There should also be a `cleanups' directory in the same format, but
whose scripts just clean up the side-effects of the tests (even if the
tests failed), if possible. This includes deleting any files the
tests created.
To assist in writing test scripts, there should perhaps be support
package(s) which contain useful utilities for (eg) applying regexps to
the output of things, etc.
Ian.
---------------------------------------
Received: (at 41902-done) by bugs.debian.org; 13 Jun 2001 18:16:58 +0000
>From steveg@molehole.dyndns.org Wed Jun 13 13:16:57 2001
Return-path: <steveg@molehole.dyndns.org>
Received: from 206.180.143.9.adsl.hal-pc.org (speedy.private) [::ffff:206.180.143.9]
by master.debian.org with esmtp (Exim 3.12 1 (Debian))
id 15AFCL-0003UB-00; Wed, 13 Jun 2001 13:16:57 -0500
Received: by speedy.private (Postfix, from userid 1000)
id B82BC4711; Wed, 13 Jun 2001 13:16:54 -0500 (CDT)
To: 41902-done@bugs.debian.org
Subject: Re: Bug #41902: Test suite proposal
Message-Id: <20010613181654.B82BC4711@speedy.private>
Date: Wed, 13 Jun 2001 13:16:54 -0500 (CDT)
From: steveg@molehole.dyndns.org (Steve Greenland)
Delivered-To: 41902-done@bugs.debian.org
This note is being sent as part of a project to clean out old (> 1yr)
debian-policy proposals. If you disagree with action below please
respond to bug#@bugs.debian.org, not to me, so that the discussion may
be carried out publically in debian-policy. Feel free to re-open the
bug while it's being discussed -- I'm not trying to force any
particular disposition, just taking my best shot at resolving dead
issues.
Bug #41902: Test suite proposal
Summary: An proposal for providing regression tests alongside the
Debian packages: what they should do, what they shouldn't do, etc.
No comments. No reply from submitter when pinged 6 months later.
Discussion: An interesting document, but apparently a dead
proposal. Since we do archive closed bug reports now, I'm just going
to close it.
Action: close
Reply to: