Re: regression
(quotes of Ben, James & Raul at end of mail)
I would suggest a smaller target first.
To make easier this release and all that will come
later we should make tests for *installation* of binary
packages. These would have small, definite goals
and big payback during release integration.
I think this would be the way to go, because:
- making a binary package is much more frequent
than making a source package
- since we are testing a package, not a program
the set of bugs is much more limited
- testing is likely be quite similar across
packages, so effort can be re-used many times
- these tests need not be integrated into
packages (initially)
- this answers partly to our immediate high priority
goal of getting a good debian 2.0
- this requires less of specialized testing knowledge
- it is a good learning experience with high
motivation and low resource demand (when compared
with comprehensive testing of arbitrary apps)
so, let us make this our first step on our
road to fully tested os with fully tested apps. ;-)
I have been slowly contemplating something like this:
- use dejagnu, that uses expect, which is based on tcl
- example package gnats (problem reporting system)
- run installation with dpkg, since it is cmd line
- check that no drastic happens during installation
- answer any questions (asks mail address for admin)
- check existence, locations and permissions of files
- check presence of crontab entries
- use at least once each exe
- send a problem report
- send-pr
- wait 15 mins (handling of pr:s every 10 mins)
- check that it has been placed into db
- check that sender and admin have received mail
...
etc. in similar fashion.
As you can see even this very limited goal of
building confidence that app that already works
really works also this time
requires a non-trivial amount of effort. Still, I
think that this effort is achievable for many
packages.
t.aa
Ben Armstrong, Nov 25: >
> On Mon, 24 Nov 1997, James R. Van Zandt wrote:
> > Raul Miller <rdm@test.legislate.com> writes:
> > >it occurs to me that we might
> > >want to include regression tests in the standard package build
> > >process.
> >
> > Excellent idea. The GNU coding standards suggest this Makefile target:
> >
> > `check'
> > Perform self-tests (if any). The user must build the program
> > before running the tests, but need not install the program; you
> > should write the self-tests so that they work when the program is
> > built but not installed.
> >
> > We could incorporate self-tests into the source package this way, and
> > invoke them from debian/rules. Ideally, this should be fed to the
> > upstream developer too.
>
> Sounds like a great idea. I've never written such a test before,
> however. So let's just take my package typist, for example, which
> is curses based. Should the self-tests be built using 'expect'?
> I'm definitely going to need to be educated in the tools before
> I can start adding self-tests to my packages.
--
TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word "unsubscribe" to
debian-devel-request@lists.debian.org .
Trouble? e-mail to templin@bucknell.edu .
Reply to: