Regression tests after build time.
> * Charles Plessy <plessy@debian.org>, 2013-01-17, 10:35:
> >
> >for the packages I maintain, I am now replacing the regression
> >tests that were ran during the build process by autopkgtest test
> >suites.
Le Thu, Jan 17, 2013 at 10:12:50AM +0100, Jakub Wilk a écrit :
>
> DEP-8 tests are not supposed to _replace_ build-time tests. They
> have different, though overlapping to a large degree, purposes.
Le Thu, Jan 17, 2013 at 08:02:32AM -0800, Steve Langasek a écrit :
>
> So now packages that fail their build-time test suites will be accepted into
> the archive, exposing users to these bugs.
Hi all,
I think that building packages and testing them are separate tasks. For
instance, Lintian is not ran during the build, but it is typically done
after the build and before upload.
The advantage of having regression tests isolated from the build is that it is
possible to re-test the package later, for instance to verify that something
that was uploaded years ago is still functional in the context of a system that
where a large number of libraries and services have been updated. This is
something I am very interested to have.
Time is a limiting factor, one somtimes has to chose to support (which often
means writing from scratch) build-time tests or autopkgtest suites. I chose
autopkgtest suites. I of course run them before uploading the package. I also
run Lintian with a sbuild hook, and inspect the diff with the previous build
log before uploading. I read the bugs reported by the Ubuntu users in addition
to the Debian bugs, and actually Ubuntu runs the autopkgtest suites
systematically, which is neat. So I am quite confident with what I upload to
Debian.
I guess that the core of our disagreement is that the autobuilt binary packages
will not be tested unless somebody else does so. I think that this is the
responsiblity of the community of developers supporting each port. The more
manpower they have, the more they can search for bugs to fix in rarely used
packages. I think that in the case of my packages, I still have not been
proven wrong that they are not used on anything else than amd64 and i386. And
I recommend to avoid i386 for bioinformatics, as I have encountered more than
once programs that assume blindly a 64-bit platform, and which will silently
give erroneous results or crash on i386, in a way that is not covered by
regression tests.
To conclude, even if autopkg test suites are not meant to replace build-time
regression tests, implementing each of them are non-overlaping tasks, so they
are competing for time, and I made my choice.
Have a nice Sunday,
--
Charles Plessy
Tsurumi, Kanagawa, Japan
Reply to: