Bug#656569: debian-policy: Mandorary Testsuite or Demo requirement for all Libraries.
Package: debian-policy
Severity: normal
Dear Maintainer
What has lead up to this situation was the discovery of libogre 1.7.3-3 in the
testing branch. I will be very simple it don't work at all even with the most
basic OGRE test command line no graphical and it segfaulted out yet no simple
application existed for it that a end user could be requested to install and
run to find out if it was dud or not. This is not a suitable location for
support end user ends up blaming the program not the library as they should.
This is not the first time I have seen 100 percent non functional parts make it
to testing.
What I am requesting that its Mandatory requirement to provide a testsuite
package or a demo package matching up with all libraries.
This way something to test the functionality of the libraries even to the most
basic level has to exist.
--The Rules I am putting Forwards START--
If library has a upstream test-suite the test-suite be build and bundled as
package-testsuite.
If library has upstream demos/sample applications they will be packaged up
package-demos..
If those packages happen to contain parts that are license incompatible with
main repo but can be shipped in contrib or non-free without causing the
maintainer any legal trouble. They should still be shipped.
If the upstream testsuite or demos cannot be shipped for any reason in main,
contrib or non-free repositories a package-testsuite-min must be made.
If required the package-testsuite-min will contain the smallest functional
program for the library will be provided where possible without requiring x11
packaged basically initing the basic structs of the library and shutting
library down cleanly.
Package-testsuite-min, package-testsuite and package-demos will all be built
with the exact same complier and envorment used to make the library.
It is mandatory that the Maintainer before sending package up stream to run in
order of preference the upstream testsuite, upstream demos or the smallest
functional program. If Maintainer will report in package description what test
was run being one of testsuite,demo or testsuite-min along with any known
failures.
Integration of upstream testsuite with deb8 is preferred if possible.
--The Rules I am putting Forwards END--
Libraries missing all 3 will be pretty simple to search for. I would be kind
and say maintainers have 12 months to come into line with the new policy.
This should allow end users making applications to work out if it complier or
library failure. The idea that a library will be run some time in 10 days
before it moves from unstable to testing is a flawed one. Only way to be 100
percent sure it run its make it mandatory for the maintainer todo so.
Issue of maintainers being able to skip out on providing test-suites means they
also get to skip out on running them. If they have had to build them anyhow
they might as well run them. Maintainers praying that someone tries there
library before the 10 days is up is not suitable to say the least. Even in 10
days a all users of the unstable branch might not use all functions of a
library to the extent the libraries own testsuite will. So detectable faults
are slipping threw that should be stopped.
Issues of testsuites not existing removes means to for users to test against
hardware particular errors. This will become more important as more items use
opencl and other acceleration methods. Saying users can build from source is
not going to cut it. In case of a complier error the source code build will
not work of the test-suite but the binary build should work the same as
everyone else with the binary build. Reverse is also true if the binary
testsuite does not work yet everyone rebuilding the testsuite with a different
complier it works you know you had a complier issue and the package need to be
rebuilt.
Really go to repo and search testsuite and the problem is straight up in your
face. Less than 20 items return. End user cannot test there system unless
they know how to code even then building the testsuite from source may give a
false reading due to complier defect. That the library is bust when the
complier is,
List of people who should be able to check. Maintainer, Automated system
about the maintainer reason for deb8 support recommendation and the end user.
The issue causing me to request this is a segfault all kind of errors can be
sneaking past due to testing not being done. There can be security issues
sneaking past as well.
Will everyone like this most likely not. I know it will mean more disk space.
Of course one option might be extra branches in the repos test and test-non-
free for containing the test-suites so all mirror sites don't have major
storage requirement expand.
There are reasons at times test suites cannot be allowed to be modified so they
fall outside the normal permitted licenses by Debian even that they are
perfectly allowed to be shipped as long as there source code is not modified.
Solution to prevent this issue need to be design. Needs to be mandatory that
every package maintainer has to consider when packaging. Failure to provide in
time see package rejected until it is.
10 line program to run the library to confirm it does the basics is really not
asking much. Proper splitting of packages so leading to requesting upstream
proper splits there licensing parts is also not asking too much.
Solution of lets just delete what does not conform is not suitable but it is
acceptable under the current policy. Package contains a test-suite containing
license parts that cannot go into main repo deleting it out right is a suitable
solution to conflict by current policy. Yes libogre contains quite a decent
testsuite that basically is not provided even in the source debs due to license
issue blocking it from main. Not blocking from contrib or non-free. Simpler
on the maintainer not to have to make those packages.
You will find many maintainers only doing what the bar min of the rules
required.
This issue is grave because it can be a source of poor QA control leading to
every problem under the sun. Programs crashing as demoed by libogre. Secuirty
flaws can be missed due to testsuites not being run, Data-loss possible with
libogre that is build inside policy rules.
Something has to be changed in the policy to prevent this. If not my change
something else equally mandorary.
Yes its better to reject a under-tested package outright and force a maintainer
to make a package that is tested by testsuite than let a undertested package
get to testing then with a possible hope of sneaking into stable.
-- System Information:
Debian Release: wheezy/sid
APT prefers stable
APT policy: (700, 'stable'), (650, 'testing'), (600, 'unstable'), (500, 'proposed-updates'), (400, 'experimental')
Architecture: amd64 (x86_64)
Kernel: Linux 3.1.0-1-amd64 (SMP w/2 CPU cores)
Locale: LANG=en_AU.UTF-8, LC_CTYPE=en_AU.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Reply to: