override_dh_auto_test (and other overrides)

[Jonathan Yu <jawnsy@cpan.org>]

Hi all,

While taking a look at my libjio packaging, Jakub Wilk mentioned an
issue with my use of override_dh_auto_test to run "$(MAKE) tests" --
namely, the override itself does not take into account the
DEB_BUILD_OPTIONS "nocheck" flag, which is used to disable tests. What
this means is that overrides to dh_auto_test that do not depend on
dh_auto_test to do its work (e.g. those that call ./Build test
directly) are currently violating policy.

Relevant Policy Manual link:
http://www.debian.org/doc/debian-policy/ch-source.html#s-debianrules-options

Relevant bug report against debhelper (tagged wontfix, see Joey Hess'
rationale): http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=568897

I would conjecture that the majority of our packages are using the
dh_auto_test override to add environment variables for testing. In
this case, no further action is required on our part. However, I
noticed there may be an issue with eekboek:

        override_dh_auto_test:
                EB_SKIPDBTESTS=1 ./Build test

This is the full list of packages which use an override to dh_auto_test:

dh-make-perl/debian/rules:override_dh_auto_test:
eekboek/debian/rules:override_dh_auto_test:
jifty/debian/rules:override_dh_auto_test:
kephra/debian/rules:override_dh_auto_test:
libalgorithm-permute-perl/debian/rules:override_dh_auto_test:
libapache2-reload-perl/debian/rules:override_dh_auto_test:
libapache-sessionx-perl/debian/rules:override_dh_auto_test:
libarchive-zip-perl/debian/rules:override_dh_auto_test:
libaudio-mpd-common-perl/debian/rules:override_dh_auto_test:
libauthen-krb5-admin-perl/debian/rules:override_dh_auto_test:
libauthen-pam-perl/debian/rules:override_dh_auto_test:
libauthen-sasl-cyrus-perl/debian/rules:override_dh_auto_test:
libautodie-perl/debian/rules:override_dh_auto_test:
libbio-primerdesigner-perl/debian/rules:override_dh_auto_test:
libbsd-resource-perl/debian/rules:override_dh_auto_test:
libcarp-clan-perl/debian/rules:override_dh_auto_test:
libcatalyst-model-cdbi-perl/debian/rules:override_dh_auto_test:
libcatalystx-leakchecker-perl/debian/rules:override_dh_auto_test:
libcgi-application-server-perl/debian/rules:override_dh_auto_test:
libclass-dbi-sweet-perl/debian/rules:override_dh_auto_test:
libcompress-raw-zlib-perl/debian/rules:override_dh_auto_test:
libconfig-model-itself-perl/debian/rules:override_dh_auto_test:
libcpan-inject-perl/debian/rules:override_dh_auto_test:
libcrypt-saltedhash-perl/debian/rules:override_dh_auto_test:
libdata-section-perl/debian/rules:override_dh_auto_test:
libdatetime-locale-perl/debian/rules:override_dh_auto_test:
libdbd-csv-perl/debian/rules:override_dh_auto_test:
libdbd-mysql-perl/debian/rules:override_dh_auto_test:
libdbd-odbc-perl/debian/rules:override_dh_auto_test:
libdbd-oracle-perl/debian/rules:override_dh_auto_test:
libdbix-class-fixtures-perl/debian/rules:override_dh_auto_test:
libdbix-class-helpers-perl/debian/rules:override_dh_auto_test:
libdbix-class-perl/debian/rules:override_dh_auto_test:
libdbix-class-resultset-recursiveupdate-perl/debian/rules:override_dh_auto_test:
libdbix-class-schema-loader-perl/debian/rules:override_dh_auto_test:
libdbix-class-timestamp-perl/debian/rules:override_dh_auto_test:
libdbix-password-perl/debian/rules:override_dh_auto_test:
libdbm-deep-perl/debian/rules:override_dh_auto_test:
libdigest-crc-perl/debian/rules:override_dh_auto_test:
libdist-zilla-plugin-git-perl/debian/rules:override_dh_auto_test:
libemail-folder-perl/debian/rules:override_dh_auto_test:
libfeed-find-perl/debian/rules:override_dh_auto_test:
libfile-homedir-perl/debian/rules:override_dh_auto_test:
libfile-path-perl/debian/rules:override_dh_auto_test:
libfile-userconfig-perl/debian/rules:override_dh_auto_test:
libfile-which-perl/debian/rules:override_dh_auto_test:
libgeo-osm-tiles-perl/debian/rules:override_dh_auto_test:
libgnome2-gconf-perl/debian/rules:override_dh_auto_test:
libgnupg-perl/debian/rules:override_dh_auto_test:
libgraphics-gnuplotif-perl/debian/rules:override_dh_auto_test:
libgraphics-libplot-perl/debian/rules:override_dh_auto_test:
libgtk2-ex-simple-list-perl/debian/rules:override_dh_auto_test:
libgtk2-imageview-perl/debian/rules:override_dh_auto_test:
libgtk2-mozembed-perl/debian/rules:override_dh_auto_test:
libgtk2-notify-perl/debian/rules:override_dh_auto_test:
libgtk2-perl/debian/rules:override_dh_auto_test:
libgtk2-sourceview2-perl/debian/rules:override_dh_auto_test:
libgtk2-spell-perl/debian/rules:override_dh_auto_test:
libgtk2-unique-perl/debian/rules:override_dh_auto_test:
libgtk2-webkit-perl/debian/rules:override_dh_auto_test:
libhtml-formhandler-model-dbic-perl/debian/rules:override_dh_auto_test:
libhtml-widget-perl/debian/rules:override_dh_auto_test:
libhttp-body-perl/debian/rules:override_dh_auto_test:
libimage-math-constrain-perl/debian/rules:override_dh_auto_test:
libimap-admin-perl/debian/rules:override_dh_auto_test:
libio-all-perl/debian/rules:override_dh_auto_test:
libio-compress-perl/debian/rules:override_dh_auto_test:
libipc-system-simple-perl/debian/rules:override_dh_auto_test:
libjifty-plugin-oauth-perl/debian/rules:override_dh_auto_test:
liblatex-driver-perl/debian/rules:override_dh_auto_test:
liblatex-encode-perl/debian/rules:override_dh_auto_test:
liblivejournal-perl/debian/rules:override_dh_auto_test:
liblog-handler-perl/debian/rules:override_dh_auto_test:
libmath-random-isaac-perl/debian/rules:override_dh_auto_test:
libmath-random-oo-perl/debian/rules:override_dh_auto_test:
libmodule-cpants-analyse-perl/debian/rules:override_dh_auto_test:
libmodule-info-perl/debian/rules:override_dh_auto_test:
libmodule-pluggable-fast-perl/debian/rules:override_dh_auto_test:
libmoo-perl/debian/rules:override_dh_auto_test:
libmoosex-attributehelpers-perl/debian/rules:override_dh_auto_test:
libmoosex-log-log4perl-perl/debian/rules:override_dh_auto_test:
libmysql-diff-perl/debian/rules:override_dh_auto_test:
libnet-arp-perl/debian/rules:override_dh_auto_test:
libnet-https-any-perl/debian/rules:override_dh_auto_test:
libnet-imap-simple-perl/debian/rules:override_dh_auto_test:
libnet-ntp-perl/debian/rules:override_dh_auto_test:
libnet-subnets-perl/debian/rules:override_dh_auto_test:
libnet-twitter-perl/debian/rules:override_dh_auto_test:
libnet-z3950-zoom-perl/debian/rules:override_dh_auto_test:
libopengl-perl/debian/rules:override_dh_auto_test:
libopenoffice-oodoc-perl/debian/rules:override_dh_auto_test:
liborlite-mirror-perl/debian/rules:override_dh_auto_test:
liborlite-statistics-perl/debian/rules:override_dh_auto_test:
libpango-perl/debian/rules:override_dh_auto_test:
libparams-coerce-perl/debian/rules:override_dh_auto_test:
libparams-validate-perl/debian/rules:override_dh_auto_test:
libpar-packer-perl/debian/rules:override_dh_auto_test:
libpar-perl/debian/rules:override_dh_auto_test:
libpod-eventual-perl/debian/rules:override_dh_auto_test:
libpoe-component-client-http-perl/debian/rules:override_dh_auto_test:
libpoe-component-client-mpd-perl/debian/rules:override_dh_auto_test:
libpoe-component-irc-perl/debian/rules:override_dh_auto_test:
libpoe-component-pcap-perl/debian/rules:override_dh_auto_test:
libpoe-loop-event-perl/debian/rules:override_dh_auto_test:
libpoe-loop-tk-perl/debian/rules:override_dh_auto_test:
libpoe-perl/debian/rules:override_dh_auto_test:
libpoex-role-sessioninstantiation-perl/debian/rules:override_dh_auto_test:
libppi-html-perl/debian/rules:override_dh_auto_test:
libquota-perl/debian/rules:override_dh_auto_test:
librose-db-perl/debian/rules:override_dh_auto_test:
libschedule-at-perl/debian/rules:override_dh_auto_test:
libshell-posix-select-perl/debian/rules:override_dh_auto_test:
libsoap-lite-perl/debian/rules:override_dh_auto_test:
libsocialtext-resting-utils-perl/debian/rules:override_dh_auto_test:
libspreadsheet-parseexcel-perl/debian/rules:override_dh_auto_test:
libspreadsheet-read-perl/debian/rules:override_dh_auto_test:
libspreadsheet-writeexcel-perl/debian/rules:override_dh_auto_test:
libsql-abstract-perl/debian/rules:override_dh_auto_test:
libstring-formatter-perl/debian/rules:override_dh_auto_test:
libsub-exporter-perl/debian/rules:override_dh_auto_test:
libterm-query-perl/debian/rules:override_dh_auto_test:
libterm-readline-gnu-perl/debian/rules:override_dh_auto_test:
libterm-readline-perl-perl/debian/rules:override_dh_auto_test:
libtest-cpan-meta-perl/debian/rules:override_dh_auto_test:
libtest-distmanifest-perl/debian/rules:override_dh_auto_test:
libtest-poe-client-tcp-perl/debian/rules:override_dh_auto_test:
libtest-simple-perl/debian/rules:override_dh_auto_test:
libtest-valgrind-perl/debian/rules:override_dh_auto_test:
libtest-www-mechanize-catalyst-perl/debian/rules:override_dh_auto_test:
libtext-diff-perl/debian/rules:override_dh_auto_test:
libtext-greeking-perl/debian/rules:override_dh_auto_test:
libtext-markdown-perl/debian/rules:override_dh_auto_test:
libtext-multimarkdown-perl/debian/rules:override_dh_auto_test:
libtext-recordparser-perl/debian/rules:override_dh_auto_test:
libtie-dbi-perl/debian/rules:override_dh_auto_test:
libtk-objscanner-perl/debian/rules:override_dh_auto_test:
libtk-pod-perl/debian/rules:override_dh_auto_test:
liburi-todisk-perl/debian/rules:override_dh_auto_test:
libvideo-fourcc-info-perl/debian/rules:override_dh_auto_test:
libwarnings-illegalproto-perl/debian/rules:override_dh_auto_test:
libweather-google-perl/debian/rules:override_dh_auto_test:
libwebservice-youtube-perl/debian/rules:override_dh_auto_test:
libwww-curl-perl/debian/rules:override_dh_auto_test:
libwww-freshmeat-perl/debian/rules:override_dh_auto_test:
libwww-mechanize-perl/debian/rules:override_dh_auto_test:
libwww-mechanize-shell-perl/debian/rules:override_dh_auto_test:
libwww-search-perl/debian/rules:override_dh_auto_test:
libwww-wikipedia-perl/debian/rules:override_dh_auto_test:
libwx-perl-datawalker-perl/debian/rules:override_dh_auto_test:
libwx-perl/debian/rules:override_dh_auto_test:
libwx-perl-processstream-perl/debian/rules:override_dh_auto_test:
libx11-xcb-perl/debian/rules:override_dh_auto_test:
libxml-libxml-perl/debian/rules:override_dh_auto_test:
libxml-libxslt-perl/debian/rules:override_dh_auto_test:
libxml-stream-perl/debian/rules:override_dh_auto_test:
libyaml-perl/debian/rules:override_dh_auto_test:
xacobeo/debian/rules:override_dh_auto_test:

These packages should be examined more carefully (they may be affected):
- eekboek (runs ./Build test)
- kephra (uses xvfb-run, may be OK)
- libbio-primerdesigner-perl (runs ./Build test)
- libconfig-model-itself-perl (uses xvfb-run, may be OK)
- libgnome2-gconf-perl (uses dbus-launch, may be OK)
- libgraphics-gnuplotif-perl (uses xvfb-run, may be OK)
- libgraphics-libplot-perl (uses xvfb-run, may be OK)
- libgtk2-ex-simple-list-perl (uses xvfb-run, may be OK)
- libgtk2-imageview-perl (uses xvfb-run, may be OK)
- libgtk2-mozembed-perl (uses xvfb-run, may be OK)
- libgtk2-notify-perl (uses dbus-launch and xvfb-run, may be OK)
- libgtk2-sourceview2-perl (uses xvfb-run, may be OK)
- libgtk2-spell-perl (uses xvfb-run, may be OK)
- libgtk2-unique-perl (uses xvfb-run, may be OK)
- libgtk2-webkit-perl (uses xvfb-run, may be OK)
- libpango-perl (uses xvfb-run, may be OK)
- libpoe-loop-tk-perl (uses xvfb-run, may be OK)
- libtk-objscanner-perl (uses xvfb-run, may be OK)
- libtk-pod-perl (uses xvfb-run, may be OK)
- libwww-search-perl (does some funny stuff in dh_auto_install:
DESTDIR=$(DESTDIR) debian/make-module.sh test)
- libwx-perl (uses xvfb-run, may be OK)
- libwx-perl-processstream-perl (uses xvfb-run, may be OK)
- xacobeo (uses xvfb-run, may be OK)

Other notes:
- libauthen-sasl-cyrus-perl contains a useless dh_auto_test override

The ones that use something else to run dh_auto_test are probably OK,
but I thought I'd mention them just in case. I only looked at rules
files that override dh_auto_test -- no guarantees about the other
environment variables that should be respected according to Policy.

Cheers,

Jonathan