Re: [RFC] Fail builds when regression check shows new errors.
On Thu, Feb 19, 2004 at 05:17:45PM -0500, Carlos O'Donell wrote:
> I've implemented this framework and attached the patch below. The logic is
> as follows:
Thanks to Dan for helping me make the grep rule cleaner, it doesn't rely
on Make[3] being present, or rather it no longer relies on the
positioning of the build tree (rule is now 'make\[.*\]:.* \[/.*').
Tested it again and it still works.
I cleaned up some of the comments, and made the
debian/testsuite-checking/README more readable.
I promise I'll be a dd soon...
If someone wants to snatch all the /usr/share/doc/libc6*/log-test*
files, unpack them, and run them through convertlog.sh we would have all
the failure templates to use in testsuite-checking.
Please apply! :)
Cheers,
Carlos.
diff -urN debian-orig/rules debian/rules
--- debian-orig/rules 2004-02-19 14:59:01.957929240 -0500
+++ debian/rules 2004-02-19 18:01:05.846245792 -0500
@@ -19,7 +19,7 @@
# Create build directory
# Configure
# Build
-# Test as desired
+# Test as desired, including regression comparisons.
# Install to package directories
# Run debian magic to build packages.
@@ -70,6 +70,13 @@
# Default setup
GLIBC_PASSES ?= libc
+# Run testsuite regression testing?
+# Notes:
+# - Remember testing is skipped in a cross-compile environment.
+# - Checking the testsuite is skipped if you disable logs.
+TESTSUITE_CHECKING = $(CURDIR)/debian/testsuite-checking/$(call xx,configure_target)-test-results
+TESTSUITE_CURRENT = $(CURDIR)/debian/testsuite-checking/$(call xx,configure_target)-current
+
prefix=/usr
bindir=$(prefix)/bin
datadir=$(prefix)/share
@@ -107,6 +114,8 @@
else
log_build := /dev/tty
log_test := /dev/tty
+# Disable the testsuite check without logs...
+TESTSUITE_CHECKING = nocheck
endif
# Which build pass are we on?
diff -urN debian-orig/rules.d/build.mk debian/rules.d/build.mk
--- debian-orig/rules.d/build.mk 2004-02-19 14:59:04.201588152 -0500
+++ debian/rules.d/build.mk 2004-02-19 18:00:18.397459104 -0500
@@ -70,6 +70,25 @@
else \
echo Testing $(curpass); \
$(MAKE) -C $(DEB_BUILDDIR) -j $(NJOBS) -k check 2>&1 | tee -a $(log_test); \
+ if [ -e "$(TESTSUITE_CHECKING)" ]; then \
+ echo "Checking testsuite results against $(TESTSUITE_CHECKING)"; \
+ grep 'make\[.*\]:.* \[/.*' $(log_test) \
+ | sed 's,^.*/,,g' | sed 's/\]/,/g' >& $(TESTSUITE_CURRENT); \
+ comm -3 $(TESTSUITE_CHECKING) $(TESTSUITE_CURRENT) 2>&1 | grep -v '#' | tee -a $(log_test); \
+ differences=`comm -3 $(TESTSUITE_CHECKING) $(TESTSUITE_CURRENT) 2>&1 | grep -v '#' | wc -l`;\
+ if [ $$differences -ne '0' ]; then \
+ echo "Encountered regressions that don't match expected failures. Aborting."; \
+ false; \
+ else \
+ echo "Passed regression testing. No new failues, no changed error values."; \
+ fi \
+ elif [ -n "$(findstring nocheck,$(TESTSUITE_CHECKING))" ]; then \
+ echo "TESTSUITE_CHECKING is disabled, will not check testsuite results."; \
+ else \
+ echo "***WARNING*** Please generate testsuite results for this arch!"; \
+ echo "TESTSUITE_CHECKING is enabled, but arch has no testsuite results!"; \
+ echo "***WARNING***"; \
+ fi \
fi
touch $@
diff -urN debian-orig/testsuite-checking/README debian/testsuite-checking/README
--- debian-orig/testsuite-checking/README 1969-12-31 19:00:00.000000000 -0500
+++ debian/testsuite-checking/README 2004-02-19 18:03:37.928125832 -0500
@@ -0,0 +1,21 @@
+Testsuite Regression Testing
+============================
+
+This directory contains a file for each arch/os, listing the allowed failures
+during a testsuite run for a given arch/os pair. The files are listed as
+"${arch-os}-test-results". Where "${arch-os}" is extracted by rules as
+"$(call xx,configure_target)".
+
+A script, "convertlog.sh", can be used to process the normal log-test-*-libc
+file into a *-test-results file.
+
+Since we cannot run the testsuite without "-k", we run the enitre testsuite
+ignoring errors. Then we compare the error set with the expected errors,
+differences in signal numbers or errors fails the build. If the architecture
+does not have a *-test-results file a warning is given and no comparison is
+made.
+
+For reference please see the TESTSUITE_CHECKING variable in:
+debian/rules
+debian/rules.d/build.mk
+
diff -urN debian-orig/testsuite-checking/convertlog.sh debian/testsuite-checking/convertlog.sh
--- debian-orig/testsuite-checking/convertlog.sh 1969-12-31 19:00:00.000000000 -0500
+++ debian/testsuite-checking/convertlog.sh 2004-02-19 17:59:06.154441712 -0500
@@ -0,0 +1,15 @@
+#!/bin/bash
+
+if [ $# -ne '1' ]; then
+ echo -e "\nUsage: Converts a log-*-libc file into a *-test-results file."
+ echo -e "$0 : < Input testsuite log file >\n";
+ exit 1
+fi;
+
+echo -e '#\n# Allowed architecture testsuite failures.'
+echo '# Someone should be working towards fixing these!'
+echo '# Format: <Failed test>, Error <Make error code> [(ignored)]'
+echo '# They are listed here for the purpose of regression'
+echo -e '# testing during builds\n#'
+grep 'make\[.*\]:.* \[/.*' $1 | sed 's,^.*/,,g' | sed 's/\]/,/g'
+
diff -urN debian-orig/testsuite-checking/hppa-linux-test-results debian/testsuite-checking/hppa-linux-test-results
--- debian-orig/testsuite-checking/hppa-linux-test-results 1969-12-31 19:00:00.000000000 -0500
+++ debian/testsuite-checking/hppa-linux-test-results 2004-02-19 13:23:32.531934384 -0500
@@ -0,0 +1,17 @@
+#
+# Allowed failures for HPPA and the returned make failure
+# number, indicating the signal the process died with.
+#
+test-float.out, Error 1
+test-double.out, Error 1
+test-ifloat.out, Error 1
+tst-strtod.out, Error 1
+bug-strtod.out, Error 1
+tststatic.out, Error 139
+annexc.out, Error 1 (ignored)
+neededtest.out, Error 14
+neededtest2.out, Error 14
+neededtest3.out, Error 17
+neededtest4.out, Error 2
+circleload1.out, Error 9
+tst-tls13.out, Error 1
Reply to: