-
0f403c3d
by Felix Lechner
at 2019-04-15T09:46:09Z
Implement universal output format in Lintian.
This is a new output option that shortcuts to the output of
tagextract. This was done to avoid processing tags during the test, to
save time. This format is a tag-oriented, common subset of all the
other output formats. It is well-suited to evaluate how well a
particular check is working.
Among the notable features is the absence of the severity tag, which
has no bearing on the effectiveness of a check, as well as that the
output is always sorted with the packages on a particular order. For
example, source packages always come before binary packages.
The format looks like this:
scripts (source): binary-arch-rules-but-pkg-is-arch-indep
scripts (source): ancient-standards-version 3.2.1 (released 2000-08-24) (current is CURRENT)
scripts (binary): wrong-path-for-interpreter usr/bin/rubyfoo (#!/bin/ruby1.8 != /usr/bin/ruby1.8)
scripts (binary): wrong-path-for-interpreter usr/bin/lefty-foo (#!/usr/local/bin/lefty != /usr/bin/lefty)
scripts (binary): unusual-interpreter usr/bin/suidperlfoo #!/usr/bin/suidperl
-
ee7eb262
by Felix Lechner
at 2019-04-15T09:52:19Z
Add default match strategy 'tags' for tests; set default output format to 'universal'; drop sort.
Adjusts the test case defaults to make the matching strategy for
'tags', which was previously used in all tests, the explicit default.
This is done to so that another matching strategy, 'literal', can be
introduced. The second strategy will compare lintian output literally,
without any processing or sorting. It is useful, for example, when
comparing warning messages not in tag format.
Also sets the default output format to 'universal'. This format is
newly available and saves calls to several Perl scripts in the test
runner. Hopefully, that will shorten the test execution times.
Tags in the 'universal' format are always sorted. It generally does
not make sense to test for the order in which tags appear. The sorting
should really be done in Lintian for all output formats. It is no
longer available in the test runner.
Gbp-Dch: ignore
-
3657d2a0
by Felix Lechner
at 2019-04-15T10:33:09Z
Add literal match strategy and output format, as needed, to tests that need it.
Adjusts the test specifications for a number of tests that benefit
from employing the 'literal' match strategy.
Among those are test that use uncommon output formats, such as xml.
Their outputs were previously converted using tagextract, which took
extra time. Now their output compared directly. Very soon, their
outputs will be compared literally with a file called 'output' in the
test specification directory.
There are also some false positive test that produce no output. Those
tests have also been switched to the literal matching stategy to
minimize the processing of their output. The theory is that any
discrepancies would be more easily found.
Gbp-Dch: ignore
-
a1282aec
by Felix Lechner
at 2019-04-15T10:33:09Z
Don't offer to calibrate tests unless their match strategy is 'tags'.
The interactive test calibration, which adds or removes tags after
prompting the user, only works with the 'tags' matching strategy. The
mechanisms is now disabled for other matching stategies.
Gbp-Dch: ignore
-
02d56d16
by Felix Lechner
at 2019-04-15T10:39:12Z
Implement literal match strategy in test runner; adjust for universal tag format; drop sorting.
This provides for the new mechanisms in the test runner. The tag
processing is now done inside the runner, which saves three calls to
Perl scripts: two calls to tagextract and one call to tagdiff.
This step is thought to save about seven minutes in an average
Gitlab run (but the evidence is still unclear).
Gbp-Dch: ignore
-
68b2d7f3
by Felix Lechner
at 2019-04-15T10:39:17Z
Move expected output to 'output' for tests using a literal match strategy.
The literal matching strategy uses files named 'output' to store the
expected output from Lintian. This moves the old files, which were
called 'tags', into the new locations.
The expected tags were previously all in the EWI format, so this will
work for any tests that uses that format going forward. Tests that use
other output formats will have their 'output' files converted.
Gbp-Dch: ignore
-
6eed75c5
by Felix Lechner
at 2019-04-15T10:39:17Z
Adjust expected output for literal matching strategy for lack of sorting and revert to original output format.
Some tests using the literal matching strategy employed the "Sort"
setting, which was removed, or used uncommon output formats. For those
tests, the expected output is converted to match, literally, the
expected output from Lintian.
Gbp-Dch: ignore
-
2e744ed0
by Felix Lechner
at 2019-04-15T10:39:17Z
Convert all expected tags to universal format.
This converts the expected tags for all tests under the 'tags'
matching strategy to the universal format. It simplifies the
comparison of tags in the test runner and saves a call to tagextract,
hopefully also saving time.
Gbp-Dch: ignore
-
88b5ae8d
by Felix Lechner
at 2019-04-15T10:39:17Z
Adjust sed script for test files-foo-in-bar to new universal tag format.
This sed script now works for the universal format, but no longer for
the EWI format. This test now uses the universal format.
Gbp-Dch: ignore
-
89309da3
by Felix Lechner
at 2019-04-15T10:39:17Z
Adjust harness tests for universal tag changes.
The changes to the test runner cause the harness tests to fail. This
adjusts the harness tests to the new defaults and makes them use the
universal tag format when appropriate.
Gbp-Dch: ignore
-
7d4905d0
by Felix Lechner
at 2019-04-15T10:39:17Z
Adjust tag filter script for test runtests-calibration.
The tag filter script for this test previously worked only for the EWI
output format, but the test now uses the universal output format.
Adjusts the script to work with the new, univeral format instead.
The tag filter process was previously known as 'calibration'. We now
use that name when calibrating the expected tags in tests after
changes to the Lintian checks have been made. These scripts will
hopefully be renamed soon. The use of the word calibration for this
particular action should fade into disuse.
Gbp-Dch: ignore
-
0e323a41
by Felix Lechner
at 2019-04-15T10:39:18Z
Convert expected tags for test runtest-calibration from EWI to universal format.
Ordinarily, it would not matter which matching strategy this test
employs, but the test filtering capability (previously known as
calibration), which is being tested here, is only available in the
'tags' strategy. That strategy requires the expected tags to be in
universal format. Converts the tags for that test.
This feature will probably be tested in a harness test in the future.
Gbp-Dch: ignore