-
aa236bf3
by Felix Lechner
at 2019-02-05T17:18:19Z
Make location of log files for tests configurable.
The file name for the log was previously hardcoded to 'log'. Now it
can be configured in t/defaults/files. (It can also be overridden with
a local copy of 'files', but there is currently no use case for it.)
Gbp-Dch: ignore
-
463a69de
by Felix Lechner
at 2019-02-05T17:18:19Z
Move builder location for tests to 't/defaults/files'.
The builder file is a special template that is filled first and then
loaded into the template data. It allows other templates to receive
values seeded by the builder.
This mechanism is likely to be generalized in the future, so that
loading into the template data is possible from multiple files. Makes
the name of the builder file configurable.
Gbp-Dch: ignore
-
232f1188
by Felix Lechner
at 2019-02-05T17:18:19Z
Turn off sorting for test lintian-output-xml.
The default for sorting tags is 'yes'. It was not disabled for this
test, which produces output in XML. Sorting those lines lexically
makes little sense. Disables sorting for this test.
Gpb-Dch: ignore
-
9f9f18c9
by Felix Lechner
at 2019-02-05T17:18:19Z
Add template to parameterize the Lintian invocation in tests.
The runner puts together a command line when calling Lintian. This is
done programmatically. This adds a template that is fulled with data
such as the output format. This parameterizes the Lintian invocation.
Gbp-Dch: ignore
-
24ccdc5b
by Felix Lechner
at 2019-02-05T17:18:19Z
Add template whitelist for Lintian invocation in tests.
When templates are filled, the runner restricts itself to a list of
files known as whitelists. The purpose is to avoid filling unrelated
templates in a test when there may be another need for them. Adds a
whitelist for the template set used in the Lintian invocation.
Gbp-Dch: ignore
-
08939b9b
by Felix Lechner
at 2019-02-05T17:18:19Z
Add Lintian invocation template to test skeletons.
A template set added recently produces a command line for Lintian. It
leaves a record in the test's working directory and can be examined
for debugging. It also parameterizes the Lintian invocation. Adds the
template set to skeletons that need it.
Gbp-Dch: ignore
-
7c069485
by Felix Lechner
at 2019-02-05T17:18:19Z
Add default settings for how Lintian is called in tests.
This sets the default Lintian options in tests to '-I -E'. The new
option is intended to be used with the old 'Options': except that a
test using it no longer has to duplicate the default settings. Makes
the test suite more internally consistent between tests. It is a
preparation for cycling through different output formats.
Gbp-Dch: ignore
-
f24ae394
by Felix Lechner
at 2019-02-05T17:18:19Z
Adjust internal harness test for new Lintian options scheme.
This changes a harness tests after the defaults were changed. The
defaults are examined in this test.
Gbp-Dch: ignore
-
03d0f0de
by Felix Lechner
at 2019-02-05T17:18:19Z
Adjust test specifications for new Lintian options scheme.
The default settings are now propagated through a new parameter called
Default-Lintian-Options. Removes default settings of '-I -E' from test
specifications that use them and leaves only options that were clearly
meant to differ from the default settings. This is a preparation for
cycling through different output formats.
Gbp-Dch: ignore
-
aabb7b2a
by Felix Lechner
at 2019-02-05T17:18:19Z
Use new lintian invocation template when running lintian in tests.
Switches the runner from a programmatic invocation of Lintian to the
template-based call. This documents the options that were used and
their order, which makes debugging easier. It also aids in the
parameterization of the Lintian invocation. This is a preparation for
cycling through a variety of Lintian settings, such as output formats.
Gbp-Dch: ignore
-
908c1330
by Felix Lechner
at 2019-02-05T17:18:19Z
Add Build-Depends: libxml-libxml-perl <!nocheck>.
A new utility called tagextact will use this to parse Lintian's XML
output format.
The library is recommended over libxml-simple-perl, which is already
on the list but reportedly buggy and dated.
This library is available all the way back to oo-stable.
Gbp-Dch: ignore
-
87bceb39
by Felix Lechner
at 2019-02-05T17:18:19Z
Add Build-Depends: libtext-csv-perl <!nocheck>.
Two new utilities called tagextract and tagdiff will use this to parse
colon-based output from Lintian and also a new generalized format for
tags. The generalized format is necessary to automatically cycle
through various Lintian output formats.
This library is available all the way back to oo-stable.
Gpb-Dch: ignore
-
2fd8c0f1
by Felix Lechner
at 2019-02-05T17:18:19Z
Add Build-Depends: libtext-diff-perl <!nocheck>.
A new utility called tagdiff uses this library to find differences
between expected and actual tags in the test suite. Tagdiff operates
on a new generalized format for tag information. That format became
necessary when cycling automatically through different Lintian output
formats.
This library is available all the way back to oo-stable.
Gbp-Dch: ignore
-
26dce6d2
by Felix Lechner
at 2019-02-05T17:18:19Z
Adds tagextract, which reads tags from various Lintian output formats.
The tag specifications for tests ('tags') were previously a copy of
Lintian output. With Lintian's command line options, such formats can
differ widely.
This executable can extract tag information from all Lintian output
formats, including the experimental 'fullewi'. It will eventually make
the tag specifications output invariant. It is step toward making
Test-For and Test-Against obsolete.
The file format produced by this program is the largest subset of
values available in every output format. (Some have more, and some
have less.) It is a CSV file with a vertical bar as a separator. This
format was chosen to visually distinguish it from the colons output
format. It looks like this:
source|changelog-file-general|latest-debian-changelog-entry-without-new-date|
binary|changelog-file-general|possible-missing-colon-in-closes|"Closes #555555"
binary|changelog-file-general|misspelled-closes-bug|#666666
binary|changelog-file-general|debian-changelog-line-too-long|"line 8"
binary|changelog-file-general|debian-changelog-line-too-long|"line 15"
binary|changelog-file-general|bad-intended-distribution|"intended to experimental but uploaded to unstable"
The best way to compare two of those files is to use tagdiff.
Gbp-Dch: ignore
-
cca313ae
by Felix Lechner
at 2019-02-05T17:18:19Z
Adds tagdiff, which compares files produced by tagextract.
The tag specifications produced by tagextract have many advantages but
are designed to be readable by machines. This utility provides pretty
output for differences between two such files, hopefully even more so
in the future.
The output is sorted in reverse lexicographic order under LC_ALL=C.
Returns with a zero exit code even when the tags do not match. It
would be nice to match the behavior of standard 'diff', but that would
likely require disabling autodie and catching exceptions thrown by
Path::Tiny. For now, the exit code is non-zero only when the program
terminates abnormally.
When used in a terminal, produces colored output via Term::ANSIColor.
The test runner uses this tool to highlight discrepancies when tests
fail.
Gbp-Dch: ignore
-
7091bf3d
by Felix Lechner
at 2019-02-05T17:18:19Z
Add internal harness tests for tagextract and tagdiff.
The two utilities are essential components for how tests are
evaluated. A malfunction could make test results unreliable. Adds
internal harness tests to make sure that tagextract and tagdiff work
as intended.
Gbp-Dch: ignore
-
e922770b
by Felix Lechner
at 2019-02-05T17:18:19Z
Replace old tags with sorted EWI for lintian-output-colons.
Test that use a non-standard output format make it harder to compute
Test-For. Replaces 'tags' with the same content but in an EWI version.
Proper matching will take place via tagextract and tagdiff.
Gbp-Dch: ignore
-
e32c8d2c
by Felix Lechner
at 2019-02-05T17:18:19Z
Replace old tags with sorted EWI for lintian-output-xml.
Test that use a non-standard output format make it harder to compute
Test-For. Replaces 'tags' with the same content but in an EWI version.
Proper matching will take place via tagextract and tagdiff.
Gbp-Dch: ignore
-
db7b9248
by Felix Lechner
at 2019-02-05T17:18:19Z
Replace old tags with sorted EWI for lintian-output-letter.
Test that use a non-standard output format make it harder to compute
Test-For. Replaces 'tags' with the same content but in an EWI version.
Proper matching will take place via tagextract and tagdiff.
Gbp-Dch: ignore
-
b8c9e2d7
by Felix Lechner
at 2019-02-05T17:18:19Z
Add test for Lintian's fullewi output format.
The test suite has tests for all Lintian output formats except this
one. Adds a test. The tag specification is in EWI so the test can be
used with tagextract and tagdiff.
Gpb-Dch: ignore
-
8585b90c
by Felix Lechner
at 2019-02-05T17:18:19Z
Make "Sort: no" the default for tests.
Sorting is now done interally by tagextract and tagdiff. It is no
longer necessary to sort Lintian output.
As a side note, the tag specification files 'tags' can soon be in any
order.
Gbp-Dch: ignore
-
733f9297
by Felix Lechner
at 2019-02-05T17:19:16Z
Remove internal Lintian warnings from tag specifications in tests.
Lintian emits some warnings that are not tags. Those warnings cannot
be processed by tagextract or tagdiff.
Please also see #921112, which tracks this issue.
The messages are emitted by lib/Lintian/ProcessablePool.pm and should
probably be converted to proper lintian warnings.
The following tests are affected:
t/tags/debs/control-field-traversal-4
t/tags/source/control-field-traversal-1
t/tags/source/control-field-traversal-3
The messsages relate to relative path names in control fields and look
like this:
warning: tainted [...] package '...', skipping
The objective is to test for this warning. The tests do not contain
any other tags. As an interim solution, the tests were given the
ficticous tag 'package-tainted'. The test calibration mechanism is
used to remove the the internal warning from the actual output. This
causes the tests to fail. As a result, they were marked TODO.
-
5955b397
by Felix Lechner
at 2019-02-05T17:19:20Z
Use output-invariant tag specifications with tagextract and tagdiff.
All expected tags in the test suite are now specified as EWI. This
makes it possible to calculate Test-For automatically. That will be
implemented in short order using a new desc parameter called Check.
The line ordering in 'tags' is from now on disregarded.
Makes tag specifications invariant of output format (selected via
Output-Format) and insensitive to order. The extracted tags are sorted
automatically.
With this change the test runner no longer compares output files
verbatim. Uses tagextract and tagdiff instead.
-
94d07786
by Felix Lechner
at 2019-02-05T17:19:20Z
Fix unknown tag names in two test specifications.
The tag names were renamed (before the branch introducing them was
merged) but the Test-Against field in some tests was not adjusted.
Corrects the tags.
Gbp-Dch: ignore
-
478298a3
by Felix Lechner
at 2019-02-05T17:19:20Z
Add missing newlines to test specifications that need it.
Some desc files are not terminated with newline characters. Due to
restrictions imposed by the parser, the data in desc may not be
separated by empty lines. That makes it more difficult to add fields
automatically. Adds the missing newline characters.
Gbp-Dch: ignore
-
b549f102
by Felix Lechner
at 2019-02-05T17:19:20Z
Remove extra trailing newlines from test specifications that have them.
Some desc files have extra newline characters at the end. Due to
restrictions imposed by the parser, the data may not be separated by
empty lines. It makes it more difficult to add fields automatically.
Removes extra white space at the end of such files.
Gbp-Dch: ignore
-
1b4542e1
by Felix Lechner
at 2019-02-05T17:19:20Z
Add an empty Test-Against to all tests that do not have one.
A new parameter called Check will make it possible to calculate
Test-For automatically. In terms of set theory, it is the intersection
of the tags belonging to the checks in the new parameter and the tags
expected in 'tags'.
A meaningful measure similar to Test-Against can likewise be
calculated. It is Test-For's complement, i.e. the tags in the checks
that do not appear in 'tags'.
The mechanism works great for focused checks. Unfortunately, many
checks are too broad. That could cause the automatic Test-Against to
contain too many tags. (The only drawback would be that more tests are
run when a tag is specified with 'onlyrun'.) Anyway, to ease the
transition, Test-Against is honored if present.
To preserve the old behavior even for test that do not have a
Test-Against, we add an empty field to them. To get the new
behavior, please simply remove it.
Test-Against should never be used in narrowly written checks.
Some tests are not tag-related. Test-Against is added only when
Test-For is present. The sole exception is runtests-todo-test-for,
which claims to test a ficticous tag 'lintian-easter-egg'. That test
will be repurposed to check if the computed Test-For (rather than the
actual one) is implausible.
Gpb-Dch: ignore
-
15a62281
by Felix Lechner
at 2019-02-05T17:19:20Z
Set the new parameter 'Check' for all tests in suite 'tags'.
The purpose of the new parameter is to calculate Test-For
automatically. Adds the parameter with the appropriate value to all
tests in suite tags.
The singular in Check is aspirational. The field can contain multiple
checks separated by spaces. The value is the the sum of all checks
belonging to the tags mentioned in Test-For and in Test-Against.
Gbp-Dch: ignore
-
2719afc6
by Felix Lechner
at 2019-02-05T17:19:20Z
Add Build-Depends: liblist-compare-perl.
An earlier code version used hashed to compute the various tag sets.
That code was twice as long and probably more prone to errors going
forward. This library makes the code simpler and easier to understand.
The library is available all the way back to oo-stable.
Gbp-Dch: ignore
-
ceafc3c6
by Felix Lechner
at 2019-02-05T17:19:20Z
Calculate Test-For and Test-Against from new parameter Check in runner.
A new parameter called Check makes it possible to calculate Test-For
automatically. In terms of set theory, it is the intersection of the
tags belonging to the checks in the new parameter and the tags
expected in 'tags'. Performs this calculation in the test runner.
The parameter Test-For is obsolete.
A meaningful measure similar to Test-Against can likewise be
calculated. It is Test-For's complement, i.e. the tags in the checks
that do not appear in 'tags'.
The mechanism works great for focused checks. Unfortunately, many
checks are too broad. That could cause the automatic Test-Against to
contain too many tags. (The only drawback would be that more tests are
run when a tag is specified with 'onlyrun'.) Anyway, to ease the
transition, Test-Against is honored if present.
To preserve the old behavior, an empty Test-Against is added for tests
that do not use it. To get the new behavior, please remove it.
Test-Against should not be used in narrowly written checks.
-
71b3dcfe
by Felix Lechner
at 2019-02-05T17:19:51Z
Remove Test-For from all test specifications. (MR: !138)
The parameter Test-For is obsolete. Please use the parameter Check
instead. It provides the same results with less maintainance.
>From now on, the use of Test-For triggers an error.
-
ed4a7fe8
by Felix Lechner
at 2019-02-05T17:19:59Z
Add an internal harness test for the fiels in desc files.
Ensures that:
- test name matches the name of the encapsulating directory
- all mandatory fields are present
- no disallowed fields are present
- Test-Against is not present without Check
- there are no duplicates in Test-Against
- the entries in Check are valid Lintian checks
- there are no duplicates in Check
- the tags mentioned in Test-Against belong to checks in Check
This is a mandatory test that runs across over tests with each
invocation.
-
ec81e0c9
by Felix Lechner
at 2019-02-05T17:19:59Z
Repurpose runtests-todo-test-for for empty Test-Against.
This test checked for implausible tags in Test-For. With the parameter
Test-For obsolete, the test is repurposed to check for an implausible
computed Test-For (rather than an actual one).
When Test-Against is overridden, it cannot be empty while the expected
'tags' are also empty. That is because an empty Test-Against implies a
non-empty Test-For. That in turn requires something in 'tags'. Any
such test must fail.
To catch the failure, the test remains marked TODO.
This logic applies only to tag-related tests, which are those using
Check.
Gbp-Dch: ignore
-
d11721ed
by Felix Lechner
at 2019-02-05T17:19:59Z
Remove Test-Against from tests when it matches the computed value.
The runner computes Test-For and Test-Against from other settings.
Test-Against can be overridden to get the same behavior as before.
Removes the setting when the explicit value matches the computed one.
Gbp-Dch: ignore