Re: LSB.fhs failure digest for Debian Sample Implementation
Matt
Thanks for your feedback. I'm assuming this is the latest cut of the
LSB-FHS testsuite. Some of the failure output from the test suite would
be helpful for the ones categorized as test bugs so we can analyze
them further.
A general comment on the results below -- many of these below are
type C assertions (optional) and return FIP result codes. Many if not
all of these would resolve to PASS by a manual declaration that the
subsystem/file is not supported on the platform under test.
(i attach a note at the end of this mail on understanding the
result codes).
Some background on the process and methodology used for
the certification test suites is at:
http://www.opengroup.org/testing/lsb-fhs/process.html
Comments below on individual tests:
On Aug 15, 1:34pm in "LSB.fhs failure dige", Matt Taggart wrote:
> In the comparison of LSB 1.0 and Debian described at,
> http://people.debian.org/~taggart/lsb/
> I defined a Debian LSB 1.0 Sample Implementation and ran the LSB test suite
> against it. This message is digest of the failures for the test suite listed.
>
>
> LSB.fhs
> -------
> /tset/LSB.fhs/root/etc/etc-tc 7
> "/etc/gettydefs: file not found"
> This is only provided by the Debian "gettyps" package which is non-free.
>
Category: no change required
This is a type C assertion and gives a PASS or FIP result,
i.e. if you do not support the subsystem you can manually
resolve this to pass.
> /tset/LSB.fhs/root/etc/etc-tc 29
> "/etc/hosts.equiv: file not found"
> Provided by the Debian "rsh-server" server. rsh is not required by the LSB.
>
ditto
> /tset/LSB.fhs/root/etc/etc-tc 30
> "/etc/hosts.lpd: file not found"
> Not in Debian
>
ditto
> /tset/LSB.fhs/root/etc/etc-tc 38
> "/etc/sgml: directory not found"
>
ditto
> /tset/LSB.fhs/root/etc-x11/etc-x11-tc 2
> "/etc/X11/XF86Config"
> Needs to be done manually?
>
ditto
> /tset/LSB.fhs/root/etc-x11/etc-x11-tc 3
> "/etc/X11/Xmodmap: file not found"
> In "xbase-clients" Debian package
>
Currently this is viewed as a required file. LSB makes
higher level reqts for support of X applications than
the FHS
> /tset/LSB.fhs/root/sbin/sbin-tc 17
> "/sbin/fastboot: file does not exist or is not executable"
> Not in Debian (es and ja manpages do exist though)
>
Type C assertion , comments as per the ditto ones above
> /tset/LSB.fhs/root/sbin/sbin-tc 18
> "/sbin/fasthalt: file does not exist or is not executable"
> Not in Debian (es and ja manpages do exist though)
>
Type C assertion , comments as per the ditto ones above
> /tset/LSB.fhs/usr/bin/bin-tc 4
> "/usr/bin/tclsh: No such file or directory"
> Provided by tcl8.* (tcl8.3 is the latest currently)
> Is tcl required by the LSB?
>
Type C assertion, coments as above....
> /tset/LSB.fhs/usr/bin/bin-tc 5
> "/usr/bin/mh: directory not found"
> Provided by Debian "mh" or "nmh" packages.
> Is mh required by the LSB?
>
Type C assertion, coments as above....
> /tset/LSB.fhs/usr/bin/bin-tc 6
> "/usr/bin/wish: does not exist or is not executable"
> Provided by tk8.* (tk8.3 is the latest currently)
> Is tk required by the LSB?
>
Type C assertion, coments as above....
> /tset/LSB.fhs/usr/bin/bin-tc 7
> "/usr/bin/expect: does not exist or is not executable"
> Provided by expect* (expect5.31 is the latest currently)
> Is expect required by the LSB?
Type C assertion, coments as above....
>
> /tset/LSB.fhs/usr/share/share-tc 3
> "/usr/share/games: directory not found"
> Add to base-files
Type C assertion, coments as above....
>
> /tset/LSB.fhs/usr/share/share-tc 6
> "/usr/share/nls: directory not found"
> Add to base-files?
Type C assertion, coments as above....
>
> /tset/LSB.fhs/usr/share/share-tc 8
> "/usr/share/tmac: directory not found"
> Add to base-files?
>
Type C assertion, coments as above....
> /tset/LSB.fhs/var/cache-fonts/cache-fonts-tc 1
> "/var/cache/fonts: directory not found"
> Add to base files? Nothing in Debian uses it.
Type C assertion, coments as above....
>
> /tset/LSB.fhs/var/games/games-tc 1
> "/var/games: directory not found"
> Add to base-files
>
Type C assertion, coments as above....
> /tset/LSB.fhs/var/spool-lpd/spool-lpd-tc 2
> "/var/spool/lpd/lpd.lock: file not found"
> Not in Debian
>
Type C assertion, coments as above....
> /tset/LSB.fhs/var/spool-rwho/spool-rwho-tc 1
> "/var/spool/rwho: directory not found"
> Not in Debian
Type C assertion, coments as above....
>
> /tset/LSB.fhs/root/etc-opt/etc-opt-tc 1
> "/etc/opt: directory not found"
> Add to base-files?
Non compliance , this is a required directory
>
> /tset/LSB.fhs/root/opt/opt-tc 1
> "/opt: directory not found"
> Add to base-files?
>
Non compliance , this is a required directory
> /tset/LSB.fhs/usr/x11r6/x11r6-tc 2
> "The symbolic link /usr/bin/X11 shall exist and point to /usr/X11R6/bin"
> On Debian it looks like "/usr/bin/X11 -> ../X11R6/bin" Is this acceptable?
> Fix the test?
Can you send the test output so we can analyze the failure
>
> /tset/LSB.fhs/usr/lib/lib-tc 2
> "stderr:/usr/lib/sendmail expected to be symlink to /usr/sbin/sendmail,
> pointed to /usr/sbin/exim"
> Test is broken
Why do you think the test is broken please supply more information
>
> /tset/LSB.fhs/var/opt/opt-tc 1
> "/var/opt: directory not found"
> Add to base-files?
>
Non compliance
I attach some notes on how to understand result codes in the
test suites
UNDERSTANDING TEST RESULT CODES
===============================
- Failed
The test source code for failed operating system tests is located in
the appropriate testset directory, in the directory hierarchy starting
from tset. To analyse the results of these tests fully, you must be
able to examine the test source code to understand the test strategy
and identify the conditions which led to the test failure. This level
of expertise requires the skills of an operating system specialist;
non-specialist staff should not attempt to interpret these results.
- Uninitiated or Unresolved
Uninitiated means that the particular test in question did not start
to execute. Unresolved means that the test started but did not reach
the point where the test was able to report success or failure. When a
test is reported as uninitiated or unresolved you must identify the
reason why the test was not completed. These may be because of incorrect
parameters, preceding failures or external events, which are described
in the following paragraphs.
Incorrect Parameters
Most tests reported this way cannot be run because a parameter is
not set correctly in the execution parameters file tetexec.cfg. The
test report always identifies the tests which cannot run because of
incorrect parameters. For some tests, you can correct the parameter
and re-run the tests. For others, you may not be able to correct the
parameter because the resources required are not available on your system.
Incorrect Entries in userintf.c (Implementation Specific Routines) Tests
may be reported as unresolved or uninitiated because of incorrect entries
in SRC/userintf.c. If failures of user-supplied functions are reported,
you will need to check this file.
Preceding Failures
When earlier tests have failed, some tests cannot be performed. Before
you can re-run the tests, you must resolve the problem in the preceding
failed tests.
External Events
When an external event occurs unexpectedly, tests may not be
performed. Investigate the reason the test has not been run as for a
failed test.
- Unreported
When a test is marked as unreported a major error has occurred during
the testset execution. VSX tries to avoid such errors as far as
possible. However, if you terminate a testset with the signal SIGTERM,
tests will be unreported. Investigate the cause of the major error as
for a failed test.
- Warning
Whenever a warning is given, the functionality is acceptable, but
you should be aware that later revisions of the relevant standards
or specifications may change the requirements in this area. See the
appendix entitled ``TESTS GIVING WARNINGS'' for a list of the tests
which may give warnings and the reasons for them.
- FIP (Further Information Provided)
When a test has succeeded, additional information may sometimes be
given which needs to be inspected. Where information cannot be checked
automatically by a test it is given for you to validate. For example,
the system name and node name are given by the VSX4 uname testset.
Many of the options in the FHS specification cannot be determined
automatically and will return this result code.
- Unsupported
Unsupported means that an optional feature is not available or supported
in the implementation under test.
For example, in some modes the job control features are optional. VSX
will recognise that they are unsupported on a particular system and
report this.
- Not In Use
Where no macro version of an interface exists, or separate macro and
function testing is not required, the macro version of the testset will
report all tests as not in use. Also, some tests within a testset may not
be required in a particular test mode. For example, tests for POSIX.1-1996
functionality when running in POSIX90 mode, or where test cases required
in earlier versions of the FHS specification are not required in the
latest version. These are not failures and require no further work.
- Untested
This occurs because there is no test written to check a particular
feature, or an optional facility needed to perform a test is not available
on the system. For example, it is not possible to check that session IDs
are inherited across a fork() when job control is not available. These
are generally listed on the manual pages under ``Untestable Aspects''.
- Succeeded (PASS)
This means the test has been executed correctly and to completion without
any kind of problem.
regards
Andrew
-----
Andrew Josey
The Open Group
Reply to: