Just a reminder that there is more to developing certificaton
tests than just porting code under a particular harness.
The methodology for LSB test development, is as follows
(quote paraphrased from the docs I put together for the LSB-FHS
test development,
http://www.opengroup.org/testing/lsb-fhs/process.html
and followed by other addons such as the USERGROUPS and LSB-OS)
The process for developing the test suite involves a multi-step process
where objectivity, and relationship to a written specification are
the key.
The first step is development of a test specification which should undergo
formal review. Secondly, a test suite based on that specification.
There are criteria for both the specification and the test suite that
should be met . At regular points there should be feedback
from the customer audience / underlying specification owners to verify
the correct assumptions.
In the case of the LSB-FHS test suite, the test specification is based
on the FHS 2.0 specification, see www.pathname.com, as modified by the
requirements placed by the LSB specification (for example LSB mandates
presence of the X Window system which is optional in the FHS).
The test specification follows the POSIX 1003.3 guidelines and consists
of a set of assertions , that is a set of statements which evaluate as
true or false based on the specification under test. This can be thought
of the translation between the natural language specification and the
testable specification.
For example, the FHS has a statement in section 3.1:
Required files for /bin
The following commands have been included because they are essential. A few are present because of their traditional placement in /bin.
{A list of 34 commands}
This is written as 34 separate test assertions , for example
Reference 3.1-3 (A)
The implementation provides an exec-able version of the cat
utility in the /bin directory.
Result: PASS/FAIL
Reference 3.1-4 (A)
The implementation provides an exec-able version of the chgrp
utility in the /bin directory.
Result: PASS/FAIL
etc.
The above example is a simple case, more complex examples are for
conditional tests etc. The key is to produce logical, testable statements
that can be reasoned about, these are called test assertions. By producing
test assertions, we can then produce tests.
This allows test development to proceed based on the specifications. At
this point an iterative process occurs as test assertions are validated or
shown to be false. This may be due to one an incorrect assumption by the
author of the test assertion, or in some cases what is now considered an
error in the underlying specification. The latter should only be decided
by the authors of the specifications, and not the test suite author,
and these issues should be fed back to the specification owners so that
corrective action can be taken. The test suite will note the issue in
its output , but the final correct strategy cannot be undertaken until
the specification owners have issued corrections or possibly a version
of the specification.
regards
Andrew
--
To UNSUBSCRIBE, email to lsb-test-request@lists.linuxbase.org
with subject of "unsubscribe". Trouble? Email listmaster@lists.linuxbase.org