Assurance measures: ATE (well, ehhrmm)
Hi!
The open source development method have a very different
approach to tests from the commercial development method:
Testing is a largely uncoordinated effort, driven by the
individual needs of users. That means that zero to none
evidence is gathered on tests actually ran, and no structured
approach can be claimed on it. Not if testing would not be
made, just the depth and coverage of them cannot be
establised.
Of course there are examples on the contrary, like some
packages which run elaborate tests on 'make test' or
www.linuxtestproject.org, but they are still enough only
to EAL2, which is close to nothing.
My favourite example is happydoc: the author first writes
test cases, and the code _after_ that. It might even enough
for ATE_DPT.3, which requires tests against the implementation
representation (code).
ATE_COV.2 Analysis of coverage (EAL3-EAL5)
ATE_COV.2.1D The developer shall provide an analysis of the test
coverage.
(www.linuxtestproject.org)
ATE_COV.2.1C The analysis of the test coverage shall demonstrate the
correspondence between the tests identified in the test
documentation and the TSF as described in the functional
specification.
(See ADV_FSP to realize that this is the second step to take)
ATE_COV.2.2C The analysis of the test coverage shall demonstrate that
the correspondence between the TSF as described in the
functional specification and the tests identified in the test
documentation is complete.
(Writing tests is boring, eh?)
ATE_DPT.1 Testing: high-level design (EAL3, EAL4)
ATE_DPT.1.1D The developer shall provide the analysis of the depth of
testing.
(the lack of this may be the root cause that SLES got only
EAL2)
ATE_DPT.1.1C The depth analysis shall demonstrate that the tests
identified in the test documentation are sufficient to
demonstrate that the TSF operates in accordance with its
high-level design.
(Writing tests is still boring.)
ATE_FUN.1 Functional testing (EAL2-EAL5)
ATE_FUN.1.1D The developer shall test the TSF and document the results.
(If one is writing tests, it is relatively easy to comment
them.)
ATE_FUN.1.2D The developer shall provide test documentation.
(...and write some pages about their structure.)
ATE_FUN.1.1C The test documentation shall consist of test plans, test
procedure descriptions, expected test results and actual test
results.
(The test script with comments, plus an overview doc is the test
plan, and the proc. description, the expected test results are
some files with the expected output, and actual test results
can be gained by running the test suite.)
ATE_FUN.1.2C The test plans shall identify the security functions to be
tested and describe the goal of the tests to be performed.
(Nice objective for the comments of the test scripts.)
ATE_FUN.1.3C The test procedure descriptions shall identify the tests to
be performed and describe the scenarios for testing each
security function. These scenarios shall include any ordering
dependencies on the results of other tests.
(The code does that, the scenario is "run make;make tests":)
ATE_FUN.1.4C The expected test results shall show the anticipated
outputs from a successful execution of the tests.
(Easy, if we talk about test scripts.)
ATE_FUN.1.5C The test results from the developer execution of the tests
shall demonstrate that each tested security function behaved as
specified.
(the diff command helps a lot:)
ATE_IND.3 Independent testing - complete (EAL7)
ATE_IND.3.1D The developer shall provide the TOE for testing.
(apt source)
ATE_IND.3.1C The TOE shall be suitable for testing.
(seems easy, if the other requirements are fulfilled)
ATE_IND.3.2C The developer shall provide an equivalent set of resources
to those that were used in the developer's functional testing of
the TSF.
(an account on the designated devel machine)
--
GNU GPL: csak tiszta forrásból
Reply to: