LSB Test Status

[gk4@austin.ibm.com]

The enclosed LSB Test Status is from notes that I have gathered from recent
discussions.  By LinuxWorld this August 28th we need to have (1) a test case
roadmap with target dates and (2) a conformance test plan with a rollout
targeted for 4Q01.

Please look at the enclosed with respect to the roadmap and target dates.   The
roadmap is due before LinuxWorld and the actual completed tests are post
LinuxWorld.

Sincerely,

George (gk4)

Title: FSG: LSB Test Status

LSB Test Status

LSB Test Team Lead:  Andrew Josey - OpenGroup

Distribution Test Suite,  LSB 1.0

• LSB-VSX-PCTS (binary)
• Contact: Andrew Josey - OpenGroup
• Test purpose for core POSIX.1
• Coverage:
• Status: Nearly complete.   Waiting on final LSB v1.0 header file information.

• LSB- FHS  (binary)
• Contact: Andrew Josey - OpenGroup; Chris - IBM
• Test coverage for the filesystem hierarchy standard
• Coverage:
• Status: FHS2.1 Completed, rework in progress for FHS2.2   [Target 8/31]

• LSB - OS (binary)
• Contact: Andrew Josey - OpenGroup
• Additional test  coverage for core libc behavior above LSB-VSX
• Coverage:
• Status: Beta code complete  [Target 8/31]

•  LSB - UsersGroups (binary)
•  Contact: Kevin Caunt- IBM
• Testset for User/Groups  specification
• Coverage: 100% [some users & groups testing done in lsb-vsx-pcts]
• Status: Beta code complete  [Target 8/31]

• LSB - LIBCHK
• Contact: Stuart Anderson - Metrolink
• Test for existence of API's
• Coverage: 100%
• Status: Complete

Development Test Suite,  LSB 1.0

• LSB_HDR
• Contact: Stuart Anderson - MetroLink; Jill Jones - Compaq
• Test the binary values of the headers
• Coverage: ??
•  Status:  Complete

Application Test Suite,  LSB 1.0

• LSB-APPCHK
• Contact: Stuart Anderson - MetroLink
• Test for existence of API's
• Coverage: 100%
• Status: Complete

Post LSB v1.0 Candidates

• LSB-COMMANDS
• Commands and Utilities test coverage
• LSB-PTHREADS
• Test coverage for core pthreads
• LSB-NCURSES
• Test coverage for ncurses
• hard: never been done before
• LSB-VSW(X11)
• Test coverage for X11
• hard: current X11 tests are outdated

---

Details

The gating items are as follows:

• Specification
• Processor-specific definitions for constants and structures needs to be defined, so that a conforming development environment can be used for development of the various test tools for all three proposed programs.
• FHS2.2, in order to complete the FHS tests
• Test Set
• We need feedback on the LSB-OS testset
• We could do with further input on the requirements for application testing.
• We need input on the certification policies.
• Test case coverage.

Recommend that the Free Standards Group put some of their full time resourcing
onto this, especially the policy decisions that need drawing up.

LSB Certification

Firstly we need to define the policy for certification: This needs to define

• What does Certification mean; are we saying passed a test or met a specification? What does it mean for applications, implementations and development environments.
• We will need to define very clearly the conformance requirements, defining precisely what a conforming product must implement in order to be certified. An action on the specification review should be to check that the requirements are clear.
• Interpretations policy
• Process or processes for certification

Once the policies are complete then we can flush out the program in more
detail. Below is an outline recommendation:

It is proposed that there be three levels of certification:

1. Platform certification:

• What: A "brand" mark for platforms to signify they support an environment to run LSB conforming applications
• Test suites/tools:
• Requires a binary test suite for the runtime environment
• This will require repackaging of what we have -- includes removal of parts of the existing tests that are more suited for testing the development environment rather than the runtime environment, for example the header tests
• Binary versions of vsx-pcts, lsb-os, lsb-fhs
• Also requires running a certain number of key LSB conforming applications (we need to define which applications). This is known to be a good approach if a key set of applications exercising a wide set of APIs is chosen.
1. Development Environment/SDK Certification:

• What: A "brand" mark for compiler environments to signify they provide an environment to build an LSB conforming application
• Test suite/tools:
• We should have the capability to test the development environment using devcheck, elfcheck, libcheck
• Note that we would not use the main platform tests for these, the header tests are not thought to provide that valuable output for the development environment, better to use the binary value tests generated by the database.
1. Application certification:
• What: A "brand" mark for applications to signify they work with an LSB conforming platform
• Test Tools:
• lsbappchk
• elfchk