LSB 1.0 Test Suite Coverage
LSB 1.0 Test Suite Coverage
Kevin Caunt
IBM Linux Technology Center
LSB Core Team Member
September 20, 2001
Introduction
The goals of the Linux Standard Base (LSB) are to develop and
promote a set of standards that will increase compatibility
among Linux distributions, and to extend this compatibility to
Linux-based software applications. The processing environment
created by this compatibility is referred to in this document as
LSB-compliance.
A primary step in the direction of these goals is to determine
precisely where existing distributions and applications stand in
relationship to these standards. This measurability is achieved
through the development of test suites to analyze LSB-compliance.
This document visits the groupings of tests that form the current
LSB 1.0 Test Suite.
Section 1 -- ABI-related tests.
Section 2 -- API-related tests.
Section 3 -- Other tests.
These test suites can be downloaded from the following site:
ftp://ftp.freestandards.org/pub/lsb/test_suites/
It is important to understand the extent of the current test case
coverage in order to evaluate the completeness of the test suite
and validate the LSB test certification program. Therefore,
current test results will be reviewed in this document.
Evaluation of the LSB 1.0 Test Suite Coverage
1.0 Test Suite Composition
The full LSB 1.0 Test Suite is currently comprised of three
sections.
1.1 Application Binary Interface (ABI)-related Test Suites.
There are two tests that check for the absolute existence of the
Application Binary Interface, and for the existence of the
Application Binary Interface with the corresponding Library
version.
LSB-LIBCHK checks to ensure a distribution's ABIs (suitable
versioned) required by the LSB specification exist. This test
permits distributions to provide more libraries, functions
within libraries and further versioned symbols than those
required by the spec; however, it reports any missing
libraries, missing symbols within libraries and
wrongly-versioned symbols.
LSB-APPCHK is run against an application to determine if it is
using only LSB-defined ABIs. It is used during the post build
process like "lint"; however, instead of producing compiled
code, it checks whether the binary file(s) being produced
comply with LSB specification. LSB-APPCHK may be executed with
a -l option to allow you to add a library(s) supplied by the
application. If there is any output from LSB-APPCHK, then it
detected ABI calls that are not specified by the LSB.
1.2 Application Program Interface (API)-related Test Suites.
There are three suites that test the behavior of APIs specified
in the LSB specification.
LSB-VSX-PCTS tests the core of the POSIX.1 standard. It is
anticipated that most Linux distributions will have some
failures; waivers to cover these scenarios for this test suite
need to be developed.
LSB-OS tests for core libc behavior above the LSB-VSX-PCTS
test suite.
LSB-USERSGROUPS (subset) tests commands and a few interfaces
that have been defined in the LSB Users and Groups section of
the LSB specification.
1.3 Other Test Suites.
LSB-FHS ensures that file and directory placements follow
requirements and guidelines outlined in the File System
Hierarchy Standard for UNIX-like operating systems. This
standard enables software, system administrators, and users to
be able to predict the location of installed files and
directories. The LSB-FHS test is a building block of the LSB
1.0 test suite, as it supports tools for system administration
and development, interoperability of applications, scripts,
and provides for a greater uniformity of systems
documentation.
FHS specifies the following major directory trees:
/ contains root file system with programs and utilities
required to boot system and to mount additional file
systems
/bin essential command binaries
/boot static files of the boot loader
/dev device file
/etc host-specific system configuration
/lib essential shared libraries and kernel modules
/mnt mount point for mounting a filesystem temporarily
/opt add-on third party application software packages
(e.g., DB2, ViaVoice)
/sbin essential system binaries
/tmp temporary files
/usr secondary hierarchy
/var variable data
The rework for FHS2.2 is now in the beta stage.
It can be downloaded from the source bundles site:
ftp://ftp.freestandards.org/pub/lsb/test_suites/
LSB-USERSGROUPS tests commands that are defined in the "Users
& Groups" section of the LSB specification.
2.0 Test Suite Statistics
3,235 APIs are now documented in the LSB 1.0 Specification. 287
of these APIs are currently tested by a suite of 7,508 individual
test cases (an individual test case consist of testing one
flag/option/parameter on a interface). The LSB-FHS test suite
adds 251 test cases to this total, and LSB_USERSGROUPS
contributes an additional 15 test cases.
2.1 Summary Statistics.
The following statistics demonstrate the range of LSB test coverage
to date. Where noted, calculations reflect the current absence of
graphics testability.
Overall test coverage of the specification (API Behavior Test
and Other Test)
Results: 553 of 3401 modules ( 16.3% )
Minus Graphics Library: 553 of 1589 modules ( 34.8% )
APIs Only (API Behavior Test)
Results: 287 of 3235 modules ( 8.9% )
Minus Graphics Library: 287 of 1323 modules ( 21.6% )
2.2 Statistics by Test Area.
The following chart shows the percent of test coverage for the
Distributions and Applications by test section.
Test Section Application Distribution
ABI Existence Test LSB-APPCHK 100% LSB-LIBCHK 100%
API Behavior Test N/A LSB-VSX-PCTS
Other Test LSB_FHS Checklist 100% LSB-FHS 100%
(Manual)
2.3 Statistics for API test suites (LSB-VSX-PCTS, LSB-OS,
LSB-USERSGROUPS).
The Application Program Interface Behavior Test Suite has been
broken down by Library coverage to give a more useful representation
of API behavior test suite coverage.
System Interfaces
System Interfaces (libc) (30.1% )
Math (libm) ( 8.0% )
Dynamic Loader (libdl) (40.0% )
Encryption (libcrypt) ( 0.0% )
pThreads (libpthreads) (12.5% )
Asynchronous I/O (librt) (34.8% )
Auxiliary Interfaces
Compression (libz) ( 0.0% )
Utility Functions (libutil)( 0.0% )
Curses (libcurses) ( 0.0% )
Graphics Interfaces
X Window System (libX11) ( 0.0% )
X11 Extension (libXext) ( 0.0% )
X11 Intrinsics (libXt) ( 0.0% )
X11 Session Mgnt. (libSM) ( 0.0% )
X11 Inter-Client Exchange (libICE) ( 0.0% )
GL 3D Graphics (libGL) ( 0.0% )
Summary
At this time, we have 100% existence test for the ABIs and approximately
10% behavior test for APIs. There is still a significant amount of work
needed to develop a comprehensive test suite. The LSB must increase the
rate of development in order meet the community expectations.
Reply to: