ANN: LSB-VSX test suite V1.0-1
All
The first snapshot release of the LSB-VSX test suite for the
POSIX.1 aspects of the Linux Standard Base is now available for download from:
ftp://lsb.sourceforge.net/pub/lsb/LSB-VSX1.0-1/
This version of the VSX-PCTS has been setup to autoconfigure
on Linux systems. A front end script install_wrapper.sh is used
to auto install, setup and run the test suite. In theory this should
allow running of the test suite by those unfamiliar with POSIX.1
and its myriad of options and thus the associated test suite
configurables.
I include the release notes below.
If you encounter a problem in the suite, please send in a bug report
directly to me at ajosey@opengroup.org, cc lsb-test@linuxbase.org, please
include the test output together with any suggested fixes. Please include
the following subject line in your message
Subject: BUG in LSB-VSX (testcase-name-here)
If possible include the assertion reference number.
Acknowledgements are due to :
Chris Yeoh
Nick Petreley
Dale Scheetz
George Kraft
plus many others involved in this packaging of the POSIX test suite
for Linux.
Our next steps are now to expand our test coverage to other
areas outside the core POSIX.
----------------------------------------------------------------
LSB-VSX Release 1.0-1
Release Notes
This document provides Release notes for the Verification suite for the
POSIX.1 coverage of the Linux Standard Base.
1. Release Overview
LSB-VSX 1.0-1 is the first release of this test suite for the POSIX.1
coverage for the Linux Standard Base. LSB-VSX is built using the VSXgen
(the generic VSX test framework), with the VSX-PCTS 4.4.4 test set
merged into it. The test coverage corresponds to the core "classic"
POSIX.1 (including ISO C).
This test suite has not yet been formally approved by the Linux Standard
Base team and no claims should be made of LSB conformance to this version
of the test suite. The test assertions being tested are included with
the distribution in the MAN/tset directory.
1.1 Changes since the last release
The following changes have been made since the last
release:
This version of the VSX-PCTS has been setup to autoconfigure
on Linux systems. A front end script install_wrapper.sh is used
to auto install, setup and run the test suite.
1.2 Status
A few bugs were found in the suite during the port which were verified
with the official VSX-PCTS support team and fixed; for example the
namespace test tool did not understand GNU header files correctly
(#include_next and other features). Also a bug shown up by fclose()
behavior in glibc was fixed.
Pseudottys versus Real Ttys:
The tests for the tty subsystem run best when using real tty ports,
however since many test configuration may not have two tty ports and a loopback
cable available we have enabled the tests to run using pseudottys.
This does mean that users will encounter more failures when running using
pseudottys than real tty ports, and this is because pseudottys are
unable to emulate all the hardware characteristics of real ttys. The
list of expected failures is given with the release in the FAILURES.VSX4
file.
The remaining failures have been categorised as Linux non-compliance
issues, specification issues and test suite issues.
1.2.1. Linux compliance Issues:
Kernel level problems : Incorrect errno values being set (eg dup2, unlink)
Header file inconsistencies between the limits in limits.h
and the kernel returns from sysconf() (eg LINK_MAX).
For the tests related to mount points, the system is setting errno to
the wrong value when an attempt is made to remove a directory that is
a mount point. Since the directory exists, ENOENT is not appropriate.
The calls should be setting errno to EBUSY (if they fail).
Kernel inconsistency?
For directories in use by another process, it seems that rename() gives
EBUSY when it attempts to remove this type of busy directory, but remove()
and rmdir() can successfully remove them. (Otherwise rmdir 7 and remove 7
would show two failures, the same as in rename 13). This is a strangely
inconsistent implementation. Every implementation we have encountered so
far has treated all attempts to remove a busy directory in the same way,
regardless of which system call is used. That is why VSX4 only has the
one VSX_REMOVE_DIR_EBUSY parameter for all the calls. One opinion of
this difference between the behavior is a bug in the implementation,
however, it isn't a non-compliance.
glibc related problems
Mostly header file problems (visiblity of function prototypes, etc) -
expected to be fixed in later releases
locale related failures need to be investigated further , experiments
on glibc2.2 appear to have resolved these, they could be a test locale
setup problem
Some of the above problems may be fixed in 2.4.0 and many are reportedly
fixed in glibc 2.2 to be released soon.
1.2.2. Specification Issues
tmpfile() ignores umask. There is some debate at POSIX whether this
is a good idea or not, at the moment the specification requires tmpfile()
to honour umask.
time.h not explicitly allowed to be included in signal.h . This is
expected to be resolved in favour of the current implementation.
1.2.3. Issues Identified with VSX-PCTS
These are mainly setup related in areas that are allowed to change
for each operating system .
Two tests fail because of the way test suite is setup (for ease of use reasons,
allowing us to switch between suites).
(a) getlogin() does not match vsx0 user
(b) Home directory of vsx0 user is not the same as $HOME
These would pass once test suite setup correctly
Possible test locale setup problem (VSX4L2) under investigation (see
glibc problems listed before). At the moment we are unable to build
the locale, with localedef on glibc2.1 systems that has passed the
tests under glibc2.2.
2. Release Contents
The release consists of the following files.
LSB-VSX-1.0-1.tar.gz The source code to the suite in gzip'd tar format
vsxgde.ps The Users Guide in postscript
vsxgde.pdf The Users Guide in PDF format
relnote.txt These release notes
3. TET
LSB-VSX runs under TET3 (The Test Environment Toolkit test framework).
A suitable version of TET3 is bundled with the release. The latest
release can also be obtained at:
ftp://ftp.rdg.opengroup.org/pub/TET3/
TET3 must be configured to build as the Lite version . Do not use
the Distributed version. For more information on TET , see
http://tetworks.opengroup.org .
4. Documentation
Two files are included with the release to ease installation and
test setup: INSTALL.SETUP and INSTALL.VSX4. Please follow the instructions
within these to install, configure and execute the tests.
For more in depth information consult the User's Guide which provides
comprehensive information on the installation, configuration and use
of this test suite.
5. Support
Support related questions and problem reports should be directed to by
email to ajosey@rdg.opengroup.org.
6. Size
The size of LSB-VSX when the suite is built is approximately 40MB. The
size of the compressed distribution is roughly 3.5 MB. When running the
complete test suite, journal files will be produced that are typically
less than 1MB in length, so its recommended that at least 20MB of disk
space be allowed for a test campaign.
7. Time
The test suite takes approximately six to eight hours to run depending on the
speed of the platform under test. The signals tests are known to take
several hours to run.
Licensing information.
See the Licence file included in the distribution for licensing
information. At the moment this is the Artistic License. The Open
Group intends to change the license to that recommended by the Linux
Standard Base team.
-----
Andrew Josey The Open Group
Director, Server Platforms Apex Plaza,Forbury Road,
Email: a.josey@opengroup.org Reading,Berks.RG1 1AX,England
Tel: +44 118 9508311 ext 2250 Fax: +44 118 9500110
Reply to: