Re: vsx-pcts tests for binary test suite
hi Chris
Thanks. Detailed comments below, I have also copied this
to the spec team, as ultimately they need to decide if they
make divergences from POSIX official (and hence required )
First some thoughts on the test strategy.
When we say remove, we mean just remove from the default
scenario for LSB testing, we need to leave the POSIX test modes
intact as they are.
I would suggest that we take a copy of the scen.exec file
and copy that to be the lsbscen.exec file,
and then change our tcc wrappers that run the suite to
invoke that scenario.
In the lsbscen.exec file:
We'd need to adjust the test totals,
"total tests in ANSI.os 1244"
and use a {} notation in the testcase lines to omit tests,
for example
/tset/ANSI.os/charhandle/Misalnum/T.isalnum{1,3,5}
/tset/ANSI.os/charhandle/Misalpha/T.isalpha{1-5}
This avoids us having to change the tests themselves for
the omissions.
If we want to fork a test, then we should fork a branch of
the tree and create an equivalent branch under LSB.os leaving
the original POSIX tests as they are.
# test 2 is an LSB specific change
/tset/LSB.os/charhandle/Misalpha/T.isalpha{2}
Details below:
On May 15, 11:12pm in "vsx-pcts tests for b", Chris Yeoh wrote:
> As part of the process of putting together a binary test suite package
> we need to decide what to do with the tests that are failing (ie
> remove/modify test or possibly fix behaviour).
>
> I recently ran the test suite against an x86 system running 2.4.4 and
> glibc 2.2.3. The following is a list of most of the tests that
> reported failed, unresovled or warning (do we need to worry about
> these?) and suggestions on how we treat each one.
>
>
> Test which fail:
> /tset/ANSI.os/maths/pow/T.pow 8
>
> This test should remain. On x86 test system pow(double(1.0), NaN)
> returns 1.0. Should be reported to the glibc folks to be fixed.
>
>
> /tset/ANSI.os/streamio/tmpfile/T.tmpfile 2
>
> tmpfile() ignores umask for group/other component. This is intentional
> behaviour of glibc for reasons of security. This test should be
> removed (or rewritten to expect current behaviour).
>
This is what POSIX requires, so if we change the behavior
we need to fork the test under LSB.os/ANSI.os/
>
> /tset/POSIX.os/dataform/cpio/T.cpio 26
> /tset/POSIX.os/dataform/tar/T.tar 5
> /tset/POSIX.os/dataform/tar/T.tar 6
> /tset/POSIX.os/dataform/tar/T.tar 8
>
> The above tests only fail if cpio/tar are used for the tests instead
> of pax. I propose that these we do not require he use of pax and
> remove these tests.
>
This is because the formats produced by native tar and cpio are not
POSIX compliant, it would be better if tar and cpio were fixed, most
commercial operating systems have had these fixed for the best part
of 10 years or more
>
> /tset/POSIX.os/dataform/tar/T.tar 9
>
> "devmajor" field "" is not a zero-filled octal number
> "devminor" field "" is not a zero-filled octal number
>
> This test fails if tar or pax are used. Propose to remove/waiver
> this test.
Another POSIX compliance failure
>
>
> /tset/POSIX.os/devclass/c_iflag/T.c_iflag 19
> /tset/POSIX.os/devclass/c_lflag/T.c_lflag 6
> /tset/POSIX.os/devclass/tcdrain/T.tcdrain 2
>
> These fail only if pseudo ttys are used during testing
> instead of a loopback serial cable. Propose we remove these tests.
>
We should probably provide a lsbscen.exec for testing with
pseudottys and remove those tests using the {} notation in that scenario.
>
> /tset/POSIX.os/devclass/tcsetpgrp/T.tcsetpgrp 6
>
> tcsetpgrp(fd, 0) gave errno 1 (EPERM), expected EINVAL
>
> No known fix available for this. May need to accept that this will
> fail?
>
POSIX compliance issue
>
> /tset/POSIX.os/files/unlink/T.unlink 19
>
> Attempt to unlink a directory returns EISDIR instead of EPERM. Easy to
> fix, but maybe hard to get a patch in since EISDIR kind of makes more
> sense. Suggest we accept that EISDIR will be returned.
>
POSIX compliance issue
>
> /tset/POSIX.os/ioprim/fcntl/T.fcntl 37
>
> fcntl(6, F_DUPFD, 1024) did not return expected values
> ERRNO VALUES: expected: 22 (EINVAL), observed: 24 (EMFILE)
>
> No known patch available though likely not hard to fix.
>
POSIX compliance issue
>
> /tset/POSIX.os/ioprim/read/T.read 13
>
> time stamps incorrect: atime was not updated
> time stamps incorrect: atime was not updated
>
> Introduced between 2.2 and 2.4 kernels. atime of
> pipes/fifos not updates on read
>
POSIX compliance issue
>
> /tset/POSIX.os/procenv/sysconf/T.sysconf 10
>
> sysconf(_SC_TZNAME_MAX) failed expected >= 6 or -1, got 3
> Need to report to glibc folks?
>
> Given this basically correlates header file to runtime value can this
> test be removed (since for the binary test suites we're not testing
> header files)?
>
Agreed , if this is looking at a header value, then it should be
removed from the binary test suite.
>
> /tset/POSIX.os/procprim/sigconcept/T.sigconcept 35
>
> These fail only if pseudo ttys are used during testing
> instead of a loopback serial cable. Propose we remove these tests.
>
As above re: pseudotty
>
>
> Test which report unresolved:
> /tset/POSIX.os/devclass/i_close/T.i_close 1
> /tset/POSIX.os/devclass/i_spchars/T.i_spchars 1
> /tset/POSIX.os/devclass/i_spchars/T.i_spchars 3
> /tset/POSIX.os/devclass/i_spchars/T.i_spchars 5
> /tset/POSIX.os/devclass/i_spchars/T.i_spchars 14
> /tset/POSIX.os/devclass/i_spchars/T.i_spchars 17
> /tset/POSIX.os/devclass/tcdrain/T.tcdrain 7
> /tset/POSIX.os/devclass/tcflow/T.tcflow 2
> /tset/POSIX.os/devclass/tcflow/T.tcflow 6
> /tset/POSIX.os/devclass/tcflow/T.tcflow 7
> /tset/POSIX.os/devclass/tcflow/T.tcflow 8
> /tset/POSIX.os/devclass/tcflow/T.tcflow 13
> /tset/POSIX.os/devclass/tcflow/T.tcflow 14
> /tset/POSIX.os/devclass/tcflow/T.tcflow 15
> /tset/POSIX.os/devclass/tcflow/T.tcflow 16
> /tset/POSIX.os/devclass/tcflush/T.tcflush 1
> /tset/POSIX.os/devclass/tcflush/T.tcflush 2
> /tset/POSIX.os/devclass/tcflush/T.tcflush 3
> /tset/POSIX.os/devclass/tcflush/T.tcflush 4
> /tset/POSIX.os/devclass/tcflush/T.tcflush 5
> /tset/POSIX.os/devclass/tcflush/T.tcflush 6
> /tset/POSIX.os/devclass/tcflush/T.tcflush 10
> /tset/POSIX.os/devclass/tcflush/T.tcflush 11
> /tset/POSIX.os/devclass/tcflush/T.tcflush 12
> /tset/POSIX.os/devclass/tcflush/T.tcflush 13
> /tset/POSIX.os/devclass/tcsetattr/T.tcsetattr 1
> /tset/POSIX.os/devclass/tcsetattr/T.tcsetattr 13
> /tset/POSIX.os/procprim/sigconcept/T.sigconcept 30
> /tset/POSIX.os/procprim/sigconcept/T.sigconcept 31
> /tset/POSIX.os/procprim/sigconcept/T.sigconcept 34
> /tset/POSIX.os/procprim/sigconcept/T.sigconcept 36
> /tset/POSIX.os/procprim/sigconcept/T.sigconcept 37
>
> These fail only if pseudo ttys are used during testing
> instead of a loopback serial cable. Propose we remove these tests.
As above re pseudotty
>
> /tset/POSIX.os/procenv/sysconf/T.sysconf 3
>
> CLK_TCK - comment in header file says this is now obsolete?
>
Obsolete still usually means it should be present.
However I personally do not think its a big issue.
>
> Warnings:
> /tset/ANSI.os/maths/pow/T.pow 4
>
> pow(0.0, -1.0) gave
> RETURN VALUES: expected: -inf, observed: inf
> Bit Representation: expected value:
\000\000\000\000\000\000\360\377
> Bit Representation: observed value:
\000\000\000\000\000\000\360\177
> pow(0.0, -1.0) gave
> RETURN VALUES: expected: -inf, observed: inf
> Bit Representation: expected value:
\000\000\000\000\000\000\360\377
> Bit Representation: observed value:
\000\000\000\000\000\000\360\177
>
I think this is fine
>
> /tset/ANSI.os/streamio/fopen/T.fopen 28
>
> fopen() succeeded when FOPEN_MAX streams were already open
>
Also this one,
regards
Andrew
Reply to: