Re: numerical comparisons or not? (platform compatibility during testing)
Hi Steve,
my personal suggestion would be to change the threshold for failing tests.
Numerical tests can be very sensitive to the platform or even sometimes
to the versions of dependent libraries.
Regards
Anton
Am Di., 18. Feb. 2020 um 16:24 Uhr schrieb Stephen Sinclair
<radarsat1@gmail.com>:
>
> Hi folks,
>
> I've got my package, siconos, working nicely now on the platform I use it on (amd64) but some inspection of the build logs has revealed that buildd fails on all other platforms.
>
> https://buildd.debian.org/status/package.php?p=siconos&suite=sid&comaint=yes
>
> In fact it does build fine, but the software includes some numerical tests that compare results of the various solvers against expected results. These apparently yield slightly different values on other platforms, and the tests fail.
>
> To get the software running on other architectures, would it be better to,
>
> 1. eliminate such tests.
> 2. change the numerical threshold for failing a test.
> 3. have different thresholds or even reference files per platform.
>
> In some sense (1) or (2) could be acceptable since it seems overly brittle to depend on specific numerical outcomes -- the intent of these tests is really for the developers to detect any unexpected changes in the solver outputs, they are not meant to gate deployment of the software.
>
> So perhaps it would be better for tests for buildd to focus mostly on whether the software runs without crashing, rather than comparing specific and very precise numerical outputs. In fact if I simply remove the reference files from the software, the tests will still run and pass without comparing the numerical output, but I am not sure if this is wise.
>
> regards,
> Steve
>
Reply to: