Re: Compile the package and run the testsuite as part of the debci
On 2019-09-28 20:16:07 [+0200], Paul Gevers wrote:
> Hi Sebastian,
Hi Paul,
> On 28-09-2019 14:04, Sebastian Andrzej Siewior wrote:
> > the openssl debci testsuite has now a subset (a real small one) compared
> > to the complete testsuite. I've been looking into running just the
> > testsuite without recompiling the whole thing but it is not that simple.
> > There is no cut at the moment. It assumes to use the openssl binary from
> > the build directory, it assumes a few compiled test programs and
> > "engines" which usually are in the "engine" folder. One test checks
> > available symbols which does not work once binary is stripped so I would
> > have to skip that one.
>
> How difficult is it to compile everything but e.g. replace the build
> binaries with a (soft)link to the installed binaries (and skip tests
> that aren't appropriate in that case)? I'm not sure if this covers what
> you meant above.
Recompiling everything (the main build, skipping the DI flavour) and
replacing the .so libs with system's would work. We would just to have
to skip the one test which will fail because the libs are stripped.
> > My question is whether it is okay build the library and everything
> > required to run the testsuite or not. I've seen some packages doing
> > this, like the python modules but then compile process takes a fraction
> > of time compared to the test-suite.
>
> This question has multiple sides. Depending on how long the build
> actually takes versus how much is tested, it's perfectly fine from
> ci.d.n side to build your package and test it from there (via the
> build-needed restriction). On the other hand, autopkgtests are really
> intended to test *as installed* packages.
The whole build on binet takes 11mins and that is the slowest buildd.
For CI it should be less because we would be only one flavour.
> As a third side, because of the way the results influence migration of
> packages from unstable to testing, also the opinion of the release team
> matters. Please be reminded that if you autopkgtest fails while testing
> a reverse dependency, than that reverse dependency will be blocked from
> migrating. As an example where this went wrong, a lot of packages in the
> KDE/Qt stack stopped running their autopkgtest because their tests also
> required building and due to the lock step nature of a lot of their
> packages, this often failed in the migration setup. Hence it was not
> robust as autopkgtest and they decided to drop their tests in a number
> of cases, the latest one I know of is src:kate.
Puh. OpenSSL has almost zero dependecies. So the kernel/libc/perl have
to be real bad for a test to fail which passed previously (at build
time). Would we test the result of the compile procedure then we would
also catch possible changes to gcc/binutils. Sadly I have to say that
only once a binutils change broke the testsuite and we "fixed" the
assembly code to work with newer binutils.
> That said, if the only reasonable way to substantially test you code is
> by building the package *and* if this has quite some added value over
> only doing this during build time, then I don't see a problem to do this
> in an autopkgtest. Are you able to elaborate on what you hope to achieve?
You are selling the -3 day to migration and I would like to buy. The
added value over doing it at build time would be to test newer
libc/kernel/gcc/binutils because we don't have much dependencies in
total.
> I would encourage you though to discuss this with your upstream such
> that in some future, you could actually test the installed packages with
> the test suite, as that is very much desirable. I am not guaranteeing
> that in the further future we will still allow this.
I see. Let me ping upstream and check with them.
> Paul
Sebastian
Reply to: