On Fri, May 01, 2020 at 10:09:44PM +0200, Christian Kastner wrote: > I would like to initial a discussion on -devel on the possible merits of > packages shipping benchmark tests, specifically in the form of autopkgtests. > > Such tests being beyond the scope of autopkgtest, I would like to > solicit the maintainers' feedback before doing so. > > Specifically, I would propose using the following new restrictions > (unknown to autopkgtest, and therefore skipped by default): > > benchmark-task A typical task for which the package is used > benchmark-io An I/O intensive task > benchmark-network A task that requires connectivity > > I see at least three use cases for this: > (1) On the same platform: comparison of the effects of build > parameters, eg: > * compiler optimization flags (various ISAs) > * effects of features (eg: spectre/meltdown prevention) > (2) Comparing platforms to each other > (3) Baselining the current system, which could be interesting > especially in the cloud computing context, eg: > * Local storage speed > * Network throughput, latency > > As a concrete example: The maintainer of gzip could a "benchmark-task" > test for a typical compression scenario. > > Do you think that pursuing this would be worthwhile? Would you have any > objections or concerns regarding the suggestion of using autopkgtest to > this end? This looks useful. ATM autopkgtest does not have a mechanism for enabling unknown restrictions to support actually running those tests, but it could gain one more or less easily.
Attachment:
signature.asc
Description: PGP signature