Re: RFC: (ab)using autopkgtest for benchmarking
On 2020-05-01 22:09, Christian Kastner wrote:
> I would like to initial a discussion on -devel on the possible merits of
> packages shipping benchmark tests, specifically in the form of autopkgtests.
> [...]
> Specifically, I would propose using the following new restrictions
> (unknown to autopkgtest, and therefore skipped by default):
>
> benchmark-task A typical task for which the package is used
> benchmark-io An I/O intensive task
> benchmark-network A task that requires connectivity
>
> [...]
>
> Do you think that pursuing this would be worthwhile? Would you have any
> objections or concerns regarding the suggestion of using autopkgtest to
> this end?
I neglected to follow up on my initial proposal, which I rectify now.
I eventually rejected the idea because I believe that ultimately, it was
impractical.
My original line of thinking was from a package maintainer's point of view:
(1) maintainers are usually familiar with the software they package,
and are thus well qualified to define useful benchmark tests
(2) the workload of a broad test suite is distributed among many
maintainers.
However, switching to a user's point of view, when benchmarking a new
system, it seemed impractical to install a boat-load of packages (even
if gathered in a task-benchmark package or similar) just to run their
benchmark tests once.
Instead, users would probably prefer to install just one package with
many tests.
And while a benchmark result of eg "converting this video with ffmpeg
takes N seconds on this system" is something that is probably more
accessible to users than abstract test results, I believe that in the
end, the users /actually interested/ and able to extract value from
these benchmarks can easily also do that from abstract tests.
And to that end, there is already stress-ng which implements a ton of
functionality for performing these tests.
So in the end, I believe that creating a single package with scripts for
a variety of stress-ng runs (and also perhaps using other tools) would
be a simpler solution than extending individual packages.
Thank you all for the previous feedback!
Reply to: