Re: Preferred form of modification for binary data used in unit testing?
On 7/16/20 5:00 PM, Johannes Schauer wrote:
> Hi,
>
> Quoting Christian Kastner (2020-07-16 14:08:34)
>> On 2020-07-16 12:53, Pirate Praveen wrote:
>>>> Generally speaking, I think it's a mistake to apply the question of
>>>> "preferred form for modification" to unit test payloads. Unit tests are
>>>> purely about functionality. The original source to a payload is an
>>>> arbitrary choice (possibly even randomly generated), and could be
>>>> replaced with any other appropriate arbitrary choice at no detriment to
>>>> the software or the user.
>>> I think this needs to be clearly documented in policy. I don't think
>>> this interpretation is generally accepted. I have seen many cases where
>>> tests are disabled for this reason.
>> Perhaps I spoke too generally. For example, I can see, as one of
>> probably many counter-examples, the case where the input is not
>> completely arbitrary (eg: input is a captured stream).
>>
>> But to take the other extreme, using completely arbitrary data, as an
>> example: say my code implements a ROT13 function and I create a test for
>> it using a blob of random data as well as the expected output.
>>
>> That random data was generated somehow, eg: using Python's random
>> module, and could therefore be regenerated given the correct program and
>> seed. However, I did not include the code to generate that data.
>>
>> Would we really reasonably expect anyone to act upon that random blob in
>> any way?
>
> I have another data point with one of my packages (genext2fs) where I made a
> contribution to upstream. Their unit tests execute the program with some input
> and a given set of parameters and then check that the md5sum of the created
> ext2 filesystem image matches the expected value. Without thinking, I added the
> following into their test script:
>
> H4sIAAAAAAAAA+3WTW6DMBAF4Fn3FD6B8fj3PKAqahQSSwSk9vY1uKssGiJliFretzECJAYeY1s3JM4UKYRlLG7H5ZhdTIHZGevK+ZTYkgrypRFN17EdlKIh5/G3++5d/6N004qbA47er8/fWVduV2aLD7D7/A85C88Ba/ufA/sQIhk25VdA/2+h5t+1gx4/pd7vfv+Hm/ytmfNH/8vr+ql7e3UR8DK6uUx9L/uMtev/3P8p+KX/oyHlZMuqntX/9T34Z9yk9Gco8//xkGWf8Uj+Mbpl/Y+JVJQtq9r5/K+bj3Z474+Xk9wG4JH86/rvyzxAirfYnOw+/+vXWTb+uv9PaV3+JfiSv/WOlJVPf/f5AwAAAAAAAAAAAMD/9A0cPbO/ACgAAA==
>
> This is a base64 encoded gzipped tarball with a few test files in it. I
> generated it using GNU tar but since I found it likely that a GNU tar version
> in the future (or the past) will produce a slightly different tarball and
> because I needed some fixed input without different output on systems without
> GNU tar (like BSD or MacOS) or on older systems or on future systems, I just
> dumped that binary blob into the upstream software. In the meantime, that
> binary blob is even in the Debian package:
>
> https://sources.debian.org/src/genext2fs/1.5.0-1/test.sh/#L89
>
> The curious thing for me personally is, that I didn't feel bad about this at
> all and at no point from writing the code up to me packaging and uploading the
> Debian package containing the blob, I thought even twice about whether this is
> DFSG compliant or not. Only now after having read this thread I start wondering
> whether I have actually created an RC bug myself. Did I? I love the principles
> of the DFSG and it really surprises me that despite my love for these freedoms
> I didn't think twice about including that binary blob instead of generating it
> on the fly. Was my mind fooled by how short the blob is? A perl script
> generating the tarball such that it's bit-by-bit identical across all platforms
> would be longer than this blob.
>
> What do you guys think? Should I put work into writing a script which produces
> above binary blob as part of the test suite to avoid having my package be RC
> buggy? I would love to get some guidance.
>
> Thanks!
>
> cheers, josch
I'm not sure about this being DFSG compliant or not. Though it feels
like this bad practice anyways, because:
- It wont be obvious what you're doing, and someone will have to
reverse-engineer what you did to figure things out.
- probably there was ways to go around the "tar doesn't always produce
the same output" problem, like building the tarball, and compute its sum
during the test (I don't know enough details to be sure of this though...).
Cheers,
Thomas Goirand (zigo)
Reply to: