[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Not running tests because tests miss source code is not useful





On വെ, ഒക്ടോ 8 2021 at 10:31:16 രാവിലെ +0200 +0200, Thomas Goirand <zigo@debian.org> wrote:
On 10/7/21 11:40 AM, Pirate Praveen wrote:


On 7 October 2021 3:02:55 am IST, Thomas Goirand <zigo@debian.org> wrote:
 On 10/6/21 6:53 PM, Pirate Praveen wrote:
 [adding -devel]

On ബു, ഒക്ടോ 6 2021 at 12:16:07 വൈകു +0200 +0200, Jonas Smedegaard
 <jonas@jones.dk> wrote:
 Quoting Yadd (2021-10-06 11:43:40)
  On Lu, 04 oct 21, 16:40:48, Bastien Roucari�s wrote:
  > Source: src:node-lodash
  > Version: 4.17.21+dfsg+~cs8.31.173-1
  > Severity: serious
  > Justification: do not compile from source
  >
  > Dear Maintainer,
  >
  > The vendor directory should be emptied
  >
> The debug version is compiled without source (lintian warn) and
 moreover the
  > rest of file are already packaged
  >
  > grep -R vendor * gives only a few hit that could be cured by
 symlinking
  >
  > Bastien
  Hi,

this files are used for test only, maybe severity could be decreased.

I find the severity accurate: Relying on non-source code is a severe violation of Debian Policy, not matter the purpose of relying on it.

I think we should change the policy here. Running tests helps improve the quality of the software we ship. Many times the vendored code is used to ensure the code does not break in a specific situation. I don't
 think reducing test coverage in such situations is really helpful.

 Right, running tests helps improve the quality of software we ship.
Which is why you probably need to test using what's shipped in Debian
 rather than using a vendored source-less code.

 We are not shipping the source less code.

You are: Debian also ships source code.

I meant, not shipping in any binary package. Though as Russ mentioned in his reply. I will propose a GR.

This is used only during tests. I don't think we are not gaining anything by removing tests here. Just making it harder for the package maintainer to run tests.

You would not gain anything by removing tests, but you would win by
making these tests completely free software.

I am just saying it increases the work required to run tests and when disabling tests is an option, the incentive is to disable tests.

If we rely on non-free code for tests, that's really bad too, and that must be avoided just like we're avoiding source-less code everywhere
 else in Debian. The policy shall not change, please.


The code is not non-free here, just a specific version of a Free Software code built outside Debian.

We build from source...

We build the binary packages from source. I don't think it is useful to extend that to tests without considering the tradeoffs involved.

I think tools required for tests should be considered separately from tools required to compile. I think it should be treated similar to test data.

I don't agree.

ok, lets see how the whole project feels via a GR and settle it. I just expressed my opinion, you expressed yours and we need to make a decision now.

What you are proposing would require the package maintainer to adapt these tests to versions available (many times with different API versions) in Debian and the easier choice is disabling tests.

No. I believe it's ok to have an embedded version of the JS files in the upstream code. This is a *very* different issue, please do not mix them.
What I don't like is using a minified version of the JS files. That's
*very* easy (hum... trivial?) to add a non-minified version in your
Debian folder, and use that for tests. You don't care if running the
tests is a little bit slower (because using a source-full version), do you?

I don't think you really understand the complexities here. Building the minified version is not just running the minifier against the non minified code. The non minified code itself is generated using many other tools (typescript, transpiled using babel, bundled using rollup or webpack etc - many times the versions of these tools are very much different versions as well).


However, there's this:

On 10/7/21 6:17 PM, Richard Laager wrote:
Running tests against vendored dependencies one isn't going to use at
 run-time is of limited usefulness.

Best is, if you can, use the library packaged separately, in Debian,
both for tests, and runtime. This way, you do ensure that:
- patching Debian for security is still a thing
- the package can run with the Debian version of the lib


You are completely missing the reality here as well. The runtime dependencies are already used from the packaged versions. These vendored libraries are used only to create specific test cases or sometimes using alternative implementations to test the shipped code.

I think it's less grave than just saying "oh, we don't care about these
binary blobs, there's just for tests...". It's even worse, because by
using a different version for tests and runtime, you're faking tests...


See above. All runtime dependencies are packaged and used from packaged versions. In many cases the code is used to building dummy apps to create specific test cases.

If the lib are just use for tests and nothing else (ie: not for
runtime), then back to square one: it's ok to ship the non-minified
version in your debian folder, and use that for running tests. It's also
super easy and fast to implement.

See above explanation. It is not as simple as dropping the non minified source code.


I think blindly applying a rule without thinking of any consequences is bad too.

I think blindly saying "oh, it's ok, it's only test things..." is a
*very* dangerous path that I would like Debian to avoid.

Just because it is bad in one situation does not mean it will be bad in every situation. We should evaluate pros and cons of each situation before making a decision. Blind faith is more suitable for religions and not for a project like ours.

Sorry, but using free software from source is *NOT* opened for debate.
If you would like to do that, choose another distribution. We all
signed-up for it, when becoming DDs, this is the foundations of Debian.

You are blindly equating the binary packages we ship with tests used to check if the built software is fine or not. Lets see how the project feels about it via a GR.

I think a nocheck build profile which excludes these files from build is sufficient to ensure we are not using these to create binary package.

What's the problem with using a non-minified version of the files? It's
not difficult, and it doesn't take too much of your packaging time.


Already explained earlier. You are commenting on a situation which you don't understand.

This way we guarantee only packages in main is used to generate the binary, but still allows to run tests optionally making it easy to find problems, especially during transitions. Currently when tests are missing transitions are harder because we can't find breakages easily since tests are disabled.

What you're proposing is making the life of anyone willing to modify the
software harder.

You are simply making false equivalance. Nothing stops anyone from writing different tests or even modifying the tests or creating different test cases. It just makes modifying one particular test case different, which is anyway an arbitrary condition. To test a software you don't really need one unique test case.


 The current policy is not making Debian better.

It is. It's ensuring we build from source, and we're able to modify
what's in Debian with reasonable conditions.

We are just insisting some specific test cases are constructed from source, where as there is no real requirement for test cases to be built like that. Anyone can build different test cases if required.



Reply to: