Re: Dealing with problematic unit tests from upstream (was: Network access during build)

[Thomas Goirand <zigo@debian.org>]

On 09/13/2016 02:44 AM, Zlatan Todoric wrote:
> So, my solution to this would be to first politely talk to our upstream
> and have a statement (that we all together would make it as generic
> letter for outreach to our upstreams) why they  should change tests and
> if we can help them. Who know, maybe they even accept. If they don't, we
> just disable those tests or all tests.

Let's generalize and not just take the case of network access during
build, which is only one case.

What you propose here isn't always practical but sometimes work, at
least in my case (ie: packaging OpenStack). The "politely talk to our
upstream" only works so much if no patch is proposed. The biggest issue
is when we see the same problems over and over on many components of the
project. It may take a huge amount of time to have such discussions. So,
while engaging in a discussion is always good (which I try to do as much
as possible), it often isn't enough. When repeating problems happens
really too often, then I open a thread on the openstack-dev@ list, but
not everyone reads every message (too much traffic). What works best is
to open a CR (Change Request) on Gerrit. It may take a long time to have
it to go through, but that's what upstream prefers. There as well is a
kind of do-o-cratie.

Disabling all unit tests is really not acceptable in my case. There's
just too many moving parts, and quality would drop considerably. So that
really not an option. Disabling *selectively* some problematic unit
tests, knowing they have a low impact, is doable though.

Cheers,

Thomas Goirand (zigo)