[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Bug#1059633: paperwork: flaky autopkgtest: test_import_duplicated_docx: len(self.add_docs) is 2, should be 1



Source: paperwork
Version: 2.1.2-1
Severity: important
User: debian-ci@lists.debian.org
Usertags: flaky
X-Debbugs-Cc: debian-ci@lists.debian.org
Control: affects -1 + src:gtk+3.0 src:gobject-introspection

The paperwork test suite seems to contain an unreliable ("flaky") test
which intermittently fails on at least armel:

397s ======================================================================
397s FAIL: test_import_duplicated_docx (docimport.tests_converted.TestConvertedImport.test_import_duplicated_docx)
397s ----------------------------------------------------------------------
397s Traceback (most recent call last):
397s   File "/tmp/autopkgtest-lxc.dcawwldu/downtmp/autopkgtest_tmp/paperwork-backend/tests/docimport/tests_converted.py", line 162, in test_import_duplicated_docx
397s     self.assertEqual(len(self.add_docs), 1)
397s AssertionError: 2 != 1

This means that when packages that are depended on by the test suite,
such as gobject-introspection and gtk+3.0, are trying to migrate to
testing, the test will randomly pass or fail, causing those packages to
be detected as having caused a regression when in fact they have not.

For example see
https://ci.debian.net/packages/p/paperwork/testing/armel/41238454/
(while testing a new gtk+3.0) and
https://ci.debian.net/packages/p/paperwork/testing/armel/41380886/
(while testing a new gobject-introspection).

If this test or this feature cannot be made fully reliable, one option
is to skip it in debian/tests/unittests, and re-run it in a separate
autopkgtest script that is marked as "Restrictions: flaky". A few GNOME
packages use the convention that flaky tests are marked like this:

    @unittest.skipIf('DEB_ALLOW_FLAKY_TESTS' not in os.environ, 'https://bugs.debian.org/123456')
    def test_something_flaky(...):
        ...

so that they are normally skipped, but we can force them to be run
(to assess whether they are still a problem!) by using
"export DEB_ALLOW_FLAKY_TESTS=1".

    smcv


Reply to: