[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: On (semi-)automated testing and improved workflow of LTS uploads



Hi,

On Tue, 09 Jul 2019, Jonas Meurer wrote:
> 1. Upload packages targeted at LTS suites to some dedicated place for
>    automated testing
> 2. Run automatic tests (piuparts, autopkgtests, lintian?, ...)
> 3. If tests passed, publish the packages somewhere to do manual
>    testing (and reviews)
> 4. (Optionally?) demand acknowledgement by a second (different) LTS
>    developer
> 5. Automatically upload packages that got uploaded, passed tests and got
>    second acknowledgement to the targeted LTS upload queue
> 
> While that would be very nice to have, it's probably a long way to go
> until we have such infrastructure.

If we never start to work in this direction, it will be a long way, yes
;-)

> There seems to be some agreement that the first step would be to run
> (semi-)automated tests (e.g. piuparts and autopkgtests) against the
> packages before uploading them to ${LTS}-security, i.e. point 2 of the
> list above.

Right, how do we make sure that this has been done? Shall we require that
the logs of autopkgtest/piuparts have been uploaded somewhere?

That could be the start of the above infrastructure. Have a place where
we can create "updates" and upload artifacts related to each update.

> What's your thoughts on this? Do you think that we could implement
> most/all of the desired workflow using Salsa-CI/Gitlab-CI? Or would it
> be better to build it entirely independently of Salsa - e.g. implement
> it in dak?

I believe that the service should be separate and standalone. It should
not be part of dak either.

On Thu, 11 Jul 2019, Mike Gabriel wrote:
> > 2. Run automatic tests (piuparts, autopkgtests, lintian?, ...)
> 
> Maybe, probably not lintian. As the package maintainer, at the time
> (old)oldstable (meaning the current Debian LTS) turned from testing to
> stable, might not have had their packages in shape lintian-wise. Also only
> using an old lintian would be appropriate. A recent lintian on old packages
> just creates too much noise (which we won't fix anyway).

Running lintian might be interesting... but to compare the lintian output
on the current package and on the updated package.

> > 4. (Optionally?) demand acknowledgement by a second (different) LTS
> >    developer
> 
> Although demanding a second ACK adds an extra delay to our workflow, I sense
> that such a second pair of eyes peering at security patches might greatly
> improve the quality of the LTS work. Even if we don't come up with some
> auto-test engine, we should consider "peer-"reviewed uploads.

I'm not so sure about this. In some cases reviewing the patch itself and
the updated code is hard and I don't expect the reviewer to be willing to
invest that energy. But at least the reviewer can ensure that the tests
have been run, can double check whether we should add some autopkgtest
for extra safety, etc.

> > What's your thoughts on this? Do you think that we could implement
> > most/all of the desired workflow using Salsa-CI/Gitlab-CI? Or would it
> > be better to build it entirely independently of Salsa - e.g. implement
> > it in dak?
> 
> Personally, I think that using Salsa for this, adds an extra layer of
> complexity to the uploading workflow, because we have to pump all packages
> that we want to fix in LTS through GitLab.

So that means that we have to build some automation for this... that way
it could even be run by frontdesk whene it does CVE triage and add the
package to dla-needed.txt and the repository would be ready for work when
the contributors wants to work on it.

> the paid contributors. Some package uploads may even be embargoed, so
> generically, the LTS-team namespace on Salsa needs to be private (which
> excludes other contributors, also the usual package maintainers/uploaders,
> by default).

We can have a private sub-group for embargoed updates.

> As our intention is to operate on packages (not on upstream code in Git), so

We obviously work on source code... using git is not in opposition with
using an external infrastructure to review the updates. And the merge
request is the logical place where both parts meet.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/


Reply to: