On Wed, Jul 10, 2019 at 04:09:11PM +0100, Ian Jackson wrote:
> Sean Whitton writes ("git & Debian packaging sprint report"):
> > Main achievement
> > ----------------
> >
> > We designed and implemented a system to make it possible for DDs to
> > upload new versions of packages by simply pushing a specially formatted
> > git tag to salsa.
> >
> > Please see this blog post to learn about how it works:
> > https://spwhitton.name/blog/entry/tag2upload/
> >
> > While the cloud service part of this system has not yet been deployed,
> > and so you can't just tag to upload yet, the blog post explains how you
> > can run the cloud service in an ad-hoc mode on your laptop, and thereby
> > get a feel for how it works.
> >
> > You can also read git-debpush(1) in sid.[1]
>
> The one technical piece missing for deployment is webhook queue /
> dispatcher. Can someone familiar with this space recommend one ?
>
> In more detail, here is what I think the problem is:
>
> * The webhook client expects a quick response. But the bot's
> processing for a tag addressed to the bot takes some time (because
> it needs to fetch a fair amount of data and build a source
> package). So webhook calls must be queued.
>
> * Sometimes it will be necessary to retry a failed bot invocation.
> (The bot already knows whether a particular invocation should be
> retried, and can feed this back to the queue.) The retry interval
> and number of retries must be limited.
>
> * The bot expects to be forked/exec'd.
>
> * Significant protection against DoS is not needed because the
> web hook receiver's webserver will have a firewall protecting it
> from anything other than salsa.
>
> * The webserver and web hook receiver/queue (including the JSON
> parser which will be needed) must be high quality software, because
> we really want to avoid any vulnerabilities.
>
> * I can easily change the bot's fork/exec API to make it conveniently
> fit into whatever queue system.
>
> If necessary I can write my own but it seems like this is a problem
> that there should already be a good solution to.
Enrico has been producing some content that is related:
https://www.enricozini.org/blog/2018/debian/automatic-deploy-from-gitlab-salsa-ci/
https://debconf18.debconf.org/talks/71-autodeploy-from-salsa/
It's based on systemd .path units. the advantage is that you don't need
a new component, you only put the config in place and systemd takes care
of it for you.
another option is rabbitmq; that's what debci uses. and it has clients
for several languages. one advantage is that you can have multiple
worker processes consuming from the queue in parallel, but OTOH you have
do manage those extra daemons, care about not exposing it to the
internet, etc.
Attachment:
signature.asc
Description: PGP signature