wanted: recommendation for webhook queueing library/thing
Sean Whitton writes ("git & Debian packaging sprint report"):
> Main achievement
> ----------------
>
> We designed and implemented a system to make it possible for DDs to
> upload new versions of packages by simply pushing a specially formatted
> git tag to salsa.
>
> Please see this blog post to learn about how it works:
> https://spwhitton.name/blog/entry/tag2upload/
>
> While the cloud service part of this system has not yet been deployed,
> and so you can't just tag to upload yet, the blog post explains how you
> can run the cloud service in an ad-hoc mode on your laptop, and thereby
> get a feel for how it works.
>
> You can also read git-debpush(1) in sid.[1]
The one technical piece missing for deployment is webhook queue /
dispatcher. Can someone familiar with this space recommend one ?
In more detail, here is what I think the problem is:
* The webhook client expects a quick response. But the bot's
processing for a tag addressed to the bot takes some time (because
it needs to fetch a fair amount of data and build a source
package). So webhook calls must be queued.
* Sometimes it will be necessary to retry a failed bot invocation.
(The bot already knows whether a particular invocation should be
retried, and can feed this back to the queue.) The retry interval
and number of retries must be limited.
* The bot expects to be forked/exec'd.
* Significant protection against DoS is not needed because the
web hook receiver's webserver will have a firewall protecting it
from anything other than salsa.
* The webserver and web hook receiver/queue (including the JSON
parser which will be needed) must be high quality software, because
we really want to avoid any vulnerabilities.
* I can easily change the bot's fork/exec API to make it conveniently
fit into whatever queue system.
If necessary I can write my own but it seems like this is a problem
that there should already be a good solution to.
Thanks,
Ian.
Reply to: