[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Moving KGB service to Debian hardware

(please CC on replies)


We'd like to get some advise on the plans to move the KGB service 
provided for alioth/salsa-hosted projects and others, to 
a debian-hosted machine.


Current architecture

Three IRC bots, running on home servers. Client part is chained in 
post-commit hooks, running on the Git[1] server (alioth and others).

 [1] the client also works with Subversion and CVS (used by the web 

When a push/commit/checkin is made, the client processes the change 
and sends one of the bots a notification to be relayed on IRC. If 
there is a problem with a given bot, it retries with another, until 
the list is over, at which point a warning is issued.

Each repository needs to be configured on the bot side, so that 
requests can be authenticated. Every repository is given access to 
a fixed number of IRC channels. All requests are authenticated using 
a hash over the request content and a pre-shared secret.

Apart from post-commit hooks, there is a separate functionality which 
allows sending free-text messages to IRC, suitable for purposes other 
than commit notification - bug submission, monitoring alert, job 
completion etc.

This approach has two downsides:

 - every new project needs to be configured by bot admins (e.g. three 

 - sometimes bots are inaccessible (e.g. power outage or hardware/link 
   problems). This is somewhat mitigated by the client retrying with 
   a different bot, but there is still a delay.

The future

Most of the projects move to salsa, which doesn't support post-commit 
hooks, but can use webhooks.

For these, a webhook termination is implemented in the bot directly. 
The webhook termination code doesn't need authentication, because it 
trusts salsa's IP address. The variable parts of the notification 
(e.g. channel/network, use of colors etc) can be embedded in the 
webhook URI. We trust the projects not to abuse the system by 
broadcasting advertisements to thousands of IRC channels (because they 
obey DMUP, and there is a rate limit).

On service maintainers' side, this avoids the need to configure 
individual salsa projects. For users it is also much easier to enable 
IRC notifications - instead of fiddling with post-commit hooks and 
their configuration they enable the webhook support in gitlab as 
described in https://salsa.debian.org/kgb-team/kgb/wikis/usage

The "old" scheme with authenticated projects stays, because it would 
be used by the non-gitlab projects and the projects that need 
free-text notifications.

To address both the stability issue with home servers and the need to 
configure several servers, we'd like to move the service to a single 
debian-hosted machine.

Hosting requirements

The expected load is rather low, CPU% is seldom higher than 0.0. 
Memory footprint was seen to be 200M after 30 days of running, and the 
expected network bandwidth is about xxx MB/month.

In terms of access, we would want to be able to:

 - install/upgrade kgb-bot and kgb-client packages (plus 
   dependencies), possibly from backports or unstable (or from git)

 - read/modify KGB's configuration:

   - modify files under /etc/kgb-bot/*

   - create/modify/delete files under /etc/kgb-bot/kgb.d/

   currently permissions look like

   drwxr-xr-x root root        /etc/kgb-bot
   -rw-r----- root Debian-kgb  kgb.conf
   drwxr-x--- root Debian-kgb  kgb.conf.d
   -rw-r----- root Debian-kgb  kgb.conf.dpkg-dist
   -rw-r----- root Debian-kgb  3dprinter.conf
   -rw-r----- root Debian-kgb  android-tools.conf
   -rw-r----- root Debian-kgb  apt.conf
   -rw-r----- root Debian-kgb  archlinux-reproducible.conf
   -rw-r----- root Debian-kgb  boinc.conf

 - read /var/log/kgb-bot.log* (-rw-r--r-- Debian-kgb:root)

For the KGB service admins,
    Damyan Ivanov

Attachment: signature.asc
Description: PGP signature

Reply to: