Re: [Delayed] Summary of the web team BoF at DC18

[Wouter Verhelst <wouter@debian.org>]

On Wed, Sep 26, 2018 at 07:36:03PM +0200, Wouter Verhelst wrote:
> I have been working on improving the setup to split out a full build
> into multiple jobs -- see
> https://salsa.debian.org/wouter/webwml/pipelines/19296 for an example.
> Since the timeout is per job and not per pipeline, that would work
> around this problem. However, it doesn't seem to be working properly
> yet; the "make install" step seems to be calling "wml" occasionally,
> which means things take time, and we don't want that (because it would
> make things time out). I'm not exactly sure whether that's my fault or
> the build system's.

I've now fixed this by making every individual language step do "make
install" rather than just "make all", and by having the "pages" phase
rsync each individual language directory into the main "public"
directory, which should then do the final publication.

It works, except that deploying the created output fails with an error
message that the produced data is too large. I'm sure this is something
which the salsa administrators could fix if we asked them nicely.

At any rate, I plan to replace the code in my merge request with the
code from this branch[1], it seems like a better option.

There are still a few things that could be done to improve this,
however:

- The webwml Makefile is currently set up to ignore failures and
  continue. For a continuous integration system, this is suboptimal,
  although I understand that it makes sense for the main website. I
  would like to suggest that ignoring failures is not done if some
  environment variable is set; the CI jobs can then just set that
  variable and allow things to fail if something is broken.
- The jobs start off with running "apt-get install" for various
  packages. This works, but it is more efficient to create an image on
  the docker hub which already has those packages preinstalled. I'll
  look at doing so.
- I'm not sure what happens if changes are made to multiple languages in
  a single git push. There are two possibilities: either gitlab will
  merge pushed caches from a single stage so that the cached result will
  have all files from all translations (which is good, that's what we
  want), or pushing a cache will overwrite all other caches from the
  same phase (in which case the publish stage will just push the English
  version of the site together with and whichever translation happened
  to be finished last). I'll need to experiment and see if that needs
  fixing. Even if the latter of those two possibilities is the case,
  however, I still think that seeing an auto build break is a good idea,
  so regardless of that I still think this is progress.

Thoughts?

[1] on the train currently, no network, I'll try to remember to push
    when I get to a network connection, but I might forget, yada yada.

-- 
Could you people please use IRC like normal people?!?

  -- Amaya Rodrigo Sastre, trying to quiet down the buzz in the DebConf 2008
     Hacklab