gitlab CI configuration for the Debian website (Re: [Delayed] Summary of the web team BoF at DC18)
Hello Wouter, all
Sorry for taking so long to answer about this.
El 02/10/18 a las 17:31, Wouter Verhelst escribió:
> On Wed, Sep 26, 2018 at 07:36:03PM +0200, Wouter Verhelst wrote:
>> I have been working on improving the setup to split out a full build
>> into multiple jobs -- see
>> https://salsa.debian.org/wouter/webwml/pipelines/19296 for an example.
>> Since the timeout is per job and not per pipeline, that would work
>> around this problem. However, it doesn't seem to be working properly
>> yet; the "make install" step seems to be calling "wml" occasionally,
>> which means things take time, and we don't want that (because it would
>> make things time out). I'm not exactly sure whether that's my fault or
>> the build system's.
> I've now fixed this by making every individual language step do "make
> install" rather than just "make all", and by having the "pages" phase
> rsync each individual language directory into the main "public"
> directory, which should then do the final publication.
Thanks for all the work on this, I just merged it into the main
> It works, except that deploying the created output fails with an error
> message that the produced data is too large. I'm sure this is something
> which the salsa administrators could fix if we asked them nicely.
As we have talked in IRC during the tests, the current status is that
the CI jobs build the website but it is not published as "pages",
because of a size limitation (100MB) and (if I understood correctly) we
should review the documentation and try to use deployment environments:
Any help is appreciated on this, and I will do my best to reply quicker
with tests and feedback.
* we can learn if commits break the builds (I'm not sure who gets
notifications (I get them, but not sure if because of my particular mix
of notification-levels and project-membership), you can probably tweak
your profile settings in Salsa if you don't get them and want to get
them), and we can also have a look at the CI results prior to merge some
* People can browse/download the results of the CI builds in the Salsa
web interface, going to the webwml project, CI/CD, Jobs (or Pipelines).
Note that the artifacts corresponding to languages (other than English)
will exist (containing the /english folder) even if that language was
not built (if the language was built, the zip will contain the /english
and the /$language folders).
> At any rate, I plan to replace the code in my merge request with the
> code from this branch, it seems like a better option.
> There are still a few things that could be done to improve this,
> - The webwml Makefile is currently set up to ignore failures and
> continue. For a continuous integration system, this is suboptimal,
> although I understand that it makes sense for the main website. I
> would like to suggest that ignoring failures is not done if some
> environment variable is set; the CI jobs can then just set that
> variable and allow things to fail if something is broken
> - The jobs start off with running "apt-get install" for various
> packages. This works, but it is more efficient to create an image on
> the docker hub which already has those packages preinstalled. I'll
> look at doing so.
I will create bugs to track these two proposals.
> - I'm not sure what happens if changes are made to multiple languages in
> a single git push. There are two possibilities: either gitlab will
> merge pushed caches from a single stage so that the cached result will
> have all files from all translations (which is good, that's what we
> want), or pushing a cache will overwrite all other caches from the
> same phase (in which case the publish stage will just push the English
> version of the site together with and whichever translation happened
> to be finished last). I'll need to experiment and see if that needs
> fixing. Even if the latter of those two possibilities is the case,
> however, I still think that seeing an auto build break is a good idea,
> so regardless of that I still think this is progress.
I'll keep an eye on things in the following days, and report back to the
Laura Arjona Reina