gitlab CI configuration for the Debian website (Re: [Delayed] Summary of the web team BoF at DC18)
Hello Wouter, all
Sorry for taking so long to answer about this.
El 02/10/18 a las 17:31, Wouter Verhelst escribió:
> On Wed, Sep 26, 2018 at 07:36:03PM +0200, Wouter Verhelst wrote:
>> I have been working on improving the setup to split out a full build
>> into multiple jobs -- see
>> https://salsa.debian.org/wouter/webwml/pipelines/19296 for an example.
>> Since the timeout is per job and not per pipeline, that would work
>> around this problem. However, it doesn't seem to be working properly
>> yet; the "make install" step seems to be calling "wml" occasionally,
>> which means things take time, and we don't want that (because it would
>> make things time out). I'm not exactly sure whether that's my fault or
>> the build system's.
>
> I've now fixed this by making every individual language step do "make
> install" rather than just "make all", and by having the "pages" phase
> rsync each individual language directory into the main "public"
> directory, which should then do the final publication.
>
Thanks for all the work on this, I just merged it[1] into the main
webwml repo.
[1]: https://salsa.debian.org/webmaster-team/webwml/merge_requests/30
> It works, except that deploying the created output fails with an error
> message that the produced data is too large. I'm sure this is something
> which the salsa administrators could fix if we asked them nicely.
>
As we have talked in IRC during the tests, the current status is that
the CI jobs build the website but it is not published as "pages",
because of a size limitation (100MB) and (if I understood correctly) we
should review the documentation and try to use deployment environments:
https://docs.gitlab.com/ee/ci/environments.html
Any help is appreciated on this, and I will do my best to reply quicker
with tests and feedback.
Meanwhile:
* we can learn if commits break the builds (I'm not sure who gets
notifications (I get them, but not sure if because of my particular mix
of notification-levels and project-membership), you can probably tweak
your profile settings in Salsa if you don't get them and want to get
them), and we can also have a look at the CI results prior to merge some
merge request.
* People can browse/download the results of the CI builds in the Salsa
web interface, going to the webwml project, CI/CD, Jobs (or Pipelines).
Note that the artifacts corresponding to languages (other than English)
will exist (containing the /english folder) even if that language was
not built (if the language was built, the zip will contain the /english
and the /$language folders).
> At any rate, I plan to replace the code in my merge request with the
> code from this branch[1], it seems like a better option.
>
> There are still a few things that could be done to improve this,
> however:
>
> - The webwml Makefile is currently set up to ignore failures and
> continue. For a continuous integration system, this is suboptimal,
> although I understand that it makes sense for the main website. I
> would like to suggest that ignoring failures is not done if some
> environment variable is set; the CI jobs can then just set that
> variable and allow things to fail if something is broken
> - The jobs start off with running "apt-get install" for various
> packages. This works, but it is more efficient to create an image on
> the docker hub which already has those packages preinstalled. I'll
> look at doing so.
I will create bugs to track these two proposals.
> - I'm not sure what happens if changes are made to multiple languages in
> a single git push. There are two possibilities: either gitlab will
> merge pushed caches from a single stage so that the cached result will
> have all files from all translations (which is good, that's what we
> want), or pushing a cache will overwrite all other caches from the
> same phase (in which case the publish stage will just push the English
> version of the site together with and whichever translation happened
> to be finished last). I'll need to experiment and see if that needs
> fixing. Even if the latter of those two possibilities is the case,
> however, I still think that seeing an auto build break is a good idea,
> so regardless of that I still think this is progress.
+1
I'll keep an eye on things in the following days, and report back to the
mailing list.
Cheers
--
Laura Arjona Reina
https://wiki.debian.org/LauraArjona
Reply to: