Re: live-setup | Reintroduce regular building of live-build images (!2)

[Steve McIntyre <steve@einval.com>]

Hi Roland!

On Tue, Nov 22, 2022 at 08:40:05PM +0000, Roland Clobus (@rclobus-guest) wrote:
>Roland Clobus created a merge request: !2
>
>Project:Branches: rclobus-guest/live-setup:rclobus/reintroduce_live-build to
>images-team/live-setup:master
>Author: Roland Clobus
>Assignees:
>Reviewers:
>
>Build the bookworm images on a regular basis again.
>
>The script is based on old/run-30live-build and the script that currently runs
>in Jenkins

I've reviewed and cherry-picked your commit here into master, but I'm
afraid there's more work and refactoring needed here. Apologies if
you've been misled by the old/run-30live-build script - that's now
very outdated and doesn't fit the setup we have.

We *used* to start our build VM, then let run-30live-build do all of
its work inside the VM, then stop the VM. But (like some of the rest
of our scripts!) that didn't parallelise very well, and when I
heavily re-factored our build scripts back in 2020 I didn't update
this to match.

The new setup is driven by a top-level Makefile (Makefile.weekly) that
we call with -j<number>, and to make this work well things are
organised quite differently. We need the set of desktops listed in
that Makefile, and then using Make dependencies and targets:

 * cause the live-building VM to be started
 * start separate build processes for <arch>-live-<desktop>
 * when an architecture's worth of binary builds are all finished:
   * do the work needed for publishing
   * build the list of source files needed to match
 * when *all* the binary builds are done:
   * stop the VM#
   * use the source lists for all arches to build source tarballs

Now I can see what you're do in your script, I'm working on merging to
something more current.

I've just remembered one thing that used to cause major issues with
live-build: multiple image builds in parallel would sometimes trip
over each other. For us on release weekends, this is a critical
featire: we have a large build machine to make things go fast. I hope
this is now fixed in live-build, I guess we'll see!

Thanks for your work so far! I'm hoping to get something working in
the weekly image builds soon.

-- 
Steve McIntyre, Cambridge, UK.                                steve@einval.com
  Armed with "Valor": "Centurion" represents quality of Discipline,
  Honor, Integrity and Loyalty. Now you don't have to be a Caesar to
  concord the digital world while feeling safe and proud.