Re: Comments on live-build, vmdebootstrap, bootstrap-vz, and live-wrapper
Hi Sam,
First, thanks for this interesting long post. I've read it all, and it
is a very good read.
I'm however surprised that in your review, you didn't take into account:
- openstack-debian-images (currently the only tool which is officially
building cloud image at http://cdimage.debian.org/cdimage/).
- diskimage-builder
I'm the author of openstack-debian-images. When I wrote it, my goal was
that it would be as minimal as possible, and as hackable as possible. I
had a look at many solutions, and was very much unhappy how much time it
would take to audit the code, and find ways to tweak. Therefore, I
thought that a simple shell script (currently only ~600 lines) was the
best option. Modules and plugins can only drive you so far, and always
increase the complexity. The current version is supposed to support both
Azure and OpenStack, probably it would also work for EC2 too (if someone
tested it). Over the time, we could add more clouds to the script (I'd
love if this happened), though the goal of this script is definitively
*not* modularity (but to be easy to hack by anyone). I will *never*
accept any patch in this script adding unnecessary complexity to the
code, and make it harder to read. Other tools are a better fit for that
kind of approach. I'm very much OK for this tool to continue being
debian-centric, and used mostly (or only?) by the Debian CD team.
On the other side, diskimage-builder does the complete opposite approach
and is really huge. Instead of 600 line of shell script, you end up with
15 000 lines of Python. But then, it has support for all of the
OpenStack options that the infrastructure team (the team from OpenStack
that maintains the CI/CD) needs, and it is very modular. It has the
concept of "elements", which you can write separately, and then add to
an image. I'd suggest reading through the doc here:
http://docs.openstack.org/developer/diskimage-builder/
As much as I can read from your message, bootstrap-vz and vmdebootstrap
are in the between of these 2 tools.
I'd very much love to read you about auditing diskimage-builder. FYI, I
maintain it in Debian (under the PKG OpenStack umbrella), but I haven't
update it in a while. If we decided to choose it, I'd be more actively
maintaining it.
Now, a bit of in-line response if you permit me.
On 08/16/2016 05:56 PM, Sam Hartman wrote:
> In this discussion I'm focusing on the needs of custom images. We've
> all agreed that a robust and flexible approach for custom images is an
> important goal of our cloud (and presumably live) efforts.
Robust, definitively. Flexible, I'm not so sure, it depends what you
mean behind this. For me, what's important is "hackability", I'm not
sure that's the same thing.
> live-wrapper
> bootstrap-vz
> Bootstrap-vz failed to meet my needs in two important ways. First, I
> found it too difficult to audit how an image I would create with
> bootstrap-vz would iffer from what I'd get debootstrapping something by
> hand. That is, I found that I could not trust the changes bootstrap-vz
> might introduce for me. I'd end up needing to read most of the code to
> evaluate this, because it seems like a major goal of the design is to
> make those sorts of tweaks easy.
I 100% agree with the above. Being able to hack into the too is super
important.
> Live Build
> Unfortunately, Live Build has some significant disadvantages. It's
> written in shell. It is a triumph of modular shell programming. I
> found that given time, I was able to understand an debug it: given how
> complex the shell scripts are, that says something very positive. I
> found that even when some of what it wanted to do wasn't right for my
> needs, code reuse was possible. Between the phase commands, function
> libraries and similar, lots of thought was put into modularity.
>
> However, triumph of modular shell programming though it is, it IS ALL
> SHELL. There's only so modular a shell script can be. There weren't
> really objects—and while yes, I know that you can do object-oriented
> shell, it doesn't make things better.
There's one huge advantage of shell scripting though: everyone knows it.
I'm specifically thinking about system administrator. Not all of them
will know Python.
> Live Build doesn't do a good job at things that aren't live systems.
> You can do some things by selecting plain as the root filesystem, and
> hdd as the image type, but I seem to recall that doesn't end up
> working. I ended up using only part of the typical chain (lb
> bootstrap&&lb chroot rather than lb build, then writing my own short
> script to make the image).
By discussing with Daniel, I'm sure it's possible to do things with "lb
build". It's also worse nothing that Ubuntu was (or still is?) using
live-build for its official images.
> Do EC2 eimages really need to be built in an EBS volume? Would it be
> good enough to copy them there after, building in a loop file like
> everything else until that moment?
Well, it's been about a year I'm asking that someone tests the openstack
images on EC2:
http://cdimage.debian.org/cdimage/openstack/
IMO, they should "just work" out of the box. The jessie one though
should only work on HVM though.
Could someone really take the time to test?
Cheers,
Thomas Goirand (zigo)
Reply to: