[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Comments on live-build, vmdebootstrap, bootstrap-vz, and live-wrapper

On Fri, 19 Aug 2016 09:02:30 +0000
Riku Voipio <riku.voipio@iki.fi> wrote:

> On Tue, Aug 16, 2016 at 11:56:12AM -0400, Sam Hartman wrote:
> > Hi.  
> > I recently finished putting together a custom image of what amounts
> > to stretch for work.  Today, this is mostly not a cloud image,
> > although that's expected to change for our future customer
> > deployments.  
> Thanks for your detailed report!
> >                               bootstrap-vz  
> > Secondly, with some irony, I realized that it would be annoying for
> > me to add my own customizations to the image.  Bootstrap-vz makes
> > it easy for bootstrap-vz developers to add new tweaks, but makes it
> > challenging for custom image builders who don't want to hack on the
> > bootstrap-vz sources.  I really want something like the
> > vmdebootstrap customization script where I can run a chunk of my
> > shell or Python code  
> Ideally the codebase should be inviting enough that it should make you
> *want* to bring your improvements as part of the upstream codebase,
> or plugins that adapt neatly to it. Alas that doesn't seem to have
> been the case for you and bootstrap-vz. When building images, I tend
> to start with creating the image by hand - in shell. So shell script
> comes down naturally as what one would use for building images.
> However, as you noticed with live-build later, unstructured shell
> scripts have their limits.
> sidenote: This touches the subject of how the four freedoms of
> software as in DFSG remain academic if you can't exercise them in
> practice due to bad usability and too hard to understand code
> structures. As detailed more articulately by Matthew Garret at
> https://mjg59.dreamwidth.org/32686.html
> > A Suggestion
> > 
> > As a user, who would love to see Debian be great, but who has no
> > political skin in any previous battles, I'd like to offer a
> > suggestion for something to explore.
> > 
> > I'd like to see some folks explore bootstrap-vz layered on top of
> > vmdebootstrap.  Debootstrap would be responsible for taking Debian
> > packages and unpacking and doing initial configuration, just as it
> > always does.  vmdebootstrap would be responsible for partitioning,
> > filesystem creation, bootloader installation and running
> > debootstrap. Bootstrap-vz would be responsible for tweaking the
> > image, for providing a framework for those tweaks, and for the
> > schema/manifest language to describe what it is that we want in an
> > image.  
> I'm not a big fan of much layering. Both bootstrap-vz and
> vmdebootstrap (and most other tools too) implement partitioning as
> parted calls. A script in between doesn't add much value but
> obfuscates what parted call have been done. 
> Installing bootloader is more complicated, but boils down often to
> running grub-install (or doing something similar by hand) with
> right mounts in place. 

That's the key point for me - grub sorts itself out *inside* the build
environment. It is critical for reproducibility that the tools used
to build the image come from inside the image. We need to be able to
build the same image on stable as on testing. The only tools scripts
like vmdebootstrap should need to use from the host system are those to
create the image file, partitions and filesystems. Grub meets this
requirement well, UBoot can too (within some limits), other bootloaders
commonly fail or have stupid assumptions about special handling of
partitions and/or filesystems. Bootloaders do not *need* to be special
in terms of installation into an image and it's way beyond time for
this to be fixed properly - within the packages which provide those
bootloaders in the archive *NOT the build tools*!.

qemu-img|dd, a bin-format handler, parted, debootstrap & the mount.*,
mkfs.* pairs are all that should be allowed to write to the image from
the host system. *Everything* else can and should be done only under

The goal should be that installing a package and possibly calling a
tool within that package must be all that is required to fully install
a working bootloader into an image, just like any other package.

> > We'd probably have to give up some of the tweaks we have, and add
> > support either for plugins for some of the more basic tweaks
> > directly into vmdebootstrap.  As an example, vmdebootstrap would
> > almost certainly need to support raw images without a partition
> > table. 

I don't see what benefit that provides.

> However, I think an clear interface between vmdebootstrap and
> > bootstrap-vz would make it easier to audit what was being done to an
> > image and might improve both products.  
> I think we should look into pushing customizations and tweaks into
> packages. Say ec2-defaults, virtualbox-defaults etc. The key benefit
> being that everyone gets same settings regardless of what tool they
> used to create the image, and everyone gets new better settings with
> apt-get upgrade ( rather than via switching to a new image).

Exactly, push the special snowflake knowledge into the packages
to be installed inside the image, not into the build tool.

This is why I'm unsure about the whole plugin request - if the build
tool needs special knowledge to handle your special snowflake device,
it is *your device* which is broken. Push that knowledge into the
packages to be installed within the image - a bit like flash-kernel
does, if you have to - and then the build tool can do the work. Else,
just fix your broken snowflake. If it can't be fixed, ditch the
hardware - I don't see why it should be supported. I'm far beyond the
point of adding yet more special casing for broken hardware, I stopped
caring about such nonsense a long time ago.

> > I think we should do something.  If there's one thing I hope you
> > take away from this message, it's that none of the options I
> > explored is ready—probably not even for Stretch.  Will we have a
> > regression in our custom image functionality Stretch over Jessie?
> > Will we have a solution in time for Stretch?  I don't know the
> > answers to either of those, but I hope that we can come together in
> > a spirit of cooperation and work on those questions rather than
> > focusing on parallel, independent paths.  
> Wise words.
> Riku


Neil Williams

Attachment: pgp00ttmJ48lC.pgp
Description: OpenPGP digital signature

Reply to: