Re: Debian based Filesystems with ELBE
Hi,
On 16:30 Mon 11 Jan , Neil Williams wrote:
> On Mon, 11 Jan 2016 16:17:12 +0100
> Manuel Traut <manut@mecka.net> wrote:
[..]
> > The main focus of ELBE is to reproduce the same target image no
> > matter on which Debian Host PC it is running. You just need to archive
> > a single ISO file.
>
> That should be manageable with any of the existing tools, although how
> to compare the target image for some measure of "equality" is
> non-trivial.
>
> Same package list, same configuration - can be done and
> should work for all tools of this kind. vmdebootstrap has a test suite
> which has the beginnings of this kind of check - fast checks during
> development, build checks for releases.
ELBE currently saves the checksum of each binary package used to build the
image. If the image is built again, these checkums are compared. So you
get a notification if a package changed on the mirror. If you use the
same initvm and elbe version you should be safe.
> > > > ELBE uses debootstrap, python-apt, qemu-user-static & chroot,
> > > > mtd-utils and python-parted to generate target images. Currently
> > > > amd64, armel, armhf and i386 are tested successfully.
> > >
> > > There was a session at DebConf15 which identified about 15 existing
> > > wrappers that do much the same thing already.
> >
> > Is the output of the session available? Which wrappers exist, what
> > features or intention do they have and where are they hosted?
>
> http://saimei.acc.umu.se/pub/debian-meetings/2015/debconf15/Creating_bootable_Debian_images.webm
> http://annex.debconf.org/debconf-share/debconf15/slides/creating-bootable-debian-images/#/
Thanks a lot. I just want to copy/paste the slide about the shortcommings
in the existing tools:
http://annex.debconf.org/debconf-share/debconf15/slides/creating-bootable-debian-images/#/25
|Shortcomings in the tools
|
| Hackable user setup: default passwords!
The root password is set in XML, additional users and groups can be added by
finetuning rules in XML.
Of course all images generated by the same XML are configured with the same
passwords.
| Use-case specific hardcoded customizations
We tried to put all use-case specific stuff to be abstracted in the XML format
and be interpreted by elbe.
| Hardcoded/restricted partitiong
Defining CPIO, squashfs, .tar.gz, GPT/msdos/MTD based images with various
filesystems is posible in XML.
| Missing first-boot customizations
Also missing in ELBE :)
| Unupgradeable kernel, bootloader
Kernel should be installed and upgradeable like any other Debian packages.
grub is also instalable as Debian package. We recommend to package u-boot
etc. as Debian package, too. But elbe doesn't provide any help on doing
that.
| Copying bits from the host system
It's not wanted to be done with elbe. The resulting image should be the
same, no matter on what host it is running. User-specific config files
can be added to a base64 encoded <archive> section in the XML file.
| Cool features only in some tools
| Example: automatically grow rootfs size to media size on first boot
That's really cool, but it's not in elbe :(
| Underdocumentation - how to add an new bootloader etc
Defintely a problem with elbe.
> > > > I'm quite sure, a lot of people wrote scripts for doing this.
> > >
> > > ... and packages already in Debian.
> >
> > We didn't find one, as we started with this in 2007. But of course
> > we would also like to have the elbe packages in Debian.
> >
> > > > To ensure that the same target image can be generated again, the
> > > > target image generation is done in a defined Debian installation,
> > > > typically running in KVM. We call this machine 'initvm'.
>
> So that's an extra layer outside - vmdebootstrap is used this way when
> building live images because that allows the build to happen without
> a real root user being needed. However, it seems like extra
> encumbrance to do that by default.
You can use 'elbe buildchroot' on your host PC. You need to be root
and it might be a little bit dangerous because the code does loopback
mounts etc. but the virtual-build machine is not a must.
> Personally, I much prefer "one tool one job" and not one monolithic
> lump of code. This leads to developing wrappers and helpers along side
> the main tool to cope with the otherwise endless permutations of
> options, configurations and use cases.
This is also our opinion. This is way elbe has a lot of subcommands doing
specific things. There is just one wrapper tool 'elbe initvm' that calls
the several subcommands.
> > I didn't know about vmdebootstrap and just had a quick look at the
> > source.
> >
> > Please correct me if i'm wrong.
> >
> > A lot of the code looks very familiar for me (elbe is also written in
> > python).
>
> I suspected as much, hence my original query about adding another
> version of the same core functionality.
>
> > It covers more or less the featureset of the 'elbe
> > buildchroot' command [0]. However I see limitations in vmdebootstrap
> > regarding supported Image formats, multiple mirrors, incremental runs
> > (just add a single package and regenarate the images). However on the
> > other hand a lot of the code is more or less the same (e.g.
> > grub-install, creating partitions).
>
> Image formats can be added. Multiple mirrors are not actually supported
> by debootstrap, so that comes down to a customisation script. (Or, as
> is currently being developed, a process of generating a "common"
> tarball of the base for lots of images, customising in parallel and
> preparing the final squashfs using a single tool which calls code from
> the vmdebootstrap module. Incremental runs are a source of horrible
> bugs. debootstrap itself doesn't handle incremental runs particularly
> well. Each run would need to be independently tested before it could be
> considered reliable. Adding packages to existing images isn't a job for
> the debootstrap wrapper - it could have a helper script to do a mount
> but we haven't seen a use-case for that yet.
We use incremental builds just to speed up image development.
For building releases a plain run should be done.
> vmdebootstrap has targz, squashfs, qcow2 support. GPT is supported if
> the tools underneath support it (--part-type option but this is
> missing from the manpage, so that's something to fix).
>
> So yes, image formats additions are welcome in vmdebootstrap. Wrappers
> and helpers to solve use cases for multiple mirrors or image
> manipulation after creation (not just adding packages) would be welcome
> too.
>
> > The big difference between 'elbe buildchroot' and 'vmdebootstrap' is
> > the user interface. vmdebootstrap works with a lot of parameters and
> > shell scripts. 'elbe buildchroot' passes the same information in a
> > single XML file.
>
> vmdebootstrap options can be handled via a config file as well. Using
> the log file options, the config is dumped as a config file for re-use
> later, if required. See also the --dump-config option.
>
> > I see the potential to merge the sourc of both projects. We could
> > integrate the additional functionalities from 'elbe buildchroot' into
> > vmdebootstrap and just call this from our code. Or vmdebootstrap can
> > be just another frontend to elbepack/elbeproject.py (that is
> > currently part of the elbe-common debian binary package).
>
> Yay! There are some features you've described that can be folded into
> vmdebootstrap, there are some features that should be wrappers around
> vmdebootstrap or helpers to do things unrelated to building the
> original image(s).
I'm quiet busy at the moment, but keep an eye on merging the tools.
> I'm keen to keep the vmdebootstrap utility itself relatively clean in
> terms of just building images. Other tasks like tweaking images and
> customising a base for lots of similar images needs to be done with
> helpers.
That's good.
> It sounds like your script is doing things outside of what
> vmdebootstrap itself should be doing, so calling vmdebootstrap code
> from your code and including whatever changes we need to match the
> functionality would seem appropriate. Please ensure that you also add
> unit tests for the new functionality so that support which is "just for
> you" is not going to get broken by later changes for someone else.
OK.
> > Or integrate the additional functionality of 'elbe buildchroot' into
> > vmdebootstrap source and generate a python-vmdebootstap module both
> > and perhaps some other projects can use.
>
> vmdebootstrap 1.0 has/is a python module, it's not packaged separately
> at this time - a new binary package (with fewer dependencies) is
> possible.
seen that in git :)
> vmdebootstrap has a mailing list for discussion of development ideas:
> vmdebootstrap-devel@lists.alioth.debian.org
> https://lists.alioth.debian.org/mailman/listinfo/vmdebootstrap-devel
I just subscribed to the ML.
Regards,
Manuel
Reply to: