Re: Debian based Filesystems with ELBE
On 21:40 Wed 13 Jan , Bill Gatliff wrote:
> Guys:
> If guestfish isn't extracting and installing debian packages properly,
> you've got one seriously messed-up system.
> Recall that Debian packages are just cpio archives, and in my case their
> extraction is actually done on the local machine. I use guestfish ONLY to
> build up the SD card image, or whatever the output media is, so that I
> don't have to muck around with loopback devices and so forth. It's
> repeatable all day, every day.
> I'm fully aware that project needs to be able to recover old images. But,
> Debian can do that: archive the repositories you used when you constructed
> your image, and pull from them next time.
ELBE does this automaticly by creating ISO images containing the debian source
and binary packages of the target image and the machine it was built on,
including the elbe tool itself and the XML needed for regenarting everything.
> You can even call out the precise versions in the target system's
> meta-package, if you want belt-and-suspenders assurance.
> What's more, you can also archive away the packages containing the
> compiler and other tools you used to construct the target system---and
> have assurance that they haven't changed in the meantime either, because
> they're cryptographically-signed.
> And their meta-packages also capture
> THEIR dependencies, so if you let the tool do its job it'll do far more
> than you'd ever want to do on your own.
Which tool do you mean?
> I've found Debian to be much more than adequate for (re)creating
> Debian-based embedded systems, even years after the original image was
> published.
Definitely, this is one of many reasons why we decided to use Debian in our
embedded projects.
> If you need certification-grade ability to recover and build old images,
> then archiving the entire project after you've tested it in an air-gapped
> environment is the only way to meet that requirement in most cases.
> Nothing else will be evidential-grade---and certification usually demands
> that level of assurance or it isn't "certification".
> If what you really need is a way to build up old images and assure you've
> gotten the same result that you got last time, Debian will do that for you
> already if you're just careful to archive the repositories that you used
> to construct the image, along with the repositories you used to construct
> the workstation that constructs those images. The latter isn't strictly
> necessary, since Debian packages aren't compiled by debbootstrap, they're
> just extracted; any tool that understands the archive format will produce
> the same result.
I think that's not true. E.g. for foreign architectures we use qemu-user*static
inside the chroot for the 2nd part of debootstrap. Another version of qemu
may lead to a crash in a post-install step and lead to a different image.
MTD utils or parted used from the host system are another part of the image
generation process that is not covered by archiving only the packages for
the target.
So i think it's always necessary to backup the workstation, that was used for
image generation. The method you choose depends on how important the regenartion
of the images is.
> If what you really need is a way to reconstruct, from source code, the
> contents of the packages that you used to construct your target image,
> then Debian has that covered too. In fact, the versions of the source
> packages are signed along with their binary packages, so they'll stay
> provably related. Add a manifest of package versions for the workstation,
> and the whole development environment is completely recoverable with OR
> without building any source code for the workstation itself.
Yes, we use this argument, if elbe is compared with cross-build systems like
buildroot or open-embedded.
> If what you're actually doing is using Debian packages to build up a
> non-Debian system, then it's no surprise that you've needed to invent a
> new tool to do that.
No, we want to build Debian system images. But we also want to have a way to
add our own 'debianized' applications. Therefore we just finished with the
pbuilder integration. That generates a pbuilder environment based on the Mirror
configuration of the XML file used to build the target. Next step is to add
the generated packages to a repository using reprepro or aptly and install
them into the images.
Another thing in elbe are the different copy modes. In default mode the
complete output of 'debootstrap' is in the image. In 'diet' mode, only
the packages from the pkg-list inside the xml file and their dependencies
are in the image. The risk is, that post-install scripts may fail, because
Debian policy says, all essential packages need to be on a system. What may
not be the case in this mode. In this mode we often need some 'finetuning'
rules and addtional config files from the XML embedded archive to get a
booting system. And it's not really a 'Debian system' with apt & co.
This is just recommended if you need to reduce the footprint of the image.
Another mode, we use sometimes is the 'tighten' mode. There we just extract
the content of the packages specified in the pkg-list. We don't care about
dependencies or post-install script. This is typically used in combination
with CPIO image type to generate customized initrds.
> I'm still skeptical that what you've ended up with
> is all that interesting to Debian, though. I'm still open to being
> convinced otherwise.
I'm currently looking how it is possible the share source with vmdebootstrap.
Either integrate the tool into elbe. Or wrapping out image generation into
a python module that can be used by both tools.
> Finally, the "our customers haven't ever used Debian before" isn't really
> an excuse, given how easy it is to create Debian packages.
Packaging the own application is typically no showstopper. It's is defintely
easy.
Manuel
> b.g.
> On Tue, Jan 12, 2016 at 4:33 AM Jeremiah Foster
> <[1]jeremiah.foster@pelagicore.com> wrote:
>
> On Tue, Jan 12, 2016 at 7:57 AM, Manuel Traut <[2]manut@mecka.net>
> wrote:
>
> Hi,
> On 16:03 Mon 11 Jan , Bill Gatliff wrote:
> >
> > If you are describing package dependencies, I have found that
> > meta-packages solve that problem neatly. And the result is
> compatible
> > out-of-the-box with the rest of Debian.
>
> that's true for people that are familiar with Debian and know howto
> generate
> a meta-package. But most of the people using ELBE have nether used
> Debian or
> even any Linux on an embedded system. And they need an easy to
> understand
> tooling for creating their firmware images including their
> application.
>
> This is crucially important.
>
> > If you are describing the process to translate a directory into a
> > filesystem image, shell scripts and guestfish have worked well
> for me.
> > And, they tend to mirror the instructions of a HOWTO, so they're
> > self-documenting.
>
> That's also true, but there is no guarantee that guestfish version x
> behaves
> like version y. Also you need to add your debootstrap-wrapper script
> and
> your image generation script to your version control system and
> ensure, that
> they are compatible.
>
> Also there is absolutely no guarantee that you can reproduce an image
> that
> behaves the same in 2 years. This is what we solved by generating the
> initvm
> and archiving all debs together with the image build.
> And this is a must have, if your image is used e.g. in a part of an
> industrial
> automation process or in a health-care device.
>
> Also true in the automotive industry. There you have a 12 year
> maintenance target that you have to keep along with the build system
> that built it. In the era of continuous integration this is harder than
> it sounds. Some companies find that even if they kept their release
> images they don't have the toolchain to build them because the toolchain
> has changed so much. Some companies literally lock a machine, with a
> build system installed, in a closet to ensure they can build their
> images later. While those methods are of course unsound, having an easy
> to use way to bind your image and tooling so you can recreate it later
> is valuable.
> I'd be very happy to test ELBE and vmdebootstrap as this work goes
> forward, especially if there are regular updates on this list for
> example.
> Regards,
> Jeremiah
>
> --
>
> Bill Gatliff
> Embedded Linux training and consulting
> [3]bgat@billgatliff.com
> (309) 453-3421
>
> References
>
> Visible links
> 1. mailto:jeremiah.foster@pelagicore.com
> 2. mailto:manut@mecka.net
> 3. mailto:bgat@billgatliff.com
Reply to: