smaller cloud images, raw cloud images
Hi all,
in may 2020 we started a discussion about raw cloud images without tar, and without compression....
https://lists.debian.org/debian-cloud/2020/05/msg00057.html
esp. mails referencing <20200527100152.ufxbs75r2nzl7uc7@shell.thinkmo.de>
I want to advance this topic, because I think it's very confusing for
our user that we have two directories providing different images for
Openstack. Our goal is to have one tool to make cloud images and
provide them in one directory.
IMO we can easily provide raw uncompressed Openstack images for our
users right now.
Can't we just run a script on petterson after each release (not on the
dailys) to untar the disk image, rename it (because it's always
disk.raw) and generate a sha512sums file? Yes, the raw will be of 2G
size, but this is much less than the 310 subdirectories in our daily
builds. We also can have the sha512sum of the raw in the json file and
check this.
But this should only be a first step.
Currently the images sizes are hardcoded in the python build scripts
and fai-diskimage needs to know the size before partitioning the
virtual disk and creating all files inside it.
To get smaller raw images for Openstack (and maybe others cloud type)
a better solution would be to create the filesystem for a cloud image
(using fai dirinstall), and after that copy the files into a raw image
that perfectly matches the size needed. growfs will then resize the
partition during the first boot.
Bastian already had some ideas in
https://lists.debian.org/debian-cloud/2020/05/msg00075.html
without using fai-diskimage.
We also had some irc communication about this:
--- Day changed Wed Jul 22 2020
13:44:49< Mrfai> zigo: I'm looking at how to make cloud images smaller using fai-diskimage. resize2fs does not seem to help much, because it does not shrink the FS as much as (theoretical)
possible.
13:45:19< Mrfai> Do you build the openstack images (using your build script) with the autimatic-resize option?
13:45:35< waldi> Mrfai: that will not really help, i'm currently dropping fai-diskimage and fix a lot of other problems at the same time
13:47:24< Mrfai> waldi: it would be nice to communicate this making big changes.
13:48:33< waldi> Mrfai: i think i did already
13:49:38< Mrfai> The cloud team should have the opportunities to discuss bigger changes. You;ve mentioned this once, but IMO we should discuss the pros can cons in our irc meeting.
13:51:30< waldi> see /usr/sbin/freqanl-inject-data
13:51:32< waldi> err
13:51:37< waldi> <20200527100152.ufxbs75r2nzl7uc7@shell.thinkmo.de>
13:58:51< waldi> and there are much more problems. stray mounts, like /run/udev, if files exist in the host system. which makes it almost impossible to cleanup files that should not be there. stray processes, which could be fixed by using pid namespaces in fai-diskimage
14:16:08< Mrfai> I know that fai-diskimage needs pid namespaces. I already have a patch for this. Still need to test it.
--- Day changed Mon Jul 27 2020
21:30:30< kuLa[m]> <waldi "and there are much more problems"> waldi do you have anything written down what you suggest we should do about it? I'd suggest opening a bug against cloud.d.o to keep track of it
Here's my list of things that are needed:
create directory tree using fai dirinstall (using unshare)
du -s
create raw image (dd) of size XXX
Handle MBR/EFI partition
create partition and fs with minimal size (shell hooks?)
copy directory tree
adjust fstab?
create grub.cfg
write MBR/EFI info??
Anything missing?
I like to discuss some technical details, for e.g. when to run our tests.
What are pros and cons of implementing this in fai-diskimage or in our
build script.
Can we start this discuss during our next IRC meeting?
--
regards Thomas
Reply to: