Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases
Hi,
Currently, our scripts are generating files named for example:
debian-11-genericcloud-amd64-daily-20200524-273.tar.xz
The problem with these, is that you can't use then directly with Glance.
One must first download them, extract the content, and then upload the
result. This is very annoying, when one can just point to the URL, and
the client does the rest, with a command like this:
openstack image create --copy-from IMAGE_URL
The other annoying thing is that the extracted image is then 2GB, even
though once mounted we see the image could be a lot smaller:
# df -h /mnt/
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/loop0p1 1.9G 466M 1.3G 27% /mnt
The bigger the image is, the longer it will take to copy, which is an
operation that OpenStack can do before spawning an instance. Images are
being copied in the compute nodes before the instance is started. Yes,
there's a caching system (and even a pre-caching system now), and yes,
typically we'd use Qcow2 in such case, and the .raw format is more for
when using Ceph as storage, and in such case, there's no need to copy
the image first. Still, it'd be nicer to have smaller .raw images.
So I was wondering if we could:
1/ Make the resulting extracted disk smaller. That'd be done in FAI, and
I have no idea how that would be done. Thomas, can you help, at least
giving some pointers on how we could fix this?
2/ Published the raw disk directly without compression (together with
its compressed form), so one can just point to it with Glance for
downloading. BTW, I don't see the point of having a tarball around the
compressed form, raw.xz is really enough, and would be nicer because
then one can pipe the output of xz directly to the OpenStack client (I
haven't checked, but I think that's maybe possible).
One thing though: if I understand well, artifacts are first stored on
Salsa, and currently, there's a size limit. What is the max size? If I'm
not mistaking, it's 1GB max, right? If that's the case, then maybe
that's a problem with the current 2GB decompressed disk.raw image. I btw
have an issue with the Octavia image here:
https://salsa.debian.org/cloud-team/debian-cloud-images/-/jobs/759369
Any suggestion on how to fix publishing? I can't even test it then... :(
Another thing which bothers me, is that in our current publication,
there's no way to tell what image is from which point release. At least,
with the older OpenStack images, there's the point release in the name.
There's also a "never changing" URL which is convenient. The format we
adopted with Steve was something like this:
debian-10.4.1-20200515-openstack-arm64.qcow2.index
10.4 is obviously borrowed from Buster, .1 shows it is the 2nd
intermediary release after 14.4 (ie: after a security update), and then
we have obviously the date. I'd like that we adopt something similar for
our Buster images, and I believe it was discussed durring the last
sprint, though I prefer checking with the list here.
How can we fix this?
Your thoughts?
Cheers,
Thomas Goirand (zigo)
Reply to: