Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

[Bastian Blank <waldi@debian.org>]

On Mon, May 25, 2020 at 02:21:48AM +0200, Thomas Goirand wrote:
> >> So I was wondering if we could:
> >> 1/ Make the resulting extracted disk smaller. That'd be done in FAI, and
> >> I have no idea how that would be done. Thomas, can you help, at least
> >> giving some pointers on how we could fix this?
> > Fix what?
> The fact that the raw image is 2GB once extracted, when it could be
> 1/4th of that.

Please provide a prototype.

> >> 2/ Published the raw disk directly without compression (together with
> >> its compressed form), so one can just point to it with Glance for
> >> downloading. BTW, I don't see the point of having a tarball around the
> >> compressed form, raw.xz is really enough, and would be nicer because
> >> then one can pipe the output of xz directly to the OpenStack client (I
> >> haven't checked, but I think that's maybe possible).
> > No. Nothing in the download chain supports sparse files, so unwrapped
> > raw images are somewhat out of the question.
> I've done this for 3 Debian releases [2], I don't see why we would loose
> the feature because of a "sparse files" thing which you somehow find
> important. Truth is: nobody cares storing the raw image as sparse on an
> OpenStack cluster because:

Truth is: Nobody cares about OpenStack while we are talking about how to
store images on our own infrastructure.

Maybe ask Glance to process the images as needed?  Wait, it already is
able to do that.

The images have 700-800MB of space used, which is still three times the
size of the qcow2 file.  Why do you refuse to use what's already there?

> - the users that would download raw OpenStack images would be mainly
> those willing to store them with Ceph as a backend (where sparse files
> don't exist anyways, unless I'm mistaking).

Sorry, but this is bullshit.  Ceph RBD will try to store only the
minimal amount of data.  A RBD image is sliced into a lot of raw objects
and missing ones are considered holes.  It's even written in the
manpage:

| Create a new image and imports its data from path (use - for stdin). The
| import operation will try to create sparse rbd images if possible.
(https://docs.ceph.com/docs/master/man/8/rbd/)

> - Glance (the OpenStack VM image service), with files as backend,
> doesn't store sparse files at all in /var/lib/images either.

Maybe.

> >> One thing though: if I understand well, artifacts are first stored on
> >> Salsa, and currently, there's a size limit. What is the max size? If I'm
> >> not mistaking, it's 1GB max, right? If that's the case, then maybe
> >> that's a problem with the current 2GB decompressed disk.raw image.
> > It's 250MB.
> Then how are the ppc64el images generated? (they are bigger than this)

No, they are not.  It's just 180MB:

| $ xz -vk5T0 *.tar
| debian-sid-generic-ppc64el-official-20200525-274.tar: 184.0 MiB / 889.7 MiB = 0.207, 8.1 MiB/s, 1:49
(https://salsa.debian.org/cloud-admin-team/debian-cloud-images-daily/-/jobs/762303)

> Can't we just have something like this?!? How hard is it to understand
> how much more convenient this is? By the way, why are we keeping a
> history of 233 daily Bullseye images?

Ever heard of not implemented yet?

https://salsa.debian.org/cloud-team/debian-cloud-images/-/issues/29

Regards,
Bastian

-- 
You!  What PLANET is this!
		-- McCoy, "The City on the Edge of Forever", stardate 3134.0