Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

[Ross Vandegrift <rvandegrift@debian.org>]

On Mon, May 25, 2020 at 07:12:15PM +0200, Thomas Goirand wrote:
> On 5/25/20 5:43 PM, Ross Vandegrift wrote:
> > I don't think it's obvious how to do better.  The only ways I know to
> > make a raw image smaller than its fs are:
> >   1) sparse files
> >   2) compression
> > 
> > FAI is using #1, and you want to avoid #2.  Do you know another way?
> 
> Actually, I do. Shrink the FS once it's prepared, and leave as few space
> as possible (maybe, 20 GB), then resize the partitions. That way, the
> final HDD is as small as possible. That's what I was doing optionnally
> with openstack-debian-images, but I just don't know how this translates
> for FAI.

Oh I see- clever.  This could be a nice feature for FAI.  But it should
also be possible to post-process after FAI is done in the pipeline.

We'd either need to find the common requirements for all providers (eg AWS
requires 256MiB free), or limit the reduction to the generic images.

> > Is avoiding the extra download step more important than reducing the
> > image size? Your first mail raised both issues, and FWIW, I thought you
> > were mostly concerned about the size.
> 
> I'm always bad at explaining, and I need more words than normal people,
> sorry for this. Let me try again...
> 
> What's important is reducing the amount of time a user experience. If we
> provide a raw image file only in the form of a tarball, what's going to
> happen is that OpenStack users will have to:
> 1/ Download the image (locally?)
> 2/ Uncompress
> 3/ Upload to Glance
> 
> If that user doesn't have already a VM on the cloud, and is working
> remotely with a poor upload bandwidth (which is typical with DSL links),
> uploading to glance is going to take forever, and will be forced,
> because no other way around it (ie: the archive must be uncompressed
> before the uplaod).

This is helpful, thanks - so both matter because they both contribute to how
long the user must wait.  Though I'd avoid the DSL upload by using my cloud for
this process... :)

> > As a cloud user, I never want to care about point releases.
> > 
> > There's usually a way to identify the latest image of a given release.
> > For example, on AWS and GCP, the api can search for the latest debian 10
> > image.  Many deployment tools integrate this functionality, so I can
> > always deploy the latest debian 10 image.
> > 
> > I've never used OpenStack though, so I don't know if it has similar
> > features. 
> 
> It kind of does have the feature if the provider sets the correct
> properties on the images (like OS vendor and version, stored in the
> properties of images, which are key/value store), but it's unfortunately
> not a standard, and it may depend from one public cloud to another.
> 
> So I end up doing things like this:
> 
> $ openstack image list --format value -c Name | grep debian-10
> debian-10.0.1-20190708-openstack-amd64.qcow2
> debian-10.0.2-20190721-openstack-amd64.qcow2
> debian-10.0.3-20190815-openstack-amd64.qcow2
> debian-10.1.0-openstack-amd64.qcow2
> debian-10.1.2-20190925-openstack-amd64.qcow2
> debian-10.1.5-20191015-openstack-amd64.qcow2
> debian-10.1.6-20191114-openstack-amd64.qcow2
> debian-10.2.0-openstack-amd64.qcow2
> debian-10.3.0-openstack-amd64.qcow2
> debian-10.3.2-20200406-openstack-amd64.qcow2
> debian-10.4.0-openstack-amd64.qcow2
> 
> As you can see with this real use case example on a private cloud, I do
> download each point release, and often, intermediaries (when I'm aware
> of a grave security fix).

The order by point releases and by dates is the same.  So you can always find
the latest by picking the last - this will work without the point release, as
long as the date is always included (FAI always includes it).

That doesn't help with use-cases that require knowing which old image to use,
and it doesn't meet your preference.  But requiring everyone to track the point
releases would be too high of a cost IMO.

Ross