Follow-up on last IRC meeting (ie: 12 Aug 2020)
Hi,
Sorry for not being online, my sister was there and it's my holidays, so
I decided it was nicer to be with her and enjoy life. Though I have the
IRC logs, and want to follow-up on the discussion.
1/ Releasing updated images
What Sledge did was comparing the security repository with the versions
of packages in the archive, so we could generate a new image on Casulana
(and later upload to Paterson) if there was a package with a newer
version. The script, if I'm not mistaking, is here:
https://salsa.debian.org/images-team/live-setup/blob/master/bin/check-openstack-updates
What IMO we need, is something that opens up the .json of the last
uploaded image to Paterson, and compares it to the security repo. Though
in here:
http://cdimage.debian.org/cdimage/cloud/buster/
I'm seeing images with dates, which seems to more or less match the
release of the "other" OpenStack images, so I wonder if that work is
already done or what...
Even if we do have such implementation already (which I cannot confirm),
it is my opinion that we still don't have feature parity. With Sledge,
we made sure that the current image always has the same URL, ie:
http://cdimage.debian.org/cdimage/openstack/current/
and that the image name includes the stable minor release number. I know
I am the only one in the team that thinks it is useful though, but I was
no convince by the only argument which basically was "I don't care, why
should I ...". With the name:
"debian-10.4.3-20200610-openstack-amd64.qcow2" we know we're talking
about the 3rd micro release of the image after the 4th minor release of
Buster, and it is my opinion that this is useful. For example, I can
tell very quickly that this is the last version with cloud-init 18.8. I
wouldn't be able to tell with only the date as hint...
I still don't understand why we would need to keep daily images on
Salsa. Aren't they useless if we have the feature above? Can't we then
just drop everything that is, let's say, one week old, as we'd otherwise
just be keeping duplicates of the same thing?
2/ Making images smaller
As I showed, it's possible to resize partitions very simply. However,
the FAI produced image is having a very weird layout. The partition 1 is
at the end, and then we have partition 14 and 15 before that. Why can't
we have something more simple, like:
1/ The EFI partition as partition 1
2/ The Ext4 partition as partition 2
Is there any reason it's the way it is right now? Why is the EFI
partition FIRST, but then labelled as partition 15? By the way, what is
the partition 14 for? It has a bios_grub flag, but it cannot be mounted.
What's the point?
In what I tried, which is insipred from the automatic resize of
openstack-debian-images, resize SEEM ok, but it destroys the partition.
Here's my script:
# Uncompress the disk.raw
tar -xvf debian-10-generic-amd64-daily-20200812-356.tar.xz
# Create the loop devices
TMPFILE_KPARTX=$(mktemp -t shrink-XXXXXX)
kpartx -av disk.raw >${TMPFILE_KPARTX}
echo "kpartx output:"
cat ${TMPFILE_KPARTX}
LOOP_DEVICE=$(cat ${TMPFILE_KPARTX} | head -n 1 | awk '{print $3}')
echo "Loop device is ${LOOP_DEVICE}"
rm ${TMPFILE_KPARTX}
# Mount the first partition, calculate it's space + 60M, then unmount
MOUNT_FOLDER=$(mktemp -d -t shrink-mount-folder-XXXXXX)
mount /dev/mapper/${LOOP_DEVICE} ${MOUNT_FOLDER}
USED_SPACE=$(df --block-size=M ${MOUNT_FOLDER} | tail -n1 | awk '{print
$3}' | sed s/M//)
WANTED_SPACE=$(( ${USED_SPACE} + 60))
umount ${MOUNT_FOLDER}
rmdir ${MOUNT_FOLDER}
# Resize the filesystem...
fsck.ext4 -f /dev/mapper/${LOOP_DEVICE}
resize2fs -f /dev/mapper/${LOOP_DEVICE} ${WANTED_SPACE}M
# See how many blocks the filesystem now takes, and calculate the
partition and image size
FINAL_FS_BLOCKS=$(tune2fs -l /dev/mapper/loop0p1 | awk '/Block
count/{print $3}')
FINAL_PART_SIZE=$(expr '(' $FINAL_FS_BLOCKS + 258 ')' '*' 4 '/' 1024)
FINAL_IMAGE_SIZE=$(( ${FINAL_PART_SIZE} + 128 ))
# Remove the loop devices
kpartx -d disk.raw
# Resize the partition
yes | parted ---pretend-input-tty disk.raw resizepart 1 ${FINAL_PART_SIZE}M
# Finally, truncate the final disk
truncate -s ${FINAL_IMAGE_SIZE}M disk.raw
After the parted + truncate, the partition 1 is broken, I don't
understand why, but that's probably due to this weird layout.
2/ Octavia images larger than 256 MB
I saw that the discussion was around publishing raw images. Since we
can't store the Octavia images on Salsa, could we imagine publishing
them on Paterson with the pipeline, without storing them on Salsa at
all? Does anyone have a suggestion?
3/ Raw images
If someone wants to implement HTTP compression (IMO a good idea), I can
easily confirm if that works with:
- Openstack Mitaka (very old, from 2016)
- OpenStack Rocky (ie: what's in Buster right now, released in 2018)
- OpenStack Stein (I could ask Michal Arbet, or deploy it to test)
- OpenStack Train (the release from last October)
- OpenStack Ussuri (the latest release, from last May)
I don't have access to any other deployments.
However, I still agree that this is probably a too early optimization
that we could delay for later.
4/ Image finder
My merge request is still pending here:
https://salsa.debian.org/cloud-team/image-finder/-/merge_requests/89
I still don't understand why my debian-cloud-image-finder-db-recreate
doesn't work. Doing an strace, I can see that the configuration file is
read, and that the connection to the DB is successfuly opened (this
worked with both Postgress and MySQL), though the DB isn't re-created.
This goes in the internals of the image-finder that are beyond me. It
feels like the calls to:
db.drop_all()
db.create_all()
db.session.commit()
have no effect, and I don't understand why. Arthur hasn't follow-up on
this. Does any of you has the time to look?
Cheers,
Thomas Goirand (zigo)
Reply to: