Re: liveid
Please ignore my comment in the previous email just sent about please
using spacing between your reply and the quoted text, I see you used
spacing perfectly fine here so I've no need to make you aware of
reading difficulties, you clearly have already recognised that adding
spacing is helpful. :)
Responses embedded below...
On Thu, 2020-04-16 at 02:17 +0200, adrian15sgd wrote:
> El 8/4/20 a las 19:12, jnqnfe@gmail.com escribió:
> > (@adrian, but including the debian-live mailing list for open
> > discussion)
> >
> > Hi,
> >
> > As I said in the other email just sent in reply to your comment on
> > Debian bug #956131, I wanted to keep discussion there focussed on
> > fixing the broken ability to create working images, and discussion
> > of a
> > better disc identification solution could be continued
> > separately...
> >
> > So firstly, your spec is written with the problem being specific to
> > Secure Boot situations. Does it not apply generically to EFI?
>
> My spec handles two boot problems:
>
> The first one is Secure Boot embedded grub.cfg finding its own root.
> As
> you say if Secure Boot EFI is not triggered you probably don't have
> this
> problem.
>
> However, later in the boot process the live cd initrd (which it's
> called
> live-boot) is going to search for something like
> /live/filesystem.squashfs . If two devices (usb, partitions, etc)
> have
> both of them that /live/filesystem.squashfs you still have a problem.
>
> Why? Because live-boot initrd will not match its associated
> /live/filesystem.squashfs.
>
> > From a cursory look at liveid and bug #924053, some thoughts occur
> > to
> > me...
> >
> > As to generation of a UUID, you discuss the concept of which build
> > attributes should go into its generation, but it occurs to me that
> > such
> > a focus does not take into account other differences in user
> > customisations - e.g. modifications made by hooks, includes,
> > package
> > lists, package-includes, apt config files, etc, etc. There are many
> > such differences that live outside of what is captured in LB_*
> > variables, that really should be taken into account in generation
> > of a
> > UUID to uniquely identify non-identical images. Perhaps a hash
> > build
> > from config directory contents would largely address things,
>
> That's a nice idea I guess. Something to be added to what it's
> already
> taken into account. Why? Because you can build live cds without a
> config
> directory being present.
You will not be able to run a build without having run `lb config` to
generate a config in the next version of live-build and it has never
been the intention that you could. The fact that you could (and I take
your word for it because I've not actually tried) happens no doubt to
be a consequence of a bug that dates all the way back to the version
1.0 days, which I just fixed in commit
ea0f6b7810e67cc5176b757db06dc3db8664c657 last month.
So moving forwards it is safe to expect that during a build at the very
least the full set of saved config files will exist, holding the config
attributes.
> > but 'auto'
> > would maybe also be needed, to cover any work done in there, but
> > then
> > what about stuff done by higher level user automation scripts
> > external
> > to live-build, e.g. Kali's creation script, could there be anything
> > done by them which needs capturing by UUID uniqueness (which would
> > be
> > unfeasible).
> Yeah, I mean, what I have implemented might not be perfect but it's
> way
> better than checking /live/vmlinuz or /.disk/info .
> > Perhaps a more simple solution of basing it upon a hash of the
> > squashfs
> > file would easily cover the uniqueness of each image? though we
> > want a
> > universal solution, not just one for iso images...
> Yeah, you could hash .img, .fat32 or whatever other livesquashfs
> extensions are used. Or squash every file in the directory (there's
> an
> option for generating a directory instead of an image if I'm not
> mistaken). But I don't like this idea too much. Having to hash such
> big
> files seems overkill to me.
> > You explored a solution of using an LB_ISO_VOLUME label based
> > solution,
> > but dropped it due to concerns about remastering tools. Firstly I'm
> > not
> > sure that from the live-build image creation side we need to really
> > care about remastering tools needing to know to adjust the volume
> > label
> > to make the remastered image unique. There will always be a need
> > for
> > them to adjust something to gain uniqueness, unless a mechanism of
> > searching within the same device can be achieved (see below).
>
> LB_ISO_VOLUME has only 13 characters size if I'm not mistaken. Too
> small
> for uniqueness.
I depends upon what your requirements are for uniqueness. We do not
need necessarily making a really long ID that is almost certainly never
going to clash with another image in existence, we just care about
reducing the likely hood significantly of a clash from almost certain
to low.
Though of course if we could find a solution that avoided the search
and had no issues with knowing which filesystem ws the correct one then
that would be best.
If we cannot do that, then consider for a moment if simply a timestamp
was used as the ID. A timestamp accurate to only seconds perhaps. What
is the likelihood that say Ubuntu and Debian each create a new image
for a new release at precisely the same second of the same minute of
the same hour of the same day...? Pretty unlikely. What is the
likelihood of Debian making an i386 image and an amd64 image at exactly
the same second? Quite unlikely? What is the likelihood that out of all
the people in the world building images, that two people make one at
the same second, maybe a little more likely, but the likelihood of
their image every sitting side by side in something, very unlikely.
What is the likelihood of a user image being made at exactly the same
second as an Ubuntu or Debian or whatever image? Pretty unlikely.
What I'm getting at is that a timestamp even only as accurate as down
to seconds is going to suffice to eliminate the vast majority of
identification clashes.
You've got to consider how many clashes a timestamp based ID is likely
to not catch, and consider how worth it it is to reduce that further
with a additional data of some degree. If you involve generation of an
ID by say hashing the entire config directory, and produce a 20-digit
ID (length randomly chosen here), but only catch a few more clashes,
that's really not worth it.
There's got to be a suitable balance found between effort put in to
generating an ID and the quality of the result (percentage of clashes
prevented). A simple timestamp based ID s extremely simple and likely
to prevent most clashes so is probably going to be good enough. If we
could add on a couple of simple attributes like architecture, distro,
whatever, then this can cheaply make things notably better, but the
issue is max string length for the filename, if stored in a
filename/label. Rather than adding to the string, we could instead feed
into a fixed length hash generator, giving just enough unique info to
feed the hash sufficiently, relying upon the hash bit length for our
unique ID space.
(If 13 chars is our limit, that's a 104 bit ID we're working on, or
2.02824096×10³¹ unique IDs, which seems enormously large for our
requirements of reducing clashes to a minimum).
Of course actually using a timestamp is problematic for reproducibility
reasons, and also there's the issue of it being saved into the config
and so not replaced, yet needing to be replicated across all build
artifacts. So there are those details to solve.
But a timestamp has the very neat property of being dynamic; it can be
automatically different for different builds even otherwise with the
same config. You don't need to examine a whole config to try to grasp
at finding some small difference to make two builds different in ID.
Note that generating two exactly identical images should be expected to
clash, though it does not matter since they're identical after all.
But still, the best solution of all is not having to do a flaky search
based mechanism, but instead to find some way of the EFI/secure-boot
component just simply knowing what partition is the right one per
having knowledge of what partition it itself was loaded from.
> > Also, LB_ISO_VOLUME by default contains a timestamp (it defaults to
> > `Debian ${LB_DISTRIBUTION} \$(date +%Y%m%d-%H:%M)`), but this is
> > overrideable by the user.
> Yeah, I override it myself when building Rescatux.
> > The value is determined at time of creation
> > of the config and is saved in the saved config. It thus gets re-
> > used
> > for each image created from that same config. This is a problem,
> > even
> > ignoring the user override aspect. Furthermore it is already a
> > problem
> > outside of grub disk identification, since if a user tweaks their
> > config, say to just to change the set of packages to be included
> > and
> > then rebuilds, then the label each time they subsequently build is
> > the
> > same as before, with the timestamp really reflecting date/time of
> > config creation rather than date/time of build, unlike what a user
> > might expect, and does not get changed upon running `lb config`
> > unless
> > the user explicitly overrides that option with --iso-volume. Of
> > course
> > fixing this to not store the timestamp in the config presents
> > reproducibility issues. Use of a build timestamp on its own
> > presents an
> > excellent source of uniqueness for image identification, if we can
> > just
> > solve the reproducibility problem.
>
> Yeah, that's why I guess I use some of the LB variables as a base
> because I think they did not change between builds.
>
> And, yes, I guess that reproducible-check-tool should override the
> --iso-volume switch.
>
> > The documentation in binary_grub-efi which implements the EFI
> > grub.cfg
> > that performs the search states that some platforms do this thing
> > where
> > $root points to the EFI partition. So what happens in other cases?
> > Why
> > do we need to do the search? If the EFI partition is on the same
> > device
> > as the real grub.cfg and kernel, etc, then why can we not limit the
> > search to partitions of the same device? Is grub not smart enough
> > to
> > allow us to do that? Would than not solve the problem in the most
> > ideal
> > way? If grub cannot currently do this then can it be modified to do
> > it
> > in future, which we can then rely upon? This seems like the most
> > obvious, direct and perfect solution.
>
> No program ever runs from the hard disk.
>
> Not even your GNU/Linux installed distribution.
I think you misunderstood me. We've created an image which contains
both the main filesystem we want loaded and the little stub EFI FAT
RAM-disk image. Whatever loads this latter thing obviously knows the
device/parition it got it from, and if it could simply pass this
knowledge on to the grub instance that gets run and does the search,
that grub instance would then be able to just load the main filesystem
from the device/partition it itself had come from! This hypothetically
would be a perfect solution by eliminating the flaky search.
> The reason why your GNU/Linux distro boots is because Grub gives the
> kernel (actually the initrd) a filesystem uuid where to find its
> root
> filesystem. So the kernel is not smart enough to figure out its own
> root
> filesystem.
>
>
> Same thing happens with this RAM-disk grub EFI image. It's loaded in
> RAM
> and you are alone. It reads its own ram-filesytem where it finds its
> stub grub.cfg .
As above, something loads the EFI RAM-disk image, if it could just pass
on the knowledge of what device/partition it loaded it from, the grub
instance could use that knowledge to perfectly jump to the right
filesystem. Hypothetically.
> Yeah, you could tweak that specific stub grub.cfg to have the same
> contents as the live cd grub.cfg but, then, if you add too many
> fancy
> stuff (like background images) it might mean that the UEFI firmware
> is
> not able to boot your EFI image into RAM (because of being too big).
>
>
> And, additionally, you would have the same repeated grub.cfg twice
> which
> would make things complex when modifying it.
I'd certainly not suggest that replacing the stub config with the full
one would be a good idea. And how would that ever work since it still
needs to know where the kernel files, etc, are located.
> > You discuss not being able to read the first line of a file into a
> > variable, but perhaps there's something in the ability to do
> > `source
> > <file>`, if say we wrote `set uuid=UUID` into the file rather than
> > just
> > `UUID`?
>
> Yeah, that's a nice idea. That could work. I haven't thought of that.
>
> However that modification would imply having to modify live-boot in
> order for it to be able to read "set uuid=UUID".
>
> And that does not make sense. I mean, yeah, you can parse it with
> some
> tools but... this "set uuid=UUID" would be in the EFI ramdisk and
> not
> accessible to live-boot.
I think you misunderstood the idea. the content of the file is already
grub syntax, and you'd simply import it into the grub config such that
effectively you've got a line setting a variable.
So in `/.disk/uuid` or whatever live-build would generate:
```
set uuid=ABC...
```
Then in the stub grub.cfg file live-build would generate something like
this:
```
set search=ABC...
for device in $devices; do
#imports `set uuid=UUID` line
source $devices/.disk/uuid
if [ "$uuid" = "$search" ]; then
#we found our device
fi
done
```
the main problem being getting a list of devices in the first place.
this is all just conceptual of course.
> I prefer my approach of /LIVEID/ABCD/EDCD/EFFE/1234 so that it's
> easier
> to handle with more simple tools.
Indeed this aspect is simple, and I do not expect the previous solution
would actually be feasible, but:
1. There's still the issue of how much effort is going into the
generation of the ID. This can be greatly reduced if relying upon a
timestamp as a key piece of info, and I strongly feel that doing so
needs to be explored.
2. There's still the issue of how long an ID you're bothering to
generate. As suggested earlier, if a single filename can only be 13
chars, that's a very nicely large ID space as it is. So can we not just
get away with using a single filename for the ID, stopping 99.99% of
clashes.
3. I have to be honest that although I can understand why you've
constructed it as you have, this directory structure just has too much
of an element of being ugly/messy to me to be sufficiently acceptable.
(Though this is just my opinion and I do not make the final decision of
what solution gets merged).
> > At least if this does not lend itself to finding the right
> > disc, it could lend itself to comparison of the UUID of the disc
> > that
> > is found by searching for a '.disk/info' or '.disk/uuid' file that
> > contains this, with the same ID embedded in the EFI grub.cfg doing
> > the
> > search, with it giving an error if the disc found does not match
> > the ID
> > searched for, prompting the user to fix the situation rather than
> > carry
> > on with loading the wrong disc?
>
> This is elegant but it's very complex. So you would have to loop on
> all
> the devices MANUALLY which it's not easy (Believe, I have implemented
> it
> in Super Grub2 Disk) and then try to loop-mount the efi-ramdisk and
> compare its set variable with your original one. Loop-umount it and
> try
> the next disk.
Ok, I take your word on it not being easy, it was only a concept on my
end.
> If I just use the grub search command with a path+file I'm done.
> Everything is done in the background by grub and it's only one line.
> The
> first match for the first file found is saved onto a variable and we
> are
> done.
>
> > My proposal:
> > 1. If we can have the search limited to partitions on the same
> > device,
> > and this works universally, then this is surely the perfect
> > solution
> > and we should use it.
>
> As I said above that's not true. The EFI ramdisk is loaded into the
> RAM
> by the UEFI firmware and grub reads what's inside. Using what's
> inside
> that filesystem grub needs to decide what it's is own root.
I understand that, but as clarified above, the EFI ramdisk originated
from the same image/device/partition as the filesystem it want to find,
and whatever loaded the EFI ramdisk know whether it got it from, so if
only it could pass on that knowledge to the EFI ramdisk when it loads
it, then it would have the info it needs to not need to do a search and
thus would be a perfect solution... (just a concept).
> > 2. live-build should be taught to replace a `@TIMESTAMP@` type
> > placeholder in LB_ISO_VOLUME, which should be used instead of
> > saving a
> > timestamp directly at time of config creation, such that the
> > timestamp
> > will change for each build at build time. For reproducibility
> > however a
> > --timestamp override option should be provided.
>
> I'm not sure basing the UUID on only LB variable instead of various
> LB
> variables is a good idea.
>
> Anyways I'm most interested in UUID being used than how it's
> generated
> in the first place.
It's not about one vs. multiple variables, it's about the timestamp
being a dynamic property that has huge weight in terms of uniqueness.
> > 3. If #1 is not possible, use either the timestamp in the same
> > way, or
> > the ISO volume label, with build-time timestamp (or override)
> > always
> > used, for the identifier. (Timestamp alone is great, using the full
> > ISO
> > volume label helps go a little further to what should be
> > sufficient).
>
> That's not possible. You cannot force the user to have a timestamp
> in
> its volume label. And the volume label is rather small for it (13
> characters).
Ok, as I say above, I think a 13 char length is massive enough an ID
space for our purposes, but of course we could not replace the disk
label with this since it needs to be readable, but we could put a
simple single file like `/.disk/XXXXXXXXXXXXX` on the disk though.
Are we talking 13 chars for an ISO volume label and HDD volume label,
or both BTW? the default LB_ISO_VOLUME is longer than 13 chars, whilst
LB_HDD_LABEL is not...
> I mean... I might agree on generating the UUID based on a timestamp
> but
> I won't agree on storing either the timestamp or the UUID on the ISO
> volume label.
A timestamp is already being written to the ISO volume label by default
anyway... Whether it's too long a string and it's getting truncated I'm
not actually certain, there are some string length warnings being
output from xorriso to investigate.
Anyway, one more thing I want to point out that is very interesting...
I recently grabbed a copy of an Ubuntu image to take a little peak at
what they were dong, since their images apparently did not seem to have
the same issue we do...
So the interesting question is how are they managing it? They do not
have a stub `grub.cfg` within `/grub/efi.img/efi/boot... They have
casper stuff which I know little about... Their loopback config is just
a copy of that of the actual `grub.cfg`. They have `/EFI/BOOT` when we
have `/EFI/boot` and `/EFI/debian`...
Regards,
Lyndon
Reply to: