Bug#776317: Jessie RC1 amd64 mini image missing efi bootloader
Hi,
i understand that debian-cd mailing list is in charge
for discussion of amd64 mini.iso and its generating software:
http://anonscm.debian.org/cgit/d-i/debian-installer.git/tree/build/config/x86.cfg
http://anonscm.debian.org/cgit/d-i/debian-installer.git/tree/build/util/geniso_hybrid_plus_firmware_partition
For the sake of persistence i add my assessment to bug 776317,
although it somewhat exceeds the scope of a bug report.
-------------------------------------------------------------
The bug reporter's EFI firmware mistook the appended firmware
partion (an empty FAT filesystem) for the EFI System Partition.
The real EFI partition is located inside the ISO filesystem and
not advertised by MBR. So the ISO did not boot from USB stick
until the file tree from the EFI partition was copied to the
firmware partition.
Other EFIs might not be so tolerant and insist in MBR partition
type 0xef or in a GPT partition in order to find the first
piece of GRUB2 software.
The very dense partition layout of debian-7*amd64-netinst.iso
obviously serves the needs and quirks of the EFIs which were
encountered so far.
But it makes partition editors scream.
The firmware partition is intended for being used on a
hard-disk-like device, where the wish for partition editing is
natural, because the device is larger than the ISO image.
So how can mini.iso become friendly towards partition editors ?
I will discuss three layout alternatives:
- UEFI compliant Legacy MBR based
- UEFI compliant GPT based
- Rogue but well tested amd64-netinst with firmware partition
My conclusion is that UEFI compliant Legacy MBR is most
promising in respect to work effort and ease of use.
The production of this layout is essentially sketched in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=776317#45
and improved in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=776317#50
The following paragraph also shows how to achieve this with
xorriso to avoid the need for post-processing the ISO image.
-----------------------------------------------------------
UEFI 2.4, 5.2.1 "Legacy Master Boot Record (MBR)" and
5.2.2 "OS Types" allow to use a pure MBR partition table
and to mark the EFI system partition by partition type 0xef.
This proposal lets three MBR partitions emerge which cover
one ISO and two FAT filesystems. The EFI partition bears
the proper type 0xef.
It would ease enlarging of the firmware partition if it is
number 3 and the EFI partition becomes number 2.
It is desirable to have the ISO filesystem in a mountable
partition. Because the El Torito boot image efi.img is inside
the ISO and thus inside partition 1, this implies the need
for an extra copy of the FAT filesystem with file path
/efi/boot/bootx64.efi in it. (I will think about El Torito
outside of ISO. It is weird but not impossible.)
In this case the appended EFI FAT image should be aligned to
full MiB, in order to fit into the cylinder ideology of
partition editors.
The wasted space is not really usable on an MBR partitioned
device, because any further partition would begin at a
cylinder boundary anyway.
If the SYSLINUX MBR template file isohdpfx.bin is in reach
then one can omit the run of isohybrid and fdisk and
rather let xorriso do the work of partitioning.
This works already with xorriso-1.3.2 as of Debian Jessie
resp. Debian testing:
xorriso -as mkisofs \
-r -J \
-isohybrid-mbr ...maybe.in.$(TEMP_SYSLINUX).../isohdpfx.bin \
-c boot.cat \
-b isolinux.bin -no-emul-boot -boot-load-size 4 -boot-info-table \
-eltorito-alt-boot \
-e boot/grub/efi.img -no-emul-boot \
-append_partition 2 0xef $(TEMP_CD_TREE)/boot/grub/efi.img \
-append_partition 3 0x01 $(FIRMWARE_VOLUME_FILE) \
-partition_cyl_align all \
-o $(TEMP_MINIISO) $(TEMP_CD_TREE); \
The variable FIRMWARE_VOLUME_FILE needs to be set to the path
of the FAT image, which is currently created in
util/geniso_hybrid_plus_firmware_partition by mkfs.msdos .
The FAT image would have to be created before the xorriso
run and removed aftwerwards.
Resulting partition layout:
System area summary: MBR isohybrid cyl-align-all
ISO image size/512 : 45056
Partition offset : 0
MBR heads per cyl : 64
MBR secs per head : 32
MBR partition table: N Status Type Start Blocks
MBR partition : 1 0x80 0x17 0 45056
MBR partition : 2 0x00 0xef 45056 2048
MBR partition : 3 0x00 0x01 47104 12288
With some more waste of space by duplicate ISO and Joliet
directory trees, one can achieve an even more conventinal
layout where the first partition does not include the MBR:
xorriso -as mkisofs \
-partition_offset 16 \
...all options from previous example...
yields:
Partition offset : 16
...
MBR partition table: N Status Type Start Blocks
MBR partition : 1 0x80 0x17 64 44992
MBR partition : 2 0x00 0xef 45056 2048
MBR partition : 3 0x00 0x01 47104 12288
The ISO filesystem can still be mounted via /dev/sdb and via
/dev/sdb1, although both differ in block offset.
In this case the price of the duplicate file trees was paid
from the waste space of cylinder alignment. With less luck,
it could also be one more cylinder (= 1 MiB) of image size.
In general it depends on the number of files and the length
of their names.
-----------------------------------------------------------
With GPT it becomes more complicated.
UEFI 2.4, 5.2.3 "Protective MBR" and table 16 state that
a neat GPT partitioning has an MBR marking a single
partition of type 0xee which covers the whole device from
LBA 1 to the end.
So actually one cannot combine MBR and GPT partitioning but
rather must decide for one of both.
BIOS needs no MBR partition table for its first boot stage.
So if the later boot stages, which might need partitions,
understand GPT then there should be no problem with a
standards compliant GPT.
Question is whether all EFIs understand GPT. {:)
If not xorriso shall do the partitioning, then one would
have to try whether gdisk or other GPT editors can equip an
isohybrid image with GPT.
Probably it is necessary to clean the partition table slot
of MBR partition 1, which was created by the isohybrid
program or by xorriso's option -isohybrid-mbr:
dd if=/dev/zero bs=1 count=16 seek=446 conv=notrunc of=mini.iso
The GPT editor could then use the first 32 kB of the ISO and
the padding at its end (by default 300 kB) in order to establish
a GPT and a first partition which covers as much of the ISO
as possible.i
If xorriso shall do the partitioning:
The new option of xorriso -as mkisofs
-appended_part_as_gpt
marks the ISO and appended partitions by a compliant GPT rather
than MBR or a non-compliant MBR+GPT mix. The partition types
are used only to distinguish EFI system partitions from others.
xorriso-1.3.9 -as mkisofs \
-r -J \
-isohybrid-mbr ...maybe.in.$(TEMP_SYSLINUX).../isohdpfx.bin \
-c boot.cat \
-b isolinux.bin -no-emul-boot -boot-load-size 4 -boot-info-table \
-eltorito-alt-boot \
-e boot/grub/efi.img -no-emul-boot \
-append_partition 2 0xef $(TEMP_CD_TREE)/boot/grub/efi.img \
-append_partition 3 0x01 $(FIRMWARE_VOLUME_FILE) \
-appended_part_as_gpt \
-o $(TEMP_MINIISO) $(TEMP_CD_TREE); \
This yields:
System area summary: MBR isohybrid cyl-align-off GPT
ISO image size/512 : 45056
Partition offset : 0
MBR heads per cyl : 64
MBR secs per head : 32
MBR partition table: N Status Type Start Blocks
MBR partition : 1 0x00 0xee 1 58239
GPT : N Info
GPT disk GUID : 647e4984d224e14597e29c8920e7edc9
GPT entry array : 2 248 separated
GPT lba range : 64 58176 58239
GPT partition name : 1 4700610070003000
GPT partname local : 1 Gap0
GPT partition GUID : 1 647e4984d224e14597e69c8920e7edc9
GPT type GUID : 1 a2a0d0ebe5b9334487c068b6b72699c7
GPT partition flags: 1 0x1000000000000001
GPT partition name : 2 41007000700065006e006400650064003200
GPT partname local : 2 Appended2
GPT partition GUID : 2 647e4984d224e14597e79c8920e7edc9
GPT type GUID : 2 28732ac11ff8d211ba4b00a0c93ec93b
GPT partition flags: 2 0x0000000000000000
GPT start and size : 2 45056 832
GPT partition name : 3 41007000700065006e006400650064003300
GPT partname local : 3 Appended3
GPT partition GUID : 3 647e4984d224e14597e49c8920e7edc9
GPT type GUID : 3 a2a0d0ebe5b9334487c068b6b72699c7
GPT partition flags: 3 0x0000000000000000
GPT start and size : 3 45888 12288
Note that the ISO filesystem cannot be represented by a
mountable partition here. It begins at LBA 0, whereas a
protectice MBR partition must begin at LBA 1.
UEFI 2.4, table 16:
"StartingLBA [...] Set to 0x00000001"
GPT partitions may not begin before the First Usable LBA
which is 64 in xorriso ISOs (and cannot be smaller than 34,
according to UEFI 2.4, 5.3.1).
With -partition_offset 16 the first GPT partition is mountable
and automatically gets a name which advertises its filesystem.
(One could put the ISO Volume Id here. But i am unsure whether
this is appropriate.)
The MBR partition stays unmountable.
MBR partition table: N Status Type Start Blocks
MBR partition : 1 0x00 0xee 1 58239
...
GPT partition name : 1 490053004f003900360036003000
GPT partname local : 1 ISO9660
...
GPT start and size : 1 64 44992
-----------------------------------------------------------
What if we don't care about neat partition tables ?
libisofs-1.3.9 now puts the backup GPT after eventual appended
partitions and registers them as GPT partitions if a GPT emerges
by other settings.
So one can augment the layout of amd64-netinst by a firmware
partition:
xorriso-1.3.9 -as mkisofs \
-r -J \
-isohybrid-mbr ...maybe.in.$(TEMP_SYSLINUX).../isohdpfx.bin \
-c boot.cat \
-b isolinux.bin -no-emul-boot -boot-load-size 4 -boot-info-table \
-eltorito-alt-boot \
-e boot/grub/efi.img -no-emul-boot \
-isohybrid-gpt-basdat \
-append_partition 3 0x01 $(FIRMWARE_VOLUME_FILE) \
-o $(TEMP_MINIISO) $(TEMP_CD_TREE); \
Result is this crammed hybrid:
System area summary: MBR isohybrid cyl-align-on GPT
ISO image size/512 : 45056
Partition offset : 0
MBR heads per cyl : 64
MBR secs per head : 32
MBR partition table: N Status Type Start Blocks
MBR partition : 1 0x80 0x00 0 45056
MBR partition : 2 0x00 0xef 344 832
MBR partition : 3 0x00 0x01 45056 12288
MBR partition path : 2 /boot/grub/efi.img
GPT : N Info
GPT disk GUID : 9b618f4db2eb824eb10f15cd926e3c72
GPT entry array : 2 248 overlapping
GPT lba range : 64 57344 57407
GPT partition name : 1 490053004f00480079006200720069006400
GPT partname local : 1 ISOHybrid
GPT partition GUID : 1 9b618f4db2eb824eb10c15cd926e3c72
GPT type GUID : 1 a2a0d0ebe5b9334487c068b6b72699c7
GPT partition flags: 1 0x1000000000000001
GPT start and size : 1 0 45056
GPT partition name : 2 490053004f004800790062007200690064003100
GPT partname local : 2 ISOHybrid1
GPT partition GUID : 2 9b618f4db2eb824eb10d15cd926e3c72
GPT type GUID : 2 a2a0d0ebe5b9334487c068b6b72699c7
GPT partition flags: 2 0x1000000000000001
GPT start and size : 2 344 832
GPT partition path : 2 /boot/grub/efi.img
GPT partition name : 3 41007000700065006e006400650064003300
GPT partname local : 3 Appended3
GPT partition GUID : 3 9b618f4db2eb824eb10e15cd926e3c72
GPT type GUID : 3 a2a0d0ebe5b9334487c068b6b72699c7
GPT partition flags: 3 0x0000000000000000
GPT start and size : 3 45056 12288
List of sins:
This mixes the exclusive alternatives given by UEFI 2.4, 5.2:
"LBA 0 (i.e. the first logical block) of the hard disk contains
either a legacy Master Boot Record (MBR) (see Section 5.2.1)
or a protective MBR (see Section 5.2.2 [actually 5.2.3])."
It ignores the demand of UEFI 2.4, 5.2.2 for the legacy MBR:
"Each [MBR] partition must not overlap with other partitions."
and 5.3.1 for GPT:
"Each defined [GPT] partition must not overlap with any other
defined partition."
Further there is a law against start LBA 0 in GPT partition 1:
"All data stored on the volume must be stored between the First
Usable LBA through Last Usable LBA, and only the data structures
defined by UEFI to manage partitions may reside outside of the
usable space."
The First Usable LBA is 64, the Last Usable LBA is 57280 here.
The (non-)specification of ISOLINUX isohybrid GPT by
http://git.kernel.org/cgit/boot/syslinux/syslinux.git/tree/utils/isohybrid.c
defines the type GUID efi_system_partition but does
not use it. It rather marks the EFI boot image by
basic_partition, which is Microsoft Basic Data Partition.
Since i learned from isohybrid.c and mjg's Fedora-LiveCD.iso,
libisofs does it the same way in the isohybrid case.
The code for grub-mkrescue style preparations does it right:
GPT type GUID : 2 28732ac11ff8d211ba4b00a0c93ec93b
The code for isohybrid does not comply to UEFI 2.4, table 19:
GPT type GUID : 2 a2a0d0ebe5b9334487c068b6b72699c7
But is it worth to fix this ?
isohybrid GPT is so deep in the mud that only its successful
tests can justify it. And the tests succeeded with the bug.
After all, the xorrisofs option is -isohybrid-gpt-basdat
and not something like "-isohybrid-gpt-efisys".
End of sin list.
Option
-partition_offset 16
can be added for more conventional start addresses of
the first partitions:
Partition offset : 16
...
MBR partition table: N Status Type Start Blocks
MBR partition : 1 0x80 0x00 64 44992
MBR partition : 2 0x00 0xef 568 832
MBR partition : 3 0x00 0x01 45056 12288
...
GPT start and size : 1 64 44992
...
GPT start and size : 2 568 832
...
GPT start and size : 3 45056 12288
-----------------------------------------------------------
What do the above header-style lines mean ?
They stem from xorriso command -report_system_area "plain".
Parameter "help" prints the help text from libisofs:
$ xorriso-1.3.8 -report_system_area help
...
System area summary: word ... word
human readable interpretation of system area options and other info
The words are from the set:
{ MBR, CHRP, PReP, GPT, APM, MIPS-Big-Endian, MIPS-Little-Endian,
SUN-SPARC-Disk-Label, HP-PA-PALO,
protective-msdos-label, isohybrid, grub2-mbr,
cyl-align-{auto,on,off,all}, not-recognized, }
The acronyms indicate boot data for particular hardware/firmware.
protective-msdos-label is an MBR conformant to specs of GPT.
isohybrid is an MBR implementing ISOLINUX isohybrid functionality.
grub2-mbr is an MBR with GRUB2 64 bit address patching.
cyl-align-on indicates that the ISO image MBR partition ends at a
cylinder boundary. cyl-align-all means that more MBR partitions
exist and all end at a cylinder boundary.
not-recognized tells about unrecognized non-zero system area data.
ISO image size/512 : decimal
size of ISO image in block units of 512 bytes.
If an MBR is detected, with at least one partition entry of non-zero size,
then there may be:
Partition offset : decimal
if not 0 then a second ISO 9660 superblock was found to which MBR
partition 1 is pointing.
MBR heads per cyl : decimal
conversion factor between MBR C/H/S address and LBA. 0=inconsistent.
MBR secs per head : decimal
conversion factor between MBR C/H/S address and LBA. 0=inconsistent.
MBR partition table: N Status Type Start Blocks
headline for MBR partition table.
MBR partition : X hex hex decimal decimal
gives partition number, status byte, type byte, start block,
and number of blocks. 512 bytes per block.
MBR partition path : X path
the path of a file in the ISO image which begins at the partition
start block of partition X.
...
GUID Partition Table can coexist with MBR:
GPT : N Info
headline for GPT partition table. The fields are too wide for a
neat table. So they are listed with a partition number and a text.
...
GPT disk GUID : hex_digits
32 hex digits giving the byte string of the disk's GUID
GPT entry array : decimal decimal word
start block of partition entry array and number of entries. 512 bytes
per block. The word may be "separated" if partitions are disjoint,
"overlapping" if they are not. In future there may be "nested"
as special case where all overlapping partitions are superset and
subset, and "covering" as special case of disjoint partitions
covering the whole GPT block range for partitions.
GPT lba range : decimal decimal decimal
addresses of first payload block, last payload block, and of the
GPT backup header block. 512 bytes per block.
GPT partition name : X hex_digits
up to 144 hex digits giving the UTF-16LE name byte string of
partition X. Trailing 16 bit 0-characters are omitted.
GPT partname local : X text
the name of partition X converted to the local character set.
This line may be missing if the name cannot be converted, or is
empty.
GPT partition GUID : X hex_digits
32 hex digits giving the byte string of the GUID of partition X.
GPT type GUID : X hex_digits
32 hex digits giving the byte string of the type GUID of partition X.
GPT partition flags: X hex
64 flag bits of partition X in hex representation.
Known bit meanings are:
bit0 = "System Partition" Do not alter.
bit2 = Legacy BIOS bootable (MBR partition type 0x80)
bit60= read-only
GPT start and size : X decimal decimal
start block and number of blocks of partition X. 512 bytes per block.
GPT partition path : X path
the path of a file in the ISO image which begins at the partition
start block of partition X.
-----------------------------------------------------------
I am still testing the new features with GPT.
Commit to repositories and upload of a new snapshot
xorriso-1.3.9.tar.gz will happen in a few days.
Have a nice day :)
Thomas
Reply to: