Proposal to change partition table layouts for EFI bootable ISOs
Hi,
this text is about the partition table layouts of EFI-ready bootable ISO 9660
images and the opportunity to unify and improve them.
Overview:
- Situation
- Flaws and remedy options
- Change proposals for debian-cd
- Change proposals for live-wrapper
- Examples of ISO partition layouts and El Torito
------------------------------------------------------------------------
Situation:
UEFI specifies that on hard-disk-like media a GPT or a MBR partition
table leads to the block range of a FAT filesystem with start-up programs.
The same block range is marked by the El Torito catalog for booting from
CD, DVD, or BD media.
There are mainly three partition table layouts of ISO 9660 images around
for this purpose:
- Fully specs compliant MBR partition table with partition of type 0xef
and a partition for mounting the ISO 9660 filesystem.
Produced by debian-cd for arm64 and ia64.
- Barely specs compliant MBR partition table with nested partitions
and invalid GPT. The outer partition is mountable as ISO 9660 filesystem,
the inner one is of type 0xef (and mountable as FAT).
Invented for program isohybrid option -u.
Produced by debian-cd for i386 and amd64.
- Fully specs compliant GPT announced by Protective MBR.
Produced by grub-mkrescue.
For tangible info about partitions and El Torito of existing Debian ISOs
see below "Examples".
------------------------------------------------------------------------
Flaws and remedy options:
GPT partitioning of an image file has the disadvantage that the GPT backup
at the end of the image will not sit at the end of the physical storage
device after the ISO was copied to it.
This causes complaints from Linux when assessing the partition table and
demands the expert operation of GPT backup moving before a partition editor
can create data partitions for the unised space on the storage device.
MBR partitioning with nested partitions is disallowed by UEFI. Only the
hack of setting the type of the outer partition to 0x00 prevents EFI
from recognizing the bad situation. Only by luck, Linux still recognizes
that partition as valid device.
Partition editors either complain or ignore the outer partiton.
The presence of unannounced GPT and backup GPT data makes them believe
that a clueless tool has mistaken a GPT Protective MBR for the place
where to add more partitions.
So really useful for USB sticks is only a fully specs compliant MBR
partition table. Intermediate congrats to arm64 and ia64.
But the Debian ISOs for arm64 and ia64 have two disadvantages towards
the ISOs for amd64:
- The ISO 9660 superblock cannot claim the blocks of the EFI partition
as part of its realm. That's because ISO 9660 partition 1 starts at
LBA 0 and does not include the EFI partition.
This causes program /sbin/isosize to report a size smaller than the
ISO image file.
This is bad for verifying a Debian ISO that is already copied to USB
stick or a DVD type which returns more blocks than were written.
xorriso-1.5.0 can prevent this if partition start is not LBA 0.
- The EFI partition image is contained twice in the ISO image. Once as
data file and referred to by El Torito. Once as appended partition of
type 0xef.
The reason is that prior to xorriso-1.4.6 it was not possible to let
El Torito point to a block range that is not a data file in the ISO.
xorriso-1.5.0 is now in the stable release. So it seems time to tackle
both problems:
- Quite old xorrisofs option
-partition_offset 16
moves the start of the ISO 9660 partition to 512-byte block 64 and thus
gives xorriso the option to claim the full image size in the ISO
superblock (PVD) at offset 0.
The prize is a second superblock at 2048-block offset 16 and a second
directory tree which will be used when the partition gets mounted.
This superblock claims only the partition's range as its realm.
xorriso-1.5.0 takes advantage of the situation and writes the full image
file size into the superblock at offset 0, which is in no partition.
/sbin/isosize reads that superblock and thus tells the full image size.
- Since xorriso-1.4.6 it is possible to let El Torito point to an
appended EFI partition by using
-e --interval:appended_partition_2:all::
instead of
-e boot/grub/efi.img
Therefore no copy of the EFI partition image is needed as data file
inside the image.
This reduction of storage space can compensate for the duplication of
superblock and directory tree in the proposal above.
------------------------------------------------------------------------
Change proposals for debian-cd:
So i propose to append for amd64 and i386 the EFI partition as it is
done for arm64 and ia64.
- Remove xorrisofs options
-isohybrid-gpt-basdat -isohybrid-apm-hfsplus
- Add xorrisofs option
-append_partition 2 0xef CD1/boot/grub/efi.img
The next two proposals apply to all four EFI bootable architectures.
- Add xorrisofs option
-partition_offset 16
- Move boot/grub/efi.img out of ./CD1 and append that file instead of
the above -append_partition gesture:
Then let El Torito point to the appended partition:
-append_partition 2 0xef ...somewhere.else.../efi.img
-e --interval:appended_partition_2:all::
(instead of: -e boot/grub/efi.img)
Plus some cosmetic change, possible since xorriso-1.4.8:
- Add option to set the type of partition 1 to 0x83 "Linux" (default is 0xcd)
-iso_mbr_part_type 0x83
------------------------------------------------------------------------
Change proposals for live-wrapper:
These are the native xorriso command equivalents of above proposals.
- Remove xorriso command
-boot_image isolinux partition_entry=gpt_basdat
- Add xorriso commands
-append_partition 2 0xef ...somewhere.outside.input.tree.../efi.img
-boot_image any partition_offset=16
- Change xorriso command (from "efi_path=boot/grub/efi.img")
-boot_image any efi_path=--interval:appended_partition_2:all::
The input tree in debian-live-9.8.0-amd64-xfce.iso was
/w/work/free/xfce/tmp/tmpBXqfAP . As with debian-cd, efi.img should not
be in there, because it is anyways in partition 2.
- Add xorriso command
-boot_image any iso_mbr_part_type=0x83
========================================================================
Examples of ISO partition layouts and El Torito:
This final part of my text is of interest only if there are questions what
i am writing about. The reader may skip it if not interested in the boot
sector and partition table details of existing and proposed ISOs.
In this case: Thank you for reading up to here.
The following reports were obtained by
xorriso -indev "$ISO" -report_system_area plain -report_el_torito plain
------------------------------------------------------------------------
Debian amd64 as is now:
------------------------------------------------------------------------
Volume id : 'Debian 9.3.0 amd64 n'
System area options: 0x00000102
System area summary: MBR isohybrid cyl-align-on GPT APM
ISO image size/512 : 593920
Partition offset : 0
MBR heads per cyl : 64
MBR secs per head : 32
MBR partition table: N Status Type Start Blocks
MBR partition : 1 0x80 0x00 0 593920
MBR partition : 2 0x00 0xef 3760 832
MBR partition path : 2 /boot/grub/efi.img
GPT : N Info
GPT disk GUID : 6215a6529f54bb43805f66f9583c139f
GPT entry array : 12 208 overlapping
GPT lba range : 64 593866 593919
GPT partition name : 1 490053004f00480079006200720069006400
GPT partname local : 1 ISOHybrid
GPT partition GUID : 1 6215a6529f54bb43805e66f9583c139f
GPT type GUID : 1 a2a0d0ebe5b9334487c068b6b72699c7
GPT partition flags: 1 0x1000000000000001
GPT start and size : 1 0 593864
GPT partition name : 2 490053004f004800790062007200690064003100
GPT partname local : 2 ISOHybrid1
GPT partition GUID : 2 6215a6529f54bb43805d66f9583c139f
GPT type GUID : 2 a2a0d0ebe5b9334487c068b6b72699c7
GPT partition flags: 2 0x1000000000000001
GPT start and size : 2 3760 832
GPT partition path : 2 /boot/grub/efi.img
APM : N Info
APM block size : 2048
APM gap fillers : 0
APM partition name : 1 EFI
APM partition type : 1 Apple_HFS
APM start and size : 1 940 208
APM partition path : 1 /boot/grub/efi.img
El Torito catalog : 939 1
El Torito cat path : /isolinux/boot.cat
El Torito images : N Pltf B Emul Ld_seg Hdpt Ldsiz LBA
El Torito boot img : 1 BIOS y none 0x0000 0x00 4 1148
El Torito boot img : 2 UEFI y none 0x0000 0x00 832 940
El Torito img path : 1 /isolinux/isolinux.bin
El Torito img opts : 1 boot-info-table isohybrid-suitable
El Torito img path : 2 /boot/grub/efi.img
------------------------------------------------------------------------
Note the absence of partition 0xee in MBR and the presence of 0xef.
So the GPT info is not properly announced.
The Apple Partition Map is useless anyways, because it points to no HFS+
filesystem.
------------------------------------------------------------------------
Debian arm64 as is now:
------------------------------------------------------------------------
Volume id : 'Debian 9.4.0 arm64 n'
System area options: 0x00000b00
System area summary: MBR cyl-align-all
ISO image size/512 : 411648
Partition offset : 0
MBR heads per cyl : 64
MBR secs per head : 32
MBR partition table: N Status Type Start Blocks
MBR partition : 1 0x00 0x83 0 411648
MBR partition : 2 0x00 0xef 411648 2048
El Torito catalog : 907 1
El Torito cat path : /boot.catalog
El Torito images : N Pltf B Emul Ld_seg Hdpt Ldsiz LBA
El Torito boot img : 1 UEFI y none 0x0000 0x00 832 908
El Torito img path : 1 /boot/grub/efi.img
------------------------------------------------------------------------
------------------------------------------------------------------------
grub-mkrescue with packages "grub-pc", "grub-efi-ia32-bin", and
"grub-efi-amd64-bin":
------------------------------------------------------------------------
Volume id : 'ISOIMAGE'
System area options: 0x00004201
System area summary: MBR protective-msdos-label grub2-mbr cyl-align-off GPT APM
ISO image size/512 : 34780
Partition offset : 0
MBR heads per cyl : 64
MBR secs per head : 32
MBR partition table: N Status Type Start Blocks
MBR partition : 1 0x00 0xee 1 34779
GPT : N Info
GPT disk GUID : faf27d34cdf44449a198b767638986bd
GPT entry array : 20 176 separated
GPT lba range : 64 34734 34779
GPT partition name : 1 4700610070003000
GPT partname local : 1 Gap0
GPT partition GUID : 1 faf27d34cdf44449a199b767638986bd
GPT type GUID : 1 a2a0d0ebe5b9334487c068b6b72699c7
GPT partition flags: 1 0x1000000000000001
GPT start and size : 1 64 276
GPT partition name : 2 450046004900200062006f006f007400200070006100720074006900740069006f006e00
GPT partname local : 2 EFI boot partition
GPT partition GUID : 2 faf27d34cdf44449a19ab767638986bd
GPT type GUID : 2 28732ac11ff8d211ba4b00a0c93ec93b
GPT partition flags: 2 0x1000000000000001
GPT start and size : 2 340 5760
GPT partition path : 2 /efi.img
GPT partition name : 3 48004600530050004c0055005300
GPT partname local : 3 HFSPLUS
GPT partition GUID : 3 faf27d34cdf44449a19bb767638986bd
GPT type GUID : 3 005346480000aa11aa1100306543ecac
GPT partition flags: 3 0x1000000000000001
GPT start and size : 3 6100 28032
GPT partition name : 4 4700610070003100
GPT partname local : 4 Gap1
GPT partition GUID : 4 faf27d34cdf44449a19cb767638986bd
GPT type GUID : 4 a2a0d0ebe5b9334487c068b6b72699c7
GPT partition flags: 4 0x1000000000000001
GPT start and size : 4 34132 600
APM : N Info
APM block size : 2048
APM gap fillers : 2
APM partition name : 1 Gap0
APM partition type : 1 ISO9660_data
APM start and size : 1 16 1509
APM partition name : 2 HFSPLUS_Hybrid
APM partition type : 2 Apple_HFS
APM start and size : 2 1525 7008
APM partition name : 3 Gap1
APM partition type : 3 ISO9660_data
APM start and size : 3 8533 162
El Torito catalog : 1672 1
El Torito cat path : /boot.catalog
El Torito images : N Pltf B Emul Ld_seg Hdpt Ldsiz LBA
El Torito boot img : 1 BIOS y none 0x0000 0x00 4 4047
El Torito boot img : 2 UEFI y none 0x0000 0x00 5760 85
El Torito img path : 1 /boot/grub/i386-pc/eltorito.img
El Torito img opts : 1 boot-info-table grub2-boot-info
El Torito img path : 2 /efi.img
------------------------------------------------------------------------
Here we have MBR partition 0xee announcing GPT.
The Apple Partition Map points to the HFS+ superblock, directory tree,
and the ISO's data file content blocks. The APM partition is probably
not mountable by Linux because of its block size 2048.
Thus the ISO is mountable on USB by Linux only as base device, like
/dev/sdc. Not from any /dev/sdcN.
Now for my proposals.
------------------------------------------------------------------------
Debian arm64 as proposed for arm64 and ia64:
------------------------------------------------------------------------
Volume id : 'Debian 9.4.0 arm64 n'
System area options: 0x00000a00
System area summary: MBR cyl-align-off
ISO image size/512 : 414228
Partition offset : 16
MBR heads per cyl : 64
MBR secs per head : 32
MBR partition table: N Status Type Start Blocks
MBR partition : 1 0x00 0x83 64 413332
MBR partition : 2 0x00 0xef 413396 832
El Torito catalog : 1808 1
El Torito cat path : /boot.catalog
El Torito images : N Pltf B Emul Ld_seg Hdpt Ldsiz LBA
El Torito boot img : 1 UEFI y none 0x0000 0x00 832 103349
El Torito img blks : 1 208
------------------------------------------------------------------------
------------------------------------------------------------------------
Debian amd64 as proposed for amd64 and i386 (with BIOS boot stuff):
------------------------------------------------------------------------
Volume id : 'Debian 9.3.0 amd64 n'
System area options: 0x00000102
System area summary: MBR isohybrid cyl-align-on
ISO image size/512 : 691008
Partition offset : 16
MBR heads per cyl : 64
MBR secs per head : 32
MBR partition table: N Status Type Start Blocks
MBR partition : 1 0x80 0x83 64 690112
MBR partition : 2 0x00 0xef 690176 832
El Torito catalog : 1872 1
El Torito cat path : /isolinux/boot.cat
El Torito images : N Pltf B Emul Ld_seg Hdpt Ldsiz LBA
El Torito boot img : 1 BIOS y none 0x0000 0x00 4 1873
El Torito boot img : 2 UEFI y none 0x0000 0x00 832 172544
El Torito img path : 1 /isolinux/isolinux.bin
El Torito img opts : 1 boot-info-table isohybrid-suitable
El Torito img blks : 2 208
------------------------------------------------------------------------
(My repacked ISOs are slightly larger than originals, because mounting
-t iso9660 regrettably does not show the hardlink relations in the tree.)
Have a nice day :)
Thomas
Reply to: