Re: Proposal to change partition table layouts for EFI bootable ISOs
Hi Thomas!
On Sat, Jul 13, 2019 at 06:21:30PM +0200, Thomas Schmitt wrote:
>
>this text is about the partition table layouts of EFI-ready bootable ISO 9660
>images and the opportunity to unify and improve them.
>
>Overview:
>- Situation
>- Flaws and remedy options
>- Change proposals for debian-cd
>- Change proposals for live-wrapper
>- Examples of ISO partition layouts and El Torito
Thanks for your thorough work here!
>------------------------------------------------------------------------
>Situation:
>
>UEFI specifies that on hard-disk-like media a GPT or a MBR partition
>table leads to the block range of a FAT filesystem with start-up programs.
>The same block range is marked by the El Torito catalog for booting from
>CD, DVD, or BD media.
>
>There are mainly three partition table layouts of ISO 9660 images around
>for this purpose:
>
>- Fully specs compliant MBR partition table with partition of type 0xef
> and a partition for mounting the ISO 9660 filesystem.
> Produced by debian-cd for arm64 and ia64.
>
>- Barely specs compliant MBR partition table with nested partitions
> and invalid GPT. The outer partition is mountable as ISO 9660 filesystem,
> the inner one is of type 0xef (and mountable as FAT).
> Invented for program isohybrid option -u.
> Produced by debian-cd for i386 and amd64.
>
>- Fully specs compliant GPT announced by Protective MBR.
> Produced by grub-mkrescue.
>
>For tangible info about partitions and El Torito of existing Debian ISOs
>see below "Examples".
>
>------------------------------------------------------------------------
>Flaws and remedy options:
>
>GPT partitioning of an image file has the disadvantage that the GPT backup
>at the end of the image will not sit at the end of the physical storage
>device after the ISO was copied to it.
>This causes complaints from Linux when assessing the partition table and
>demands the expert operation of GPT backup moving before a partition editor
>can create data partitions for the unised space on the storage device.
Right.
>MBR partitioning with nested partitions is disallowed by UEFI. Only the
>hack of setting the type of the outer partition to 0x00 prevents EFI
>from recognizing the bad situation. Only by luck, Linux still recognizes
>that partition as valid device.
>Partition editors either complain or ignore the outer partiton.
>The presence of unannounced GPT and backup GPT data makes them believe
>that a clueless tool has mistaken a GPT Protective MBR for the place
>where to add more partitions.
ACK.
>So really useful for USB sticks is only a fully specs compliant MBR
>partition table. Intermediate congrats to arm64 and ia64.
>
>But the Debian ISOs for arm64 and ia64 have two disadvantages towards
>the ISOs for amd64:
>
>- The ISO 9660 superblock cannot claim the blocks of the EFI partition
> as part of its realm. That's because ISO 9660 partition 1 starts at
> LBA 0 and does not include the EFI partition.
> This causes program /sbin/isosize to report a size smaller than the
> ISO image file.
> This is bad for verifying a Debian ISO that is already copied to USB
> stick or a DVD type which returns more blocks than were written.
> xorriso-1.5.0 can prevent this if partition start is not LBA 0.
>
>- The EFI partition image is contained twice in the ISO image. Once as
> data file and referred to by El Torito. Once as appended partition of
> type 0xef.
> The reason is that prior to xorriso-1.4.6 it was not possible to let
> El Torito point to a block range that is not a data file in the ISO.
>
>xorriso-1.5.0 is now in the stable release. So it seems time to tackle
>both problems:
Well, ish. It might be in Buster (Debian 10.x), but our build machine
casulana is still using Stretch (Debian 9.x) for now - it hasn't yet
been upgraded. Once that's happened I'll look through your proposed
changes and do some testing. For now we're using xorriso 1.4.6.
>- Quite old xorrisofs option
> -partition_offset 16
> moves the start of the ISO 9660 partition to 512-byte block 64 and thus
> gives xorriso the option to claim the full image size in the ISO
> superblock (PVD) at offset 0.
> The prize is a second superblock at 2048-block offset 16 and a second
> directory tree which will be used when the partition gets mounted.
> This superblock claims only the partition's range as its realm.
>
> xorriso-1.5.0 takes advantage of the situation and writes the full image
> file size into the superblock at offset 0, which is in no partition.
> /sbin/isosize reads that superblock and thus tells the full image size.
>
>- Since xorriso-1.4.6 it is possible to let El Torito point to an
> appended EFI partition by using
> -e --interval:appended_partition_2:all::
> instead of
> -e boot/grub/efi.img
> Therefore no copy of the EFI partition image is needed as data file
> inside the image.
> This reduction of storage space can compensate for the duplication of
> superblock and directory tree in the proposal above.
OK, cool.
>------------------------------------------------------------------------
>Change proposals for debian-cd:
>
>So i propose to append for amd64 and i386 the EFI partition as it is
>done for arm64 and ia64.
>
>- Remove xorrisofs options
>
> -isohybrid-gpt-basdat -isohybrid-apm-hfsplus
>
>- Add xorrisofs option
>
> -append_partition 2 0xef CD1/boot/grub/efi.img
>
>The next two proposals apply to all four EFI bootable architectures.
>
>- Add xorrisofs option
>
> -partition_offset 16
>
>- Move boot/grub/efi.img out of ./CD1 and append that file instead of
> the above -append_partition gesture:
> Then let El Torito point to the appended partition:
>
> -append_partition 2 0xef ...somewhere.else.../efi.img
>
> -e --interval:appended_partition_2:all::
>
> (instead of: -e boot/grub/efi.img)
>
>Plus some cosmetic change, possible since xorriso-1.4.8:
>
>- Add option to set the type of partition 1 to 0x83 "Linux" (default is 0xcd)
>
> -iso_mbr_part_type 0x83
>
>------------------------------------------------------------------------
>Change proposals for live-wrapper:
>
>These are the native xorriso command equivalents of above proposals.
>
>- Remove xorriso command
>
> -boot_image isolinux partition_entry=gpt_basdat
>
>- Add xorriso commands
>
> -append_partition 2 0xef ...somewhere.outside.input.tree.../efi.img
>
> -boot_image any partition_offset=16
>
>- Change xorriso command (from "efi_path=boot/grub/efi.img")
>
> -boot_image any efi_path=--interval:appended_partition_2:all::
>
>The input tree in debian-live-9.8.0-amd64-xfce.iso was
>/w/work/free/xfce/tmp/tmpBXqfAP . As with debian-cd, efi.img should not
>be in there, because it is anyways in partition 2.
>
>- Add xorriso command
>
> -boot_image any iso_mbr_part_type=0x83
>
>========================================================================
>Examples of ISO partition layouts and El Torito:
>
>This final part of my text is of interest only if there are questions what
>i am writing about. The reader may skip it if not interested in the boot
>sector and partition table details of existing and proposed ISOs.
>In this case: Thank you for reading up to here.
>
>
>The following reports were obtained by
>
> xorriso -indev "$ISO" -report_system_area plain -report_el_torito plain
>
>------------------------------------------------------------------------
>Debian amd64 as is now:
>------------------------------------------------------------------------
>Volume id : 'Debian 9.3.0 amd64 n'
>System area options: 0x00000102
>System area summary: MBR isohybrid cyl-align-on GPT APM
>ISO image size/512 : 593920
>Partition offset : 0
>MBR heads per cyl : 64
>MBR secs per head : 32
>MBR partition table: N Status Type Start Blocks
>MBR partition : 1 0x80 0x00 0 593920
>MBR partition : 2 0x00 0xef 3760 832
>MBR partition path : 2 /boot/grub/efi.img
>GPT : N Info
>GPT disk GUID : 6215a6529f54bb43805f66f9583c139f
>GPT entry array : 12 208 overlapping
>GPT lba range : 64 593866 593919
>GPT partition name : 1 490053004f00480079006200720069006400
>GPT partname local : 1 ISOHybrid
>GPT partition GUID : 1 6215a6529f54bb43805e66f9583c139f
>GPT type GUID : 1 a2a0d0ebe5b9334487c068b6b72699c7
>GPT partition flags: 1 0x1000000000000001
>GPT start and size : 1 0 593864
>GPT partition name : 2 490053004f004800790062007200690064003100
>GPT partname local : 2 ISOHybrid1
>GPT partition GUID : 2 6215a6529f54bb43805d66f9583c139f
>GPT type GUID : 2 a2a0d0ebe5b9334487c068b6b72699c7
>GPT partition flags: 2 0x1000000000000001
>GPT start and size : 2 3760 832
>GPT partition path : 2 /boot/grub/efi.img
>APM : N Info
>APM block size : 2048
>APM gap fillers : 0
>APM partition name : 1 EFI
>APM partition type : 1 Apple_HFS
>APM start and size : 1 940 208
>APM partition path : 1 /boot/grub/efi.img
>El Torito catalog : 939 1
>El Torito cat path : /isolinux/boot.cat
>El Torito images : N Pltf B Emul Ld_seg Hdpt Ldsiz LBA
>El Torito boot img : 1 BIOS y none 0x0000 0x00 4 1148
>El Torito boot img : 2 UEFI y none 0x0000 0x00 832 940
>El Torito img path : 1 /isolinux/isolinux.bin
>El Torito img opts : 1 boot-info-table isohybrid-suitable
>El Torito img path : 2 /boot/grub/efi.img
>------------------------------------------------------------------------
>Note the absence of partition 0xee in MBR and the presence of 0xef.
>So the GPT info is not properly announced.
>The Apple Partition Map is useless anyways, because it points to no HFS+
>filesystem.
>
>------------------------------------------------------------------------
>Debian arm64 as is now:
>------------------------------------------------------------------------
>Volume id : 'Debian 9.4.0 arm64 n'
>System area options: 0x00000b00
>System area summary: MBR cyl-align-all
>ISO image size/512 : 411648
>Partition offset : 0
>MBR heads per cyl : 64
>MBR secs per head : 32
>MBR partition table: N Status Type Start Blocks
>MBR partition : 1 0x00 0x83 0 411648
>MBR partition : 2 0x00 0xef 411648 2048
>El Torito catalog : 907 1
>El Torito cat path : /boot.catalog
>El Torito images : N Pltf B Emul Ld_seg Hdpt Ldsiz LBA
>El Torito boot img : 1 UEFI y none 0x0000 0x00 832 908
>El Torito img path : 1 /boot/grub/efi.img
>------------------------------------------------------------------------
>
>------------------------------------------------------------------------
>grub-mkrescue with packages "grub-pc", "grub-efi-ia32-bin", and
>"grub-efi-amd64-bin":
>------------------------------------------------------------------------
>Volume id : 'ISOIMAGE'
>System area options: 0x00004201
>System area summary: MBR protective-msdos-label grub2-mbr cyl-align-off GPT APM
>ISO image size/512 : 34780
>Partition offset : 0
>MBR heads per cyl : 64
>MBR secs per head : 32
>MBR partition table: N Status Type Start Blocks
>MBR partition : 1 0x00 0xee 1 34779
>GPT : N Info
>GPT disk GUID : faf27d34cdf44449a198b767638986bd
>GPT entry array : 20 176 separated
>GPT lba range : 64 34734 34779
>GPT partition name : 1 4700610070003000
>GPT partname local : 1 Gap0
>GPT partition GUID : 1 faf27d34cdf44449a199b767638986bd
>GPT type GUID : 1 a2a0d0ebe5b9334487c068b6b72699c7
>GPT partition flags: 1 0x1000000000000001
>GPT start and size : 1 64 276
>GPT partition name : 2 450046004900200062006f006f007400200070006100720074006900740069006f006e00
>GPT partname local : 2 EFI boot partition
>GPT partition GUID : 2 faf27d34cdf44449a19ab767638986bd
>GPT type GUID : 2 28732ac11ff8d211ba4b00a0c93ec93b
>GPT partition flags: 2 0x1000000000000001
>GPT start and size : 2 340 5760
>GPT partition path : 2 /efi.img
>GPT partition name : 3 48004600530050004c0055005300
>GPT partname local : 3 HFSPLUS
>GPT partition GUID : 3 faf27d34cdf44449a19bb767638986bd
>GPT type GUID : 3 005346480000aa11aa1100306543ecac
>GPT partition flags: 3 0x1000000000000001
>GPT start and size : 3 6100 28032
>GPT partition name : 4 4700610070003100
>GPT partname local : 4 Gap1
>GPT partition GUID : 4 faf27d34cdf44449a19cb767638986bd
>GPT type GUID : 4 a2a0d0ebe5b9334487c068b6b72699c7
>GPT partition flags: 4 0x1000000000000001
>GPT start and size : 4 34132 600
>APM : N Info
>APM block size : 2048
>APM gap fillers : 2
>APM partition name : 1 Gap0
>APM partition type : 1 ISO9660_data
>APM start and size : 1 16 1509
>APM partition name : 2 HFSPLUS_Hybrid
>APM partition type : 2 Apple_HFS
>APM start and size : 2 1525 7008
>APM partition name : 3 Gap1
>APM partition type : 3 ISO9660_data
>APM start and size : 3 8533 162
>El Torito catalog : 1672 1
>El Torito cat path : /boot.catalog
>El Torito images : N Pltf B Emul Ld_seg Hdpt Ldsiz LBA
>El Torito boot img : 1 BIOS y none 0x0000 0x00 4 4047
>El Torito boot img : 2 UEFI y none 0x0000 0x00 5760 85
>El Torito img path : 1 /boot/grub/i386-pc/eltorito.img
>El Torito img opts : 1 boot-info-table grub2-boot-info
>El Torito img path : 2 /efi.img
>------------------------------------------------------------------------
>Here we have MBR partition 0xee announcing GPT.
>The Apple Partition Map points to the HFS+ superblock, directory tree,
>and the ISO's data file content blocks. The APM partition is probably
>not mountable by Linux because of its block size 2048.
>Thus the ISO is mountable on USB by Linux only as base device, like
>/dev/sdc. Not from any /dev/sdcN.
>
>
>Now for my proposals.
>
>------------------------------------------------------------------------
>Debian arm64 as proposed for arm64 and ia64:
>------------------------------------------------------------------------
>Volume id : 'Debian 9.4.0 arm64 n'
>System area options: 0x00000a00
>System area summary: MBR cyl-align-off
>ISO image size/512 : 414228
>Partition offset : 16
>MBR heads per cyl : 64
>MBR secs per head : 32
>MBR partition table: N Status Type Start Blocks
>MBR partition : 1 0x00 0x83 64 413332
>MBR partition : 2 0x00 0xef 413396 832
>El Torito catalog : 1808 1
>El Torito cat path : /boot.catalog
>El Torito images : N Pltf B Emul Ld_seg Hdpt Ldsiz LBA
>El Torito boot img : 1 UEFI y none 0x0000 0x00 832 103349
>El Torito img blks : 1 208
>------------------------------------------------------------------------
>
>------------------------------------------------------------------------
>Debian amd64 as proposed for amd64 and i386 (with BIOS boot stuff):
>------------------------------------------------------------------------
>Volume id : 'Debian 9.3.0 amd64 n'
>System area options: 0x00000102
>System area summary: MBR isohybrid cyl-align-on
>ISO image size/512 : 691008
>Partition offset : 16
>MBR heads per cyl : 64
>MBR secs per head : 32
>MBR partition table: N Status Type Start Blocks
>MBR partition : 1 0x80 0x83 64 690112
>MBR partition : 2 0x00 0xef 690176 832
>El Torito catalog : 1872 1
>El Torito cat path : /isolinux/boot.cat
>El Torito images : N Pltf B Emul Ld_seg Hdpt Ldsiz LBA
>El Torito boot img : 1 BIOS y none 0x0000 0x00 4 1873
>El Torito boot img : 2 UEFI y none 0x0000 0x00 832 172544
>El Torito img path : 1 /isolinux/isolinux.bin
>El Torito img opts : 1 boot-info-table isohybrid-suitable
>El Torito img blks : 2 208
>------------------------------------------------------------------------
>
>(My repacked ISOs are slightly larger than originals, because mounting
> -t iso9660 regrettably does not show the hardlink relations in the tree.)
>
>
>Have a nice day :)
>
>Thomas
>
>
--
Steve McIntyre, Cambridge, UK. steve@einval.com
Who needs computer imagery when you've got Brian Blessed?
Reply to: