Bug#740504: cdimage.debian.org: Released ISO images have invalid GPT tables
Hi,
Chris Bainbridge:
> I still can't create a partition, which was the original aim. Is it
> supposed to work?
I succeeded with gdisk on my oldstable test machine (help texts and
clueless actions of mine not shown):
------------------------------------------------------------------
$ dd if=debian-testing-amd64-netinst.iso of=/dev/sdb bs=2048
$ /bin/gdisk /dev/sdb
GPT fdisk (gdisk) version 0.8.1
Partition table scan:
MBR: MBR only
BSD: not present
APM: not present
GPT: present
Found valid MBR and GPT. Which do you want to use?
1 - MBR
2 - GPT
3 - Create blank GPT
Your answer: 2
Using GPT and creating fresh protective MBR.
Command (? for help): p
Disk /dev/sdb: 3915776 sectors, 1.9 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 2A2942FB-D14C-4710-A41F-DECF92DA20FE
Partition table holds up to 208 entries
First usable sector is 64, last usable sector is 446410
Partitions will be aligned on 4-sector boundaries
Total free space is 3 sectors (1.5 KiB)
Number Start (sector) End (sector) Size Code Name
2 3572 4403 416.0 KiB 0700 ISOHybrid1
Command (? for help): x
Expert command (? for help): e
Relocating backup data structures to the end of the disk
Expert command (? for help): m
Command (? for help): p
Disk /dev/sdb: 3915776 sectors, 1.9 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 2A2942FB-D14C-4710-A41F-DECF92DA20FE
Partition table holds up to 208 entries
First usable sector is 64, last usable sector is 3915712
Partitions will be aligned on 4-sector boundaries
Total free space is 3469305 sectors (1.7 GiB)
Number Start (sector) End (sector) Size Code Name
2 3572 4403 416.0 KiB 0700 ISOHybrid1
Command (? for help): n
Partition number (1-208, default 1): 3
First sector (446408-3915712, default = 446408) or {+-}size{KMGTP}:
Last sector (446408-3915712, default = 3915712) or {+-}size{KMGTP}:
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300):
Changed type of partition to 'Linux filesystem'
Command (? for help): p
Disk /dev/sdb: 3915776 sectors, 1.9 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 2A2942FB-D14C-4710-A41F-DECF92DA20FE
Partition table holds up to 208 entries
First usable sector is 64, last usable sector is 3915712
Partitions will be aligned on 4-sector boundaries
Total free space is 0 sectors (0 bytes)
Number Start (sector) End (sector) Size Code Name
2 3572 4403 416.0 KiB 0700 ISOHybrid1
3 446408 3915712 1.7 GiB 8300 Linux filesystem
Command (? for help): w
Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!
Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT).
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot.
The operation has completed successfully.
$ /sbin/gdisk /dev/sdb
GPT fdisk (gdisk) version 0.8.1
Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present
Found valid GPT with protective MBR; using GPT.
Command (? for help): p
Disk /dev/sdb: 3915776 sectors, 1.9 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 2A2942FB-D14C-4710-A41F-DECF92DA20FE
Partition table holds up to 208 entries
First usable sector is 64, last usable sector is 3915712
Partitions will be aligned on 4-sector boundaries
Total free space is 0 sectors (0 bytes)
Number Start (sector) End (sector) Size Code Name
2 3572 4403 416.0 KiB 0700 ISOHybrid1
3 446408 3915712 1.7 GiB 8300 Linux filesystem
Command (? for help): q
------------------------------------------------------------------
The overwritten MBR might be needed for booting, if the firmware
does not boot via GPT.
In this case, use an MBR partition editor. In worst case one has
to kill the GPT first:
# Kill main GPT header block imported with the ISO image
$ dd if=/dev/zero bs=512 count=1 conv=notrunc seek=1 of=/dev/sdb
The backup GPT of the ISO will not be recognized as soon as the
main GPT is gone. But there may be an old backup GPT at the end
of the stick.
# Kill old backup GPT at the end of the stick (coarsely: 1900 MB to end)
$ dd if=/dev/zero bs=1M seek=1900 of=/dev/sdb
Now the stick should be recognized as MBR only.
(gdisk will spoil the MBR then. But fdisk should work.)
------------------------------------------------------------------
I see two non-standard aspects in the original GPT, caused by the
fact that debian-cd produces images rather than formatting disks,
and by the fact that isohybrid is quite a hack.
The partition editor must be able to adjust the GPT to a larger disk.
Further the GPT which get produced according to isohybrid
prescriptions bear partition 1 with an undigestible start block 0.
This is probably why GPT partition 1 does not show up in gdisk.
So we might be motivated to develop and test a deviation from
the isohybrid specifications.
Is debian-cd willing to undertake such a test ?
If so, then i will have to enhance xorriso so that it can write
partition start 64 into the first GPT partition of an isohybrid
image. debian-cd will have to re-enable -partition_offset 16.
-----------------------------------------------------------------
Reasoning about size adjustment of GPT:
GPT has the fundamental concept of covering the whole "disk".
Other than with MBR there is no unused space but only space that
is not claimed by any partition entry. The overall GPT claims all
blocks of the disk. What it does not claim, does not exist
officially.
The situation on USB stick gets consistent by moving the backup GPT
to the end of the stick and by enlarging the nominal disk size
accordingly. (gdisk can do this by commands "x", "e".)
-----------------------------------------------------------------
Reasoning about isohybrid quirk:
According to the isohybrid prescriptions partition 1 is supposed
to cover the whole ISO image. This is because all isohybrid
partitions are supposed to be mountable.
The image starts at block 0. So the partition starts there too.
But the first valid start block according to the GPT header would
be 64. This number is written there, because it is supposed to
be a block address after the end of the GPT array.
Changing the first valid block number to 0 would probably produce
a non-standard GPT header block.
So starting the partition at block 64 seems more worthwhile.
To keep the partition mountable, debian-cd would have to re-enable
option -partition_offset 16 with the ISO production.
Mountability is important if the booting system uses udev's
/dev/disk/by-label to find the ISO image.
One can study the negative effects of unmountable partitions with
grub-mkrescue images.
GRUB2 prescribes a a more standard approach to GPT:
- Protective MBR with only one non-mountable partition starting
at LBA 1 (isohybrid has a real MBR with several mountable
partitions).
- Non-mountable first GPT partition starting at LBA 64.
- Mountable GPT FAT partiton pointing to the EIF boot image.
- Non-mountable third GPT partition ending at the backup GPT.
This should be more acceptable to partition editors.
But the unmountable MBR partition runs into a bug-or-feature of
udev resp. blkid which attribute the "disk label" of the base
device to unmountable partitions. This makes the softlink of the
unmountable partition overwrite the softlink of the base device
in /dev/disk/by-label.
I.e. /dev/disk/by-label/'Debian testing amd64 1' would show up
but would not be mountable to continue the boot process.
If d-i uses /dev/disk/by-label to find the ISO after booting,
then debian-cd must avoid any unmountable partition.
(Question to users of udev at boot time:
Do you believe it is wise to rely on this asynchronous software
with its quite obscure entrails ? What is wrong with just testing
the devices /dev/sr* and /dev/sd[a-z] by blkid ?)
-----------------------------------------------------------------
Gparted seems unable to cope with the situation:
> # parted /dev/sdb
> Error: The backup GPT table is not at the end of the disk, as it
> should be. This might mean that
> another operating system believes the disk is smaller. Fix, by moving
> the backup to the end (and removing the old backup)?
This complaint points out that the GPT does not fit the disk size.
> Fix/Ignore/Cancel? Fix
> Warning: Not all of the space available to /dev/sdb appears to be
> used, you can fix the GPT to use
> all of the space (an extra 14905344 blocks) or continue with the
> current setting?
The main GPT has a size field which can hardly match your storage
device either.
> Fix/Ignore? Fix
> Error: Unable to satisfy all constraints on the partition.
(parted) p
> Error: The backup GPT table is corrupt, but the primary appears OK, so
> that will be used.
Now the software sends the user around in circles.
> # gparted debian-testing-amd64-netinst.iso
> [create partition table in gui]
> No partition table found on device debian-testing-amd64-netinst.iso
That's astounding. Is Gparted supposed to work with disk files ?
Have a nice day :)
Thomas
Reply to: