Hi Steve, Happy New Year to you too! On 2021.01.12 11:56, Steve McIntyre wrote:
Hey Pete, HNY etc. Thanks for joining the discussion! :-) On Sun, Jan 10, 2021 at 08:19:15PM +0000, Pete Batard wrote:On 2021.01.10 18:56, Lou Poppler wrote:On Sun, 2021-01-10 at 18:13 +0100, Thomas Schmitt wrote:Hi, Lou Poppler wrote:If you want to recommend rufus, please do so ONLY with a prominent, un-ignoreable warning explaining that the user MUST select rufus' "DD mode" at the prompt immediately before writing the media.For people who seem to be under the impression that "(Rufus) will replace parts of the debian image with its own recipe of loaders and configurations", I have to vehemently dispute that it is doing any such thing. The goal of Rufus is to create a bootable USB with content that is as close as possible to the ISO content *AND* is bootable as USB media, period. As such, when booting in UEFI mode, the *only* element that Rufus may modify are the labels used in the GRUB/Syslinux config files (kernel option) for source media lookup, to accommodate FAT32 limitations on that matter (since FAT limits labels to 11 uppercase characters, and ISO volumes can and usually do use labels with much longer names).Thanks for clarifying that.Apart from this, everything else is a binary identical copy of the ISO content, which you can easily validate if needed. Then, for BIOS mode, we of course need to install GRUB or Syslinux bootloaders, which can't come from the ISO but which we produce from vanilla official releases, and rely on the distro not to have deviated too much in terms of patches from vanilla GRUB/Syslinux. However, if such deviation occurs, you usually get an early boot error that is easy to diagnose, and we also have provision, through the bootloader download facility of Rufus, to produce GRUB/Syslinux bootloaders that include specific fixes, such as the ones required for BootHole, since GRUB has yet to produce a mainline release that includes those. So I have to be adamant that if the media doesn't boot, especially in UEFI mode, then it's not a matter of Rufus introducing "its own recipe of loaders and configuration" but rather a problem of distro maintainers not having properly tested UEFI/FAT32 boot mode by extracting the ISO content onto a FAT32 partition, and trying to boot said media. And I am pretty positive that you will find that the users who reported issues after creating their media through Rufus, will report the exact same issues if creating the media themselves without using a third party utility. Which brings me nicely to the point that the current Debian Bullseye test images have broken said mode of operation (UEFI install from ISO content extracted on FAT32) early in December, which is causing problems for people trying to install Debian on Pi 4 for instance ([1] but my testing shows that this applies to more than arm64. And yes, I've been meaning to file a bug for this, but I haven't had a chance to do that yet).Argh. Thanks for raising this now, at least. I certainly haven't consciously made any changes to debian-cd that would cause this, so I'm a little surprised to hear about it. I'll take a look in the d-i code...
Further testing (which I also mentioned in this thread) seems to indicate that this only seems to apply for UAS media, so it's possible that it wasn't a recent change, but that install media detection has not been compatible with UAS devices for some time, which we only started to pick it up on account that some of us have been testing Pi 4 Debian installation with UAS.
I'm still planning to look more closely into this when I get a chance, and create a new bug (unless someone beats me to it), but if you have a UAS device lying around, you should be able to replicate the issue with the current Bullseye installation ISOs, including amd64 ones, as I confirmed that the problem was (still) present in the latest netinst testing amd64 ISOs.
FYI, the USB subsystem should tell you if a device is UAS with something like this (from dmesg):
[549114.637578] usb 8-1: new SuperSpeed Gen 1 USB device number 19 using xhci-hcd [549114.658815] usb 8-1: New USB device found, idVendor=174c, idProduct=55aa, bcdDevice= 1.00 [549114.658837] usb 8-1: New USB device strings: Mfr=2, Product=3, SerialNumber=1
[549114.658852] usb 8-1: Product: Ventura Ultra [549114.658867] usb 8-1: Manufacturer: Mushkin [549114.658881] usb 8-1: SerialNumber: 000000000025 [549114.666050] scsi host0: uas[549114.668168] scsi 0:0:0:0: Direct-Access Mushkin Ventura Ultra 0 PQ: 0 ANSI: 6
So le me reiterate this plea: PLEASE, PLEASE, PLEASE, do not treat ISOHybrid as an excuse to ignore non DD mode of installation, because, when you look at it objectively, you should find that DD is *NOT* a panacea (for instance, DD mode is completely useless for installation of vanilla Bullseye on a Pi 4, whereas FAT32 extraction, *when* not broken, gives you the same experience as if you were installing Debian on a UEFI x86 PC) and users do and will rely on a non DD mode of creating their install media.Hmmm. What exactly is the problem with the Pi 4, out of curiosity?
The problem with the Pi 4 (as with all Pi's) is that it needs to load a set of binary files, from SD or USB media, before it can boot anything, and we also need to load the UEFI firmware from SD or USB.
Now the nice thing, that was introduced a few months ago on the Pi 4 is that you can load all these files from an ESP, with the caveat that the ESP has to be at the beginning of your media (which means for instance that even if adding files onto it wasn't an issue, the partition layout used by the current ISOHybrid when copied in DD mode would still make that media unbootable on a Pi 4).
This logically leads to the idea that, if you are okay with sacrificing a bit of space for the ESP, you can place all of these Raspberry Pi specific boot files on it, along with the Debian installation content extracted from a netinst or mini ISO (we advise netinst as it makes it faster to get to a minimal serviceable install than mini for people with slow connection, and because we don't see the extra space needed as that big of an issue) and once you have that, you can boot and go through a full default installation of Debian and end up with a fully working system.
And what makes it even better is that you don't even need to provision two medias for this (install media + target), as the same media can transparently act as both the installer and target.
Considering that the Pi has no "internal" disk or flash to install a distro to, and that one must use either an SD card or USB drive, it does make a lot of sense in my opinion not to require users to juggle with multiple media, especially as, and this is one of the main pain points we are trying to avoid through this method, using two media would force users to go through the shell during the install process to ensure that they copy all the Pi boot files and UEFI firmware from the install media to the target before they reboot, as, again, a Pi will not boot any media where these extra files aren't present.
But by reusing the ESP, where these files have already been copied, no extra step is necessary.
Furthermore, if you simply go with the defaults during Debian installation, the disk partitioner will recognize the ESP as reusable and set it as the ESP to use for the new system, and therefore when GRUB installs for the final system, it simply replaces the one that was used for booting the installer, which ensures that, once the user reboot, they get into their newly installed system.
In other words, when everything works properly, installing Debian through this method is virtually the same as installing Debian on an x86 PC, with no need for extra caveats or having users perform additional configuartion.
The only slightly problematic point is the creation of the bootable/target media, but even that is not that difficult, as it can be summarized in:
- Create a GPT/ESP that's large enough in size and format it to FAT32. - Copy all the necessary Raspberry Pi boot and UEFI firmware files onto it. - Extract all the content from the Debian netinst/mini ISO onto the ESP.But of course, this relies on the Debian installation process to be able to handle source content being located on a FAT32 drive, and having been extracted from the ISO through file copy, i.e. on the Debian installer being able to properly locate the installation content from USB media that wasn't created through dd.
And once again, I have to reiterate that, from my experience as the purveyor of software that people may use to install Debian, that primarily relies on this mode of installation, Debian has always done a very good job with handling FAT32 (for instance, Debian is not over relying on labels to identify the source media, but instead on a lookup that checks for the presence of specific files), even as I know this requires extra efforts from the maintainers, so I am fairly confident that this is probably just something that we failed to notice previously, on account that it is thankfully not as widespread an issue due to an environmental component (only seems to affect USB drives with UAS support), and that can hopefully be fixed without incurring too much effort.
Oh, and in case other people are looking at this, I'm going to bring up that we recently found what appears to be a more straightforward regression this time, with broken support for the NIC that the Pi 4 uses (details at [1]) which of course makes networked installation very problematic. But that's a different matter (which I'm also planning to look more closely into when I get a chance).
Regards, /Pete[1] https://www.raspberrypi.org/forums/viewtopic.php?f=50&t=282839&sid=757bdd11714483ca73fea35af8bb9b64&start=50#p1791942