A start job is running for dev-mapper-sda4_crypt.device (1 min 21s/ no limit)

[David Christensen <dpchrist@holgerdanske.com>]

debian-user:

I have an SSD in my daily driver laptop:

# cat /etc/debian_version ; uname -a
9.12
Linux tinkywinky 4.9.0-12-amd64 #1 SMP Debian 4.9.210-1 (2020-01-20) 
x86_64 GNU/Linux

# lsblk /dev/sda
NAME           MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda              8:0    0 55.9G  0 disk
|-sda1           8:1    0  953M  0 part  /boot
|-sda2           8:2    0  954M  0 part
| `-sda2_crypt 254:1    0  954M  0 crypt [SWAP]
|-sda3           8:3    0 11.2G  0 part
| `-sda3_crypt 254:0    0 11.2G  0 crypt /
`-sda4           8:4    0 42.9G  0 part
  `-sda4_crypt 254:2    0 42.9G  0 crypt


sda1 has unencrypted ext4 boot, sda2 has dm-crypt swap, sda3 has LUKS 
root, and sda4 is leftover space with LUKS that I use for various 
experiments.


sda4_crypt previous had a ZFS pool.  I wanted to change to ext4:

        # zpool destroy tank

        # mkfs.ext4 -L tinkywinky_scratch /dev/mapper/sda4_crypt
        Warning: label too long; will be truncated to 'tinkywinky_scrat'

        mke2fs 1.43.4 (31-Jan-2017)
        /dev/mapper/sda4_crypt contains a zfs_member file system 
labelled 'tank'
        Proceed anyway? (y,N) y
        Creating filesystem with 11235072 4k blocks and 2809856 inodes
        Filesystem UUID: 43dac738-4892-4aa0-a025-0aad0821ee0f
        Superblock backups stored on blocks:
                32768, 98304, 163840, 229376, 294912, 819200, 884736, 
1605632, 2654208,
                4096000, 7962624

        Allocating group tables: done
        Writing inode tables: done
        Creating journal (65536 blocks): done
        Writing superblocks and filesystem accounting information: done

        # mkdir /scratch


Added an fstab entry:

    /dev/mapper/sda4_crypt /scratch ext4 defaults 0 2


Now when I boot, the bootloader hangs trying to open the sda4_crypt LUKS 
container:

    A start job is running for dev-mapper-sda4_crypt.device (1 min 21s/ 
no limit)


I have not changed the crypttab entry:

    sda4_crypt UUID=a6e9b9d2-04c4-4fcd-a058-a596b201c7ed 
/root/tinkywinky.key luks


I tried updating GRUB:

        # update-grub
        Generating grub configuration file ...
        Found background image: .background_cache.png
        Found linux image: /boot/vmlinuz-4.9.0-12-amd64
        Found initrd image: /boot/initrd.img-4.9.0-12-amd64
        Found linux image: /boot/vmlinuz-4.9.0-11-amd64
        Found initrd image: /boot/initrd.img-4.9.0-11-amd64
        done


I verified the mode on the keyfile:

    # ll /root/tinkywinky.key
    -r-------- 1 root root 64 2020-02-15 18:09:04 /root/tinkywinky.key


I updated initramfs:

        # update-initramfs -u
        update-initramfs: Generating /boot/initrd.img-4.9.0-12-amd64
        cryptsetup: WARNING: target sda2_crypt has a random key, skipped


If I comment out the fstab entry, the system will boot and sda4_crypt is 
opened:

2020-05-18 11:12:42 root@tinkywinky ~
# cryptsetup status sda4_crypt
/dev/mapper/sda4_crypt is active.
  type:    LUKS1
  cipher:  aes-xts-plain64
  keysize: 256 bits
  device:  /dev/sda4
  offset:  4096 sectors
  size:    89880576 sectors
  mode:    read/write


If I then enable the fstab entry, I am able to mount the filesystem 
manually:

# mount /scratch

# mount | grep scratch
/dev/mapper/sda4_crypt on /scratch type ext4 (rw,relatime,data=ordered)


My guess is that there is some leftover metadata that is used by some 
stage of the Debian boot process related to mounting the /scratch ext4 
filesystem, and that this metadata did not get cleared or set correctly 
when I switched from ZFS to ext4 (?).


I have never been able to find a current, correct, and complete 
explanation for how Debian boots.


Comments?  Suggestions?


David