Re: systemd, headless, SSH, manual decryption (was: Re: Change systemd to not be default in Stretch)
Christian,
Thanks for this. I've printed it on paper for study and reference. It
is slow going for me, but I finally have a feeling that I might have a
chance at understanding systemd. The printed version is eleven
USletter pages. So far, I've gotten to middle of page three.
Best regards,
pec
On 20150623_1852+0200, Christian Seiler wrote:
> On 06/23/2015 12:59 PM, Erwan David wrote:
> > Note that I use policy-rc.d to check whether the encrypted disk is
> > mounted for the daemons that need it (it allows not to change the init
> > files)
>
> That works? policy-rc.d should only affect invoke-rc.d, which shouldn't
> be relevant at boot, but only in maintainer scripts. (AFAIK at least.)
>
> > For what I need to know : I have a headless machine with an encrypted disk.
> > I cannot ask the password on console, so
> > 1) at boot I do not mount the encrypted disk, and start a minimal set
> > of daemons, among them the ssh daemon.
> >
> > 2) I ssh to the machine then mount encrypted disk and start remaining
> > daemons.
> >
> > How can I do this with systemd ?
>
> This is a great question because it presents a nice little problem that
> covers quite a few of topics regarding systemd. I've sat down and
> solved your little problem from a systemd perspective, and hopefully my
> solution will help you in understanding how systemd works.
>
> First of all a couple of very basic explanations: what is a unit? A
> unit is anything that systemd manages and/or monitors. Relevant for
> our case are the types service, target and mount, but systemd supports
> more than that (see man systemd.unit and the references therein for
> more details).
>
> A service unit is probably the simplest to understand, it corresponds
> to that which was previously provided by /etc/init.d scripts. I'm not
> going to go into much detail here about that.
>
> A mount unit represents a mount in the system. If you manually mount
> something via the mount(8) command, systemd will not interfere, but it
> will synthesize a dynamic mount unit for it. So for example, if you
> have:
> mkdir /mnt/a /mnt/b
> mount --bind /mnt/a /mnt/b
> Then you can also umount /mnt/b via the command:
> systemctl stop mnt-b.mount
> (umount /mnt/b will continue to work, of course)
>
> For manual mounts this is not really that relevant (I suspect even most
> systemd developers will use plain old umount in that case), but mount
> units are the method systemd uses to handle /etc/fstab: for every entry
> in /etc/fstab a mount unit is generated, that's how /etc/fstab is
> integrated into the boot process. (See below for further details.)
>
> A target is in some sense the simplest kind of unit: it can only have
> dependencies (and a description), but nothing else. You can achieve the
> same thing with a dummy /bin/true service unit. Targets are useful for
> grouping things together and also for providing synchronization points.
> For example, as is documented in man 7 bootup, there's a target called
> local-fs.target that has the semantics that every local filesystem
> mount in /etc/fstab will be ordered before it, and so that every
> service that orders after it can be sure that local filesystems are
> already mounted at that point. (Most services are implicitly ordered
> after local-fs.target because by default everything is ordered after
> basic.target, which itself is ordered after local-fs.target. You can
> override these kind of things if you need to, however.)
>
>
>
> How does systemd boot up a system? After doing some very basic
> initialization (such as setting the hostname from /etc/hostname and
> mounting some essential kernel filesystems such as /proc and /sys), it
> will try to start a specific unit, either the default (called
> 'default.target') or any unit specified on the kernel command line via
> the option systemd.unit=XXX. default.target is a symlink to
> graphical.target on Debian, but that can be overridden. (If you don't
> have a GUI installed, graphical.target is equivalent to
> multi-user.target, in analogy to the runlevels 2-5 on Debian with
> sysvinit.) graphical.target itself depends on a lot of things, and thus
> all services required for system startup are automatically pulled in.
>
>
>
> Now how does one solve the problem you have? There are multiple ways to
> do so, but for the sake of simplicity I'm going to show just one, the
> one I personally prefer (YMMV).
>
> The basic outline would be this:
>
> - tell systemd to NOT automatically decrypt the drive at boot and to
> NOT automatically mount it
> - create a new target unit that will serve as a new boot target, and
> it will only contain ssh + syslog as services (beyond the very basic
> early-boot things that should always be there)
> - create a second target unit that will pull in the encrypted drive
> and the mount and then ulitmately start the original
> multi-user.target, thus pulling in all other installed daemons on
> the system - that second target will be started manually by you
> after you log in and decrypt the drive
>
>
> Let's begin. First of all you need to add your drive to /etc/crypttab.
> I've done this in a KVM and the entry looks like this:
> crypto /dev/vda5 none luks,noauto
> (This means that /dev/vda5 is the device with the encrypted LUKS data
> on it and /dev/mapper/crypto will be the name of the virtual device
> with the plain text data.)
> Note that here there is a 'noauto' setting, which means that the drive
> should NOT be decrypted automatically at boot (otherwise systemd would
> wait for you to enter the password on the console before even mounting
> the local filesystems, so it would hang...).
>
> What happens internally here is that systemd uses a so-called generator
> to dynamically transform the contents /etc/crypttab into systemd units.
> In this case, it generates a service for each line, in our case it will
> generate systemd-cryptsetup@crypto.service. If we had not put noauto
> here, the service would be started automatically at boot, asking you
> for your password at the console. But with 'noauto' here, the service
> will be synthesized by the generator, but not started automatically.
>
> You can now see if systemd picks up on that crypttab entry be running
> systemctl daemon-reload
> That does the following:
> - re-runs all generators (for reading /etc/crypttab, /etc/fstab, ...)
> - re-reads all static and generated unit files on the disk
> => so every time you change something, you need to run the above
> command for systemd to pick it up
> You can see that the unit was detected by systemd via:
> systemctl status systemd-cryptsetup@crypto.service
> But it should also tell you it's inactive (i.e. hasn't been started so
> far).
>
> Now we add the entry for the filesystem in /etc/fstab:
> /dev/mapper/crypto /srv ext4 rw,noauto 0 0
>
> Here we also have a noauto setting to make sure it's not automatically
> mounted at boot. This is really important here, since this is a local
> filesystem and systemd considers any local filesystem that couldn't be
> mounted at boot to be a fatal error. So if you leave out the noauto
> here, systemd will wait 90s by default for the device to appear at boot
> and (since it won't appear) will stop the boot process and start an
> emergency shell instead. With a headless system that's a really bad
> thing to happen, which is why you should definitely put 'noauto' here.
> Then systemd won't automatically try to mount it at boot at all.
>
>
>
>
> Now that that's done, let's create our target unit for booting the
> system with just a minimal set of daemons. We don't want to interfere
> with the early-boot process, so we only want to replace the default
> graphical.target, but nothing else.
>
> When editing static units, you should know that systemd knows two
> directories where those may be stored: /lib/systemd/system and
> /etc/systemd/system. /lib/systemd/system is for things that have been
> packaged, whereas /etc/systemd/system is the territory of the
> administrator. If files with the same name exist in both directories,
> the file in /etc/systemd/system completely overrides the file in
> /lib/systemd/system.
>
> So in this case, we want to edit things in /etc/systemd/system. Let's
> create a new file /etc/systemd/system/before-decrypt.target. That file
> is supposed to be our new default target, so let's put in the
> following:
> [Unit]
> Description=System before Decryption
> Requires=basic.target
> Conflicts=rescue.service rescue.target
> After=basic.target rescue.service rescue.target
> AllowIsolate=yes
>
> The contents (apart from Description= and Documentation=) is identical
> to the default /lib/systemd/system/multi-user.target. (graphical.target
> just pulls that in and since we have a headless system, let's skip the
> extra indirection.)
>
> To understand this file, we need to talk systemd dependencies for a
> moment. There are a lot of different types of dependencies in systemd,
> but in this case you need to understand only a couple (those are also
> the most important ones):
>
> - Requires: if unit A requires the unit B, it means that if A is to
> be started, B also has to be started (and if B fails to start, A
> will not start either). Also, if B is to be stopped, A will also
> be stopped. (Note that this only affects explicit actions, so A
> would not be stopped if B segfaults and just dies unexpectedly,
> unless you tell systemd to do that with an additional setting.)
>
> - Wants: if unit A wants the unit B, it means that if A is to be
> started, B also has to be started; but A will be started regardless
> of whether A fails to start or not (or even exists or not) - and
> A will not be stopped if B is stopped. It's a weaker version of
> Requires
>
> - Conflicts: if unit B conflicts with unit B, unit B will be stopped
> if unit A is to be started and vice-versa
>
> - Before/After: declares ordering within a single transaction. Note
> that ordering is orthogonal in systemd: Requires/Wants do NOT imply
> ordering dependencies, those have to be made explicitly. (Note that
> for targets Requires/Wants do sometimes result in implicit ordering,
> see below for details; but all other unit types don't have that
> logic, so service A requiring service B will NOT lead to an ordering
> between them, unless you also specify that explicitly.)
>
> So in the case of our before-decrypt.target, we have the following
> dependencies:
>
> - Requires=basic.target
> This means that the basic system setup should be pulled in. This
> is essential; if you leave that out the most basic startup things
> will not be started if you try to boot into before-decrypt.target.
>
> - Conflicts=rescue.service rescue.target
> If you were dropped in an emergency shell at boot (or requested it
> explicitly over the command line), this conflict will tell systemd
> that once you do continue booting from the emergency shell, it
> should stop the emergency shell automatically.
>
> - After=basic.target rescue.service rescue.target
> Make sure that before-decrypt.target is started only after the very
> basic system initialization is complete.
>
> But wait a minute: we copied this unit from multi-user.target, where
> all the non-GUI services should all be running... Where are they? This
> unit file appears to only pull in basic.target and nothing else.
>
> Well, since you don't want to modify a unit file every time you install
> a new daemon on your system, systemd provides ways to add additional
> dependencies to units dynamically via symlinks.
>
> There are two directories for multi-user.target:
> /lib/systemd/system/multi-user.target.wants
> /etc/systemd/system/multi-user.target.wants
>
> Each directory contains symlinks to unit files that are automatically
> added as Wants= type dependencies to the target (you can also have
> .requires). The directory in /lib contains all native systemd services
> that shouldn't be disabled by the administrator (and systemctl disable
> won't work on them although you could manually do something), while the
> latter contains all service that can easily be disabled by the
> administrator (those include SSH, cron, MTAs, web servers, ...).
>
> Typically one wants to still enable all units from the /lib directory
> also in our target, so let's do the following:
> mkdir /etc/systemd/system/before-decrypt.target.wants
> cd /etc/systemd/system/before-decrypt.target.wants
> for i in /lib/systemd/system/multi-user.target.wants/* ; do
> ln -s /lib/systemd/system/$(basename $i) .
> done
> Then we additionally want to enable SSH and syslog (but no other
> services):
> cd /etc/systemd/system/before-decrypt.target.wants
> ln -s /etc/systemd/system/syslog.service .
> ln -s /lib/systemd/system/ssh.service .
>
> A couple of notes:
>
> - we take syslog.service from /etc because I don't know what your
> favorite syslog daemon is and syslog.service is always a symlink
> to the current one (although you may have to force it manually
> because in Jessie there's a Debian packaging bug if you change
> syslog daemons that the symlink doesn't get updated)
>
> - both syslog and ssh are native services, which is why that works
>
> - you can also enable additional services you might need in the same
> way. There's no cron nor atd started here (because those might
> require the encrypted partition to be mounted - or not, depending
> on your system) and also no MTA
>
> - note that these symlinks only work if you have native systemd
> service files. You can also enable init scripts, but then you have
> to explicitly specify Wants=[INITSCRIPTNAME].service, e.g.
> Wants=exim4.service - since the service files for init scripts are
> dynamically generated and you thus can't easily just create a simple
> symlink (in the default case they are activated because systemd's
> generator looks for /etc/rcX.d/S**INITSCRIPTNAME symlinks and then
> creates Wants= dependencies for runlevelX.target dynamically - and
> all runlevel[2-5].target are symlinks to graphical.target)
>
> Then we should be nearly ready for a test boot to see if our minimal
> system boots. To summarize what we have done so far:
>
> - modify /etc/crypttab and /etc/fstab to add all required entries with
> the 'noauto' option
> - create /etc/systemd/system/before-decrypt.target (see above for
> the contents)
> - create directory /etc/systemd/system/before-decrypt.target.wants and
> symlink all services in multi-user.target that we need before the
> partition is decrypted
>
> As a final step we can now make our new target the default boot target:
>
> systemctl set-default before-decrypt.target
>
> Now let's reboot the system. It will come up again and hopefully the
> following will have happened:
>
> - it didn't hang at boot (otherwise check noauto options)
> - SSH was started
> - other system daemons that were not included in the list we activated
> were NOT started (e.g. no cron/atd/exim4)
>
> Now for the final step: we want to be able to decrypt the partition and
> enter the password. So let's create a file
> /etc/systemd/system/decrypt.target that contains the following:
>
> [Unit]
> Description=Decrypted System
> Requires=before-decrypt.target
> After=before-decrypt.target
> Conflicts=systemd-ask-password-console.path systemd-ask-password-console.service systemd-ask-password-plymouth.path systemd-ask-password-plymouth.service
> Requires=systemd-cryptsetup@crypto.service srv.mount start-full-system.service
>
> The interesting settings in that file are:
>
> - Conflicts= lots of ask-password stuff:
> This is to work around a quirk in systemd's handling of asking
> passwords for encrypted partitions. See footnote [1] for an
> explanation if you are really interested.
>
> - Requires=...: this is the meat of the matter. Here we make sure we
> pull in
> - the service that decrypts the partition
> - the mount for /srv
> - and an additional service (see below) that will cause the rest
> of the daemons to start (start-full-system.service)
>
> This means that when one then starts this target, it will cause the
> partition to be decrypted, the filesystem to be mounted and then all
> daemons to be started.
>
> Let's look at start-full-system.service:
> [Unit]
> Description=Start full system after decryption
> After=decrypt.target
>
> [Service]
> Type=oneshot
> ExecStartPre=/bin/systemctl is-active --quiet decrypt.target
> ExecStart=/bin/systemctl --no-block start multi-user.target
>
> The unit calls systemctl to start multi-user.target. The reason this
> is done is because of ordering: we only want to start daemons once the
> filesystem is mounted. But we don't want to add an explicit After=
> dependency to each daemon (or list every daemon as a Before= depdency
> here), so instead of pulling in multi-user.target directly in
> decrypt.target and adding all those dependencies, we simply create a
> small service here that is ordered AFTER decrypt.target that causes
> systemd to start multi-user.target.
>
> A word on service ordering: unless you set DefaultDependencies=no, a
> target will always be ordered AFTER all of its other dependencies
> (Requires=, Wants=, Conflicts=, ...). However, if a service already has
> an explicit ordering dependency, this will not occur. So what we have
> here is the following ordering:
>
> decrypt.target has an implicit
> After=systemd-cryptsetup@crypto.service srv.mount
> BUT because start-full-system.service is ordered relative to
> decrypt.mount explicitly already, it doesn't get an implicit ordering.
>
> So that means we have the following order w.r.t. those 4 units:
>
> systemd-cryptsetup@crypto.service srv.mount
> | |
> \-------------------+--------------/
> |
> v
> decrypt.target
> |
> v
> start-full-system.service
>
> (In principle, one could also order srv.mount after the cryptsetup
> service, but that is unnecessary, since mount units in systemd will
> always wait for the corresponding devices to appear, so an explicit
> ordering is not required here.)
>
> If you are wondering what the ExecStartPre= is for: it's to make sure
> that decrypt.target was successful before it continues. Otherwise, if
> decrypting and/or mounting fails, it would still be executed, so add
> an additional check that it only starts if decrypting was
> successful.[2]
>
> As a final thing w.r.t. start-full-system.service, let's talk about
> the --no-block here: it causes systemctl to enqueue the start job for
> multi-user.target but will return immediately. The reason for that is
> that systemd serializes transactions to a certain extent, so that not
> using --no-block here would cause the system to deadlock because
> systemctl would wait for the multi-user.target transaction to finish
> but systemd wouldn't begin the transaction until the transaction
> containing start-full-system.service would be finished. (It's generally
> a very bad idea to call systemctl without --no-block in service files.)
> However, this means that the starting of the daemons will be initiated
> by starting decrypt.target, but it will be done in the background.
>
>
>
>
>
> Once this is done, let's do a systemctl daemon-reload to make sure
> systemd has picked up the changes and now we can try to see if it
> works:
>
> systemctl start decrypt.target
>
> You will now be asked for the encrypted partition's password. Once
> you've entered that, the filesystem will be mounted, systemctl start
> will return you to the command line and in the background all daemons
> will be started automatically.
>
>
>
> You can reboot the system, log in per SSH again, and use the command
> again - and it should work.
>
>
> Now wait a minute: how does this password prompt work exactly?
>
> - systemd-cryptsetup@XXX.service will tell systemd that it needs a
> password to decrypt the drive
>
> - systemd has to ways of getting to such a passowrd: either via a
> so-called 'ask-password agent' (typically used at boot or if the
> administrator plugs in a known encrypted drive dynamically) - or,
> if the password is asked as part of a transaction that was triggered
> via systemct, systemctl will ask the password directly.
>
> So this means that simply telling systemd to decrypt the drive will
> then lead to a direct password prompt at that point. So in the end, one
> needs only to enter one single command, then enter the password.
> Everything will be decrypted and all services will be started then.
>
> Neat, huh?
>
>
>
> Summary of what we did:
>
> - adjust /etc/crypttab and /etc/fstab : use noauto!
>
> - create /etc/systemd/system/before-decrypt.target
>
> [Unit]
> Description=System before Decryption
> Requires=basic.target
> Conflicts=rescue.service rescue.target
> After=basic.target rescue.service rescue.target
> AllowIsolate=yes
>
> - enable required services for before-decrypt.target
>
> mkdir /etc/systemd/system/before-decrypt.target.wants
> cd /etc/systemd/system/before-decrypt.target.wants
> for i in /lib/systemd/system/multi-user.target.wants/* ; do
> ln -s /lib/systemd/system/$(basename $i) .
> done
> ln -s /etc/systemd/system/syslog.service .
> ln -s /lib/systemd/system/ssh.service .
>
> - create /etc/systemd/system/decrypt.target
>
> [Unit]
> Description=Decrypted System
> Requires=before-decrypt.target
> After=before-decrypt.target
> Conflicts=systemd-ask-password-console.path systemd-ask-password-console.service systemd-ask-password-plymouth.path systemd-ask-password-plymouth.service
> Requires=systemd-cryptsetup@crypto.service srv.mount start-full-system.service
>
> - create /etc/systemd/system/start-full-system.service
>
> [Unit]
> Description=Start full system after decryption
> After=decrypt.target
>
> [Service]
> Type=oneshot
> ExecStartPre=/bin/systemctl is-active --quiet decrypt.target
> ExecStart=/bin/systemctl --no-block start multi-user.target
>
> - reboot
>
> - to decrypt: systemctl start decrypt.target, enter your password at
> the prompt
>
>
>
> A couple of comments:
>
>
> 1. This can be extended to multiple encrypted drives and also multiple
> mounts. If you want to see what unit name systemd wants to use for
> a given mount point (so you know what to set as Requires= in
> decrypt.target), you can use 'systemctl status /mount/point' to see
> how systemd mangles that into a unit name
>
> 2. If you install additional daemons, Debian will hook them up in
> multi-user.target (either directly if they are native or indirectly
> if they are sysv scripts), so if you install something that should
> already be started before decryption, you need to manually add it.
>
> On the other hand, since we don't modify multi-user.target itself,
> any new daemon will always be hooked up there automatically, so
> that if you install something, it will automatically be part of the
> start-only-when-decrypted logic.[3]
>
> 3. If you have systemd installed, it's recommended (but not strictly
> required) to have dbus installed. If you have, note that dbus is
> the only service (other than systemd's internal ones and some
> plymouth stuff that's not relevant for headless systems) that hooks
> up in /lib/systemd/system/multi-user.target.wants (and not in
> /etc, as all other services do!), and if it's installed it should
> probably be activated already for the minimal system, so if you
> install dbus only after copying all the symlinks to
> before-decrypt.target.wants (see above), then you should copy that
> one manually.
>
> 4. If you storage stack is a bit more complicated (e.g. you use LVM
> on top of the encryption), you may or may not need to manually add
> some services to activate that once the drive is decrypted. It's
> definitely possible, and might even work out of the box (I haven't
> tried it), but you may have to play around a bit there.
>
> 5. If you truly have a headless system, you should test this in a VM
> first, especially when it comes to the boot process.
>
>
>
> So there it is: how to solve your particular problem with systemd on
> Jessie. I hope I could use this use-case to illuminate a couple of
> concepts in systemd and give you enough information so that you have a
> good starting point into the topic.
>
>
> Regards,
> Christian
>
>
> [1] The Conflicts= is required since systemd's own service to notify
> root of password prompts via wall(1) has the following command:
> /bin/systemctl stop [... all units listed in Conflicts= here ...]
> This is fine because it's typically started dynamically when the
> administrator plugis in a drive dynamically or so (and is not
> active during boot, there the other services are active, which
> in turn don't really work after boot, that's why they are stopped).
> The problem here is that we start this from within a transaction,
> causing systemd to enqueue the stop of the units until after the
> transaction for decrypting is done - but then we have a deadlock
> again. Using Conflicts= here will simply tell systemd to always
> stop those services as part of the transaction to start
> decrypt.service, so nothing will deadlock.
>
> [2] One could also turn it around and NOT make decrypt.target depend
> on start-full-system.service, but the other way around (still with
> Requires=), then the ExecStartPre would not be necessary, and one
> could do systemctl start start-full-system.service... That's
> probably a matter of taste...
>
> [3] There are of course other options of overriding this; for example
> one could leave the standard target alone, just override the
> services for very few daemons and then hook those up with a new
> target. There are advantages and disadvantages with either method.
>
--
Paul E Condon
pecondon@mesanetworks.net
Reply to: