[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: systemd, headless, SSH, manual decryption (was: Re: Change systemd to not be default in Stretch)


Thanks for this. I've printed it on paper for study and reference. It
is slow going for me, but I finally have a feeling that I might have a
chance at understanding systemd. The printed version is eleven
USletter pages.  So far, I've gotten to middle of page three.

Best regards,

On 20150623_1852+0200, Christian Seiler wrote:
> On 06/23/2015 12:59 PM, Erwan David wrote:
> > Note that I use policy-rc.d to check whether the encrypted disk is
> > mounted for the daemons that need it (it allows not to change the init
> > files)
> That works? policy-rc.d should only affect invoke-rc.d, which shouldn't
> be relevant at boot, but only in maintainer scripts. (AFAIK at least.)
> > For what I need to know : I have a headless machine with an encrypted disk.
> > I cannot ask the password on console, so
> > 1) at boot I do not mount the encrypted disk, and start a minimal set
> > of daemons, among them the ssh daemon.
> > 
> > 2) I ssh to the machine then mount encrypted disk and start remaining
> > daemons.
> > 
> > How can I do this with systemd ?
> This is a great question because it presents a nice little problem that
> covers quite a few of topics regarding systemd. I've sat down and
> solved your little problem from a systemd perspective, and hopefully my
> solution will help you in understanding how systemd works.
> First of all a couple of very basic explanations: what is a unit? A
> unit is anything that systemd manages and/or monitors. Relevant for
> our case are the types service, target and mount, but systemd supports
> more than that (see man systemd.unit and the references therein for
> more details).
> A service unit is probably the simplest to understand, it corresponds
> to that which was previously provided by /etc/init.d scripts. I'm not
> going to go into much detail here about that.
> A mount unit represents a mount in the system. If you manually mount
> something via the mount(8) command, systemd will not interfere, but it
> will synthesize a dynamic mount unit for it. So for example, if you
> have:
> mkdir /mnt/a /mnt/b
> mount --bind /mnt/a /mnt/b
> Then you can also umount /mnt/b via the command:
> systemctl stop mnt-b.mount
> (umount /mnt/b will continue to work, of course)
> For manual mounts this is not really that relevant (I suspect even most
> systemd developers will use plain old umount in that case), but mount
> units are the method systemd uses to handle /etc/fstab: for every entry
> in /etc/fstab a mount unit is generated, that's how /etc/fstab is
> integrated into the boot process. (See below for further details.)
> A target is in some sense the simplest kind of unit: it can only have
> dependencies (and a description), but nothing else. You can achieve the
> same thing with a dummy /bin/true service unit. Targets are useful for
> grouping things together and also for providing synchronization points.
> For example, as is documented in man 7 bootup, there's a target called
> local-fs.target that has the semantics that every local filesystem
> mount in /etc/fstab will be ordered before it, and so that every
> service that orders after it can be sure that local filesystems are
> already mounted at that point. (Most services are implicitly ordered
> after local-fs.target because by default everything is ordered after
> basic.target, which itself is ordered after local-fs.target. You can
> override these kind of things if you need to, however.)
> How does systemd boot up a system? After doing some very basic
> initialization (such as setting the hostname from /etc/hostname and
> mounting some essential kernel filesystems such as /proc and /sys), it
> will try to start a specific unit, either the default (called
> 'default.target') or any unit specified on the kernel command line via
> the option systemd.unit=XXX. default.target is a symlink to
> graphical.target on Debian, but that can be overridden. (If you don't
> have a GUI installed, graphical.target is equivalent to
> multi-user.target, in analogy to the runlevels 2-5 on Debian with
> sysvinit.) graphical.target itself depends on a lot of things, and thus
> all services required for system startup are automatically pulled in.
> Now how does one solve the problem you have? There are multiple ways to
> do so, but for the sake of simplicity I'm going to show just one, the
> one I personally prefer (YMMV).
> The basic outline would be this:
>  - tell systemd to NOT automatically decrypt the drive at boot and to
>    NOT automatically mount it
>  - create a new target unit that will serve as a new boot target, and
>    it will only contain ssh + syslog as services (beyond the very basic
>    early-boot things that should always be there)
>  - create a second target unit that will pull in the encrypted drive
>    and the mount and then ulitmately start the original
>    multi-user.target, thus pulling in all other installed daemons on
>    the system - that second target will be started manually by you
>    after you log in and decrypt the drive
> Let's begin. First of all you need to add your drive to /etc/crypttab.
> I've done this in a KVM and the entry looks like this:
> crypto /dev/vda5 none luks,noauto
> (This means that /dev/vda5 is the device with the encrypted LUKS data
> on it and /dev/mapper/crypto will be the name of the virtual device
> with the plain text data.)
> Note that here there is a 'noauto' setting, which means that the drive
> should NOT be decrypted automatically at boot (otherwise systemd would
> wait for you to enter the password on the console before even mounting
> the local filesystems, so it would hang...).
> What happens internally here is that systemd uses a so-called generator
> to dynamically transform the contents /etc/crypttab into systemd units.
> In this case, it generates a service for each line, in our case it will
> generate systemd-cryptsetup@crypto.service. If we had not put noauto
> here, the service would be started automatically at boot, asking you
> for your password at the console. But with 'noauto' here, the service
> will be synthesized by the generator, but not started automatically.
> You can now see if systemd picks up on that crypttab entry be running
> systemctl daemon-reload
> That does the following:
>   - re-runs all generators (for reading /etc/crypttab, /etc/fstab, ...)
>   - re-reads all static and generated unit files on the disk
>   => so every time you change something, you need to run the above
>      command for systemd to pick it up
> You can see that the unit was detected by systemd via:
> systemctl status systemd-cryptsetup@crypto.service
> But it should also tell you it's inactive (i.e. hasn't been started so
> far).
> Now we add the entry for the filesystem in /etc/fstab:
> /dev/mapper/crypto /srv ext4 rw,noauto 0 0
> Here we also have a noauto setting to make sure it's not automatically
> mounted at boot. This is really important here, since this is a local
> filesystem and systemd considers any local filesystem that couldn't be
> mounted at boot to be a fatal error. So if you leave out the noauto
> here, systemd will wait 90s by default for the device to appear at boot
> and (since it won't appear) will stop the boot process and start an
> emergency shell instead. With a headless system that's a really bad
> thing to happen, which is why you should definitely put 'noauto' here.
> Then systemd won't automatically try to mount it at boot at all.
> Now that that's done, let's create our target unit for booting the
> system with just a minimal set of daemons. We don't want to interfere
> with the early-boot process, so we only want to replace the default
> graphical.target, but nothing else.
> When editing static units, you should know that systemd knows two
> directories where those may be stored: /lib/systemd/system and
> /etc/systemd/system. /lib/systemd/system is for things that have been
> packaged, whereas /etc/systemd/system is the territory of the
> administrator. If files with the same name exist in both directories,
> the file in /etc/systemd/system completely overrides the file in
> /lib/systemd/system.
> So in this case, we want to edit things in /etc/systemd/system. Let's
> create a new file /etc/systemd/system/before-decrypt.target. That file
> is supposed to be our new default target, so let's put in the
> following:
> [Unit]
> Description=System before Decryption
> Requires=basic.target
> Conflicts=rescue.service rescue.target
> After=basic.target rescue.service rescue.target
> AllowIsolate=yes
> The contents (apart from Description= and Documentation=) is identical
> to the default /lib/systemd/system/multi-user.target. (graphical.target
> just pulls that in and since we have a headless system, let's skip the
> extra indirection.)
> To understand this file, we need to talk systemd dependencies for a
> moment. There are a lot of different types of dependencies in systemd,
> but in this case you need to understand only a couple (those are also
> the most important ones):
>  - Requires: if unit A requires the unit B, it means that if A is to
>    be started, B also has to be started (and if B fails to start, A
>    will not start either). Also, if B is to be stopped, A will also
>    be stopped. (Note that this only affects explicit actions, so A
>    would not be stopped if B segfaults and just dies unexpectedly,
>    unless you tell systemd to do that with an additional setting.)
>  - Wants: if unit A wants the unit B, it means that if A is to be
>    started, B also has to be started; but A will be started regardless
>    of whether A fails to start or not (or even exists or not) - and
>    A will not be stopped if B is stopped. It's a weaker version of
>    Requires
>  - Conflicts: if unit B conflicts with unit B, unit B will be stopped
>    if unit A is to be started and vice-versa
>  - Before/After: declares ordering within a single transaction. Note
>    that ordering is orthogonal in systemd: Requires/Wants do NOT imply
>    ordering dependencies, those have to be made explicitly. (Note that
>    for targets Requires/Wants do sometimes result in implicit ordering,
>    see below for details; but all other unit types don't have that
>    logic, so service A requiring service B will NOT lead to an ordering
>    between them, unless you also specify that explicitly.)
> So in the case of our before-decrypt.target, we have the following
> dependencies:
>   - Requires=basic.target
>     This means that the basic system setup should be pulled in. This
>     is essential; if you leave that out the most basic startup things
>     will not be started if you try to boot into before-decrypt.target.
>   - Conflicts=rescue.service rescue.target
>     If you were dropped in an emergency shell at boot (or requested it
>     explicitly over the command line), this conflict will tell systemd
>     that once you do continue booting from the emergency shell, it
>     should stop the emergency shell automatically.
>   - After=basic.target rescue.service rescue.target
>     Make sure that before-decrypt.target is started only after the very
>     basic system initialization is complete.
> But wait a minute: we copied this unit from multi-user.target, where
> all the non-GUI services should all be running... Where are they? This
> unit file appears to only pull in basic.target and nothing else.
> Well, since you don't want to modify a unit file every time you install
> a new daemon on your system, systemd provides ways to add additional
> dependencies to units dynamically via symlinks.
> There are two directories for multi-user.target:
> /lib/systemd/system/multi-user.target.wants
> /etc/systemd/system/multi-user.target.wants
> Each directory contains symlinks to unit files that are automatically
> added as Wants= type dependencies to the target (you can also have
> .requires). The directory in /lib contains all native systemd services
> that shouldn't be disabled by the administrator (and systemctl disable
> won't work on them although you could manually do something), while the
> latter contains all service that can easily be disabled by the
> administrator (those include SSH, cron, MTAs, web servers, ...).
> Typically one wants to still enable all units from the /lib directory
> also in our target, so let's do the following:
> mkdir /etc/systemd/system/before-decrypt.target.wants
> cd /etc/systemd/system/before-decrypt.target.wants
> for i in /lib/systemd/system/multi-user.target.wants/* ; do
>    ln -s /lib/systemd/system/$(basename $i) .
> done
> Then we additionally want to enable SSH and syslog (but no other
> services):
> cd /etc/systemd/system/before-decrypt.target.wants
> ln -s /etc/systemd/system/syslog.service .
> ln -s /lib/systemd/system/ssh.service .
> A couple of notes:
>  - we take syslog.service from /etc because I don't know what your
>    favorite syslog daemon is and syslog.service is always a symlink
>    to the current one (although you may have to force it manually
>    because in Jessie there's a Debian packaging bug if you change
>    syslog daemons that the symlink doesn't get updated)
>  - both syslog and ssh are native services, which is why that works
>  - you can also enable additional services you might need in the same
>    way. There's no cron nor atd started here (because those might
>    require the encrypted partition to be mounted - or not, depending
>    on your system) and also no MTA
>  - note that these symlinks only work if you have native systemd
>    service files. You can also enable init scripts, but then you have
>    to explicitly specify Wants=[INITSCRIPTNAME].service, e.g.
>    Wants=exim4.service - since the service files for init scripts are
>    dynamically generated and you thus can't easily just create a simple
>    symlink (in the default case they are activated because systemd's
>    generator looks for /etc/rcX.d/S**INITSCRIPTNAME symlinks and then
>    creates Wants= dependencies for runlevelX.target dynamically - and
>    all runlevel[2-5].target are symlinks to graphical.target)
> Then we should be nearly ready for a test boot to see if our minimal
> system boots. To summarize what we have done so far:
>  - modify /etc/crypttab and /etc/fstab to add all required entries with
>    the 'noauto' option
>  - create /etc/systemd/system/before-decrypt.target (see above for
>    the contents)
>  - create directory /etc/systemd/system/before-decrypt.target.wants and
>    symlink all services in multi-user.target that we need before the
>    partition is decrypted
> As a final step we can now make our new target the default boot target:
> systemctl set-default before-decrypt.target
> Now let's reboot the system. It will come up again and hopefully the
> following will have happened:
>  - it didn't hang at boot (otherwise check noauto options)
>  - SSH was started
>  - other system daemons that were not included in the list we activated
>    were NOT started (e.g. no cron/atd/exim4)
> Now for the final step: we want to be able to decrypt the partition and
> enter the password. So let's create a file
> /etc/systemd/system/decrypt.target that contains the following:
> [Unit]
> Description=Decrypted System
> Requires=before-decrypt.target
> After=before-decrypt.target
> Conflicts=systemd-ask-password-console.path systemd-ask-password-console.service systemd-ask-password-plymouth.path systemd-ask-password-plymouth.service
> Requires=systemd-cryptsetup@crypto.service srv.mount start-full-system.service
> The interesting settings in that file are:
>  - Conflicts= lots of ask-password stuff:
>    This is to work around a quirk in systemd's handling of asking
>    passwords for encrypted partitions. See footnote [1] for an
>    explanation if you are really interested.
>  - Requires=...: this is the meat of the matter. Here we make sure we
>    pull in
>       - the service that decrypts the partition
>       - the mount for /srv
>       - and an additional service (see below) that will cause the rest
>         of the daemons to start (start-full-system.service)
> This means that when one then starts this target, it will cause the
> partition to be decrypted, the filesystem to be mounted and then all
> daemons to be started.
> Let's look at start-full-system.service:
> [Unit]
> Description=Start full system after decryption
> After=decrypt.target
> [Service]
> Type=oneshot
> ExecStartPre=/bin/systemctl is-active --quiet decrypt.target
> ExecStart=/bin/systemctl --no-block start multi-user.target
> The unit calls systemctl to start multi-user.target. The reason this
> is done is because of ordering: we only want to start daemons once the
> filesystem is mounted. But we don't want to add an explicit After=
> dependency to each daemon (or list every daemon as a Before= depdency
> here), so instead of pulling in multi-user.target directly in
> decrypt.target and adding all those dependencies, we simply create a
> small service here that is ordered AFTER decrypt.target that causes
> systemd to start multi-user.target.
> A word on service ordering: unless you set DefaultDependencies=no, a
> target will always be ordered AFTER all of its other dependencies
> (Requires=, Wants=, Conflicts=, ...). However, if a service already has
> an explicit ordering dependency, this will not occur. So what we have
> here is the following ordering:
> decrypt.target has an implicit
>   After=systemd-cryptsetup@crypto.service srv.mount
> BUT because start-full-system.service is ordered relative to
> decrypt.mount explicitly already, it doesn't get an implicit ordering.
> So that means we have the following order w.r.t. those 4 units:
>    systemd-cryptsetup@crypto.service             srv.mount
>                   |                                  |
>                   \-------------------+--------------/
>                                       |
>                                       v
>                                 decrypt.target
>                                       |
>                                       v
>                             start-full-system.service
> (In principle, one could also order srv.mount after the cryptsetup
> service, but that is unnecessary, since mount units in systemd will
> always wait for the corresponding devices to appear, so an explicit
> ordering is not required here.)
> If you are wondering what the ExecStartPre= is for: it's to make sure
> that decrypt.target was successful before it continues. Otherwise, if
> decrypting and/or mounting fails, it would still be executed, so add
> an additional check that it only starts if decrypting was
> successful.[2]
> As a final thing w.r.t. start-full-system.service, let's talk about
> the --no-block here: it causes systemctl to enqueue the start job for
> multi-user.target but will return immediately. The reason for that is
> that systemd serializes transactions to a certain extent, so that not
> using --no-block here would cause the system to deadlock because
> systemctl would wait for the multi-user.target transaction to finish
> but systemd wouldn't begin the transaction until the transaction
> containing start-full-system.service would be finished. (It's generally
> a very bad idea to call systemctl without --no-block in service files.)
> However, this means that the starting of the daemons will be initiated
> by starting decrypt.target, but it will be done in the background.
> Once this is done, let's do a systemctl daemon-reload to make sure
> systemd has picked up the changes and now we can try to see if it
> works:
> systemctl start decrypt.target
> You will now be asked for the encrypted partition's password. Once
> you've entered that, the filesystem will be mounted, systemctl start
> will return you to the command line and in the background all daemons
> will be started automatically.
> You can reboot the system, log in per SSH again, and use the command
> again - and it should work.
> Now wait a minute: how does this password prompt work exactly?
>  - systemd-cryptsetup@XXX.service will tell systemd that it needs a
>    password to decrypt the drive
>  - systemd has to ways of getting to such a passowrd: either via a
>    so-called 'ask-password agent' (typically used at boot or if the
>    administrator plugs in a known encrypted drive dynamically) - or,
>    if the password is asked as part of a transaction that was triggered
>    via systemct, systemctl will ask the password directly.
> So this means that simply telling systemd to decrypt the drive will
> then lead to a direct password prompt at that point. So in the end, one
> needs only to enter one single command, then enter the password.
> Everything will be decrypted and all services will be started then.
> Neat, huh?
> Summary of what we did:
>  - adjust /etc/crypttab and /etc/fstab : use noauto!
>  - create /etc/systemd/system/before-decrypt.target
>    [Unit]
>    Description=System before Decryption
>    Requires=basic.target
>    Conflicts=rescue.service rescue.target
>    After=basic.target rescue.service rescue.target
>    AllowIsolate=yes
>  - enable required services for before-decrypt.target
>    mkdir /etc/systemd/system/before-decrypt.target.wants
>    cd /etc/systemd/system/before-decrypt.target.wants
>    for i in /lib/systemd/system/multi-user.target.wants/* ; do
>       ln -s /lib/systemd/system/$(basename $i) .
>    done
>    ln -s /etc/systemd/system/syslog.service .
>    ln -s /lib/systemd/system/ssh.service .
>  - create /etc/systemd/system/decrypt.target
>    [Unit]
>    Description=Decrypted System
>    Requires=before-decrypt.target
>    After=before-decrypt.target
>    Conflicts=systemd-ask-password-console.path systemd-ask-password-console.service systemd-ask-password-plymouth.path systemd-ask-password-plymouth.service
>    Requires=systemd-cryptsetup@crypto.service srv.mount start-full-system.service
>  - create /etc/systemd/system/start-full-system.service
>    [Unit]
>    Description=Start full system after decryption
>    After=decrypt.target
>    [Service]
>    Type=oneshot
>    ExecStartPre=/bin/systemctl is-active --quiet decrypt.target
>    ExecStart=/bin/systemctl --no-block start multi-user.target
>  - reboot
>  - to decrypt: systemctl start decrypt.target, enter your password at
>    the prompt
> A couple of comments:
>  1. This can be extended to multiple encrypted drives and also multiple
>     mounts. If you want to see what unit name systemd wants to use for
>     a given mount point (so you know what to set as Requires= in
>     decrypt.target), you can use 'systemctl status /mount/point' to see
>     how systemd mangles that into a unit name
>  2. If you install additional daemons, Debian will hook them up in
>     multi-user.target (either directly if they are native or indirectly
>     if they are sysv scripts), so if you install something that should
>     already be started before decryption, you need to manually add it.
>     On the other hand, since we don't modify multi-user.target itself,
>     any new daemon will always be hooked up there automatically, so
>     that if you install something, it will automatically be part of the
>     start-only-when-decrypted logic.[3]
>  3. If you have systemd installed, it's recommended (but not strictly
>     required) to have dbus installed. If you have, note that dbus is
>     the only service (other than systemd's internal ones and some
>     plymouth stuff that's not relevant for headless systems) that hooks
>     up in /lib/systemd/system/multi-user.target.wants (and not in
>     /etc, as all other services do!), and if it's installed it should
>     probably be activated already for the minimal system, so if you
>     install dbus only after copying all the symlinks to
>     before-decrypt.target.wants (see above), then you should copy that
>     one manually.
>  4. If you storage stack is a bit more complicated (e.g. you use LVM
>     on top of the encryption), you may or may not need to manually add
>     some services to activate that once the drive is decrypted. It's
>     definitely possible, and might even work out of the box (I haven't
>     tried it), but you may have to play around a bit there.
>  5. If you truly have a headless system, you should test this in a VM
>     first, especially when it comes to the boot process.
> So there it is: how to solve your particular problem with systemd on
> Jessie. I hope I could use this use-case to illuminate a couple of
> concepts in systemd and give you enough information so that you have a
> good starting point into the topic.
> Regards,
> Christian
> [1] The Conflicts= is required since systemd's own service to notify
>     root of password prompts via wall(1) has the following command:
>     /bin/systemctl stop [... all units listed in Conflicts= here ...]
>     This is fine because it's typically started dynamically when the
>     administrator plugis in a drive dynamically or so (and is not
>     active during boot, there the other services are active, which
>     in turn don't really work after boot, that's why they are stopped).
>     The problem here is that we start this from within a transaction,
>     causing systemd to enqueue the stop of the units until after the
>     transaction for decrypting is done - but then we have a deadlock
>     again. Using Conflicts= here will simply tell systemd to always 
>     stop those services as part of the transaction to start
>     decrypt.service, so nothing will deadlock.
> [2] One could also turn it around and NOT make decrypt.target depend
>     on start-full-system.service, but the other way around (still with
>     Requires=), then the ExecStartPre would not be necessary, and one
>     could do systemctl start start-full-system.service... That's
>     probably a matter of taste...
> [3] There are of course other options of overriding this; for example
>     one could leave the standard target alone, just override the
>     services for very few daemons and then hook those up with a new
>     target. There are advantages and disadvantages with either method.

Paul E Condon           

Reply to: