On 06/23/2015 12:59 PM, Erwan David wrote:
> Note that I use policy-rc.d to check whether the encrypted disk is
> mounted for the daemons that need it (it allows not to change the init
> files)
That works? policy-rc.d should only affect invoke-rc.d, which shouldn't
be relevant at boot, but only in maintainer scripts. (AFAIK at least.)
> For what I need to know : I have a headless machine with an encrypted disk.
> I cannot ask the password on console, so
> 1) at boot I do not mount the encrypted disk, and start a minimal set
> of daemons, among them the ssh daemon.
>
> 2) I ssh to the machine then mount encrypted disk and start remaining
> daemons.
>
> How can I do this with systemd ?
This is a great question because it presents a nice little problem that
covers quite a few of topics regarding systemd. I've sat down and
solved your little problem from a systemd perspective, and hopefully my
solution will help you in understanding how systemd works.
First of all a couple of very basic explanations: what is a unit? A
unit is anything that systemd manages and/or monitors. Relevant for
our case are the types service, target and mount, but systemd supports
more than that (see man systemd.unit and the references therein for
more details).
A service unit is probably the simplest to understand, it corresponds
to that which was previously provided by /etc/init.d scripts. I'm not
going to go into much detail here about that.
A mount unit represents a mount in the system. If you manually mount
something via the mount(8) command, systemd will not interfere, but it
will synthesize a dynamic mount unit for it. So for example, if you
have:
mkdir /mnt/a /mnt/b
mount --bind /mnt/a /mnt/b
Then you can also umount /mnt/b via the command:
systemctl stop mnt-b.mount
(umount /mnt/b will continue to work, of course)
For manual mounts this is not really that relevant (I suspect even most
systemd developers will use plain old umount in that case), but mount
units are the method systemd uses to handle /etc/fstab: for every entry
in /etc/fstab a mount unit is generated, that's how /etc/fstab is
integrated into the boot process. (See below for further details.)
A target is in some sense the simplest kind of unit: it can only have
dependencies (and a description), but nothing else. You can achieve the
same thing with a dummy /bin/true service unit. Targets are useful for
grouping things together and also for providing synchronization points.
For example, as is documented in man 7 bootup, there's a target called
local-fs.target that has the semantics that every local filesystem
mount in /etc/fstab will be ordered before it, and so that every
service that orders after it can be sure that local filesystems are
already mounted at that point. (Most services are implicitly ordered
after local-fs.target because by default everything is ordered after
basic.target, which itself is ordered after local-fs.target. You can
override these kind of things if you need to, however.)
How does systemd boot up a system? After doing some very basic
initialization (such as setting the hostname from /etc/hostname and
mounting some essential kernel filesystems such as /proc and /sys), it
will try to start a specific unit, either the default (called
'default.target') or any unit specified on the kernel command line via
the option systemd.unit=XXX. default.target is a symlink to
graphical.target on Debian, but that can be overridden. (If you don't
have a GUI installed, graphical.target is equivalent to
multi-user.target, in analogy to the runlevels 2-5 on Debian with
sysvinit.) graphical.target itself depends on a lot of things, and thus
all services required for system startup are automatically pulled in.
Now how does one solve the problem you have? There are multiple ways to
do so, but for the sake of simplicity I'm going to show just one, the
one I personally prefer (YMMV).
The basic outline would be this:
- tell systemd to NOT automatically decrypt the drive at boot and to
NOT automatically mount it
- create a new target unit that will serve as a new boot target, and
it will only contain ssh + syslog as services (beyond the very basic
early-boot things that should always be there)
- create a second target unit that will pull in the encrypted drive
and the mount and then ulitmately start the original
multi-user.target, thus pulling in all other installed daemons on
the system - that second target will be started manually by you
after you log in and decrypt the drive
Let's begin. First of all you need to add your drive to /etc/crypttab.
I've done this in a KVM and the entry looks like this:
crypto /dev/vda5 none luks,noauto
(This means that /dev/vda5 is the device with the encrypted LUKS data
on it and /dev/mapper/crypto will be the name of the virtual device
with the plain text data.)
Note that here there is a 'noauto' setting, which means that the drive
should NOT be decrypted automatically at boot (otherwise systemd would
wait for you to enter the password on the console before even mounting
the local filesystems, so it would hang...).
What happens internally here is that systemd uses a so-called generator
to dynamically transform the contents /etc/crypttab into systemd units.
In this case, it generates a service for each line, in our case it will
generate systemd-cryptsetup@crypto.service. If we had not put noauto
here, the service would be started automatically at boot, asking you
for your password at the console. But with 'noauto' here, the service
will be synthesized by the generator, but not started automatically.
You can now see if systemd picks up on that crypttab entry be running
systemctl daemon-reload
That does the following:
- re-runs all generators (for reading /etc/crypttab, /etc/fstab, ...)
- re-reads all static and generated unit files on the disk
=> so every time you change something, you need to run the above
command for systemd to pick it up
You can see that the unit was detected by systemd via:
systemctl status systemd-cryptsetup@crypto.service
But it should also tell you it's inactive (i.e. hasn't been started so
far).
Now we add the entry for the filesystem in /etc/fstab:
/dev/mapper/crypto /srv ext4 rw,noauto 0 0
Here we also have a noauto setting to make sure it's not automatically
mounted at boot. This is really important here, since this is a local
filesystem and systemd considers any local filesystem that couldn't be
mounted at boot to be a fatal error. So if you leave out the noauto
here, systemd will wait 90s by default for the device to appear at boot
and (since it won't appear) will stop the boot process and start an
emergency shell instead. With a headless system that's a really bad
thing to happen, which is why you should definitely put 'noauto' here.
Then systemd won't automatically try to mount it at boot at all.
Now that that's done, let's create our target unit for booting the
system with just a minimal set of daemons. We don't want to interfere
with the early-boot process, so we only want to replace the default
graphical.target, but nothing else.
When editing static units, you should know that systemd knows two
directories where those may be stored: /lib/systemd/system and
/etc/systemd/system. /lib/systemd/system is for things that have been
packaged, whereas /etc/systemd/system is the territory of the
administrator. If files with the same name exist in both directories,
the file in /etc/systemd/system completely overrides the file in
/lib/systemd/system.
So in this case, we want to edit things in /etc/systemd/system. Let's
create a new file /etc/systemd/system/before-decrypt.target. That file
is supposed to be our new default target, so let's put in the
following:
[Unit]
Description=System before Decryption
Requires=basic.target
Conflicts=rescue.service rescue.target
After=basic.target rescue.service rescue.target
AllowIsolate=yes
The contents (apart from Description= and Documentation=) is identical
to the default /lib/systemd/system/multi-user.target. (graphical.target
just pulls that in and since we have a headless system, let's skip the
extra indirection.)
To understand this file, we need to talk systemd dependencies for a
moment. There are a lot of different types of dependencies in systemd,
but in this case you need to understand only a couple (those are also
the most important ones):
- Requires: if unit A requires the unit B, it means that if A is to
be started, B also has to be started (and if B fails to start, A
will not start either). Also, if B is to be stopped, A will also
be stopped. (Note that this only affects explicit actions, so A
would not be stopped if B segfaults and just dies unexpectedly,
unless you tell systemd to do that with an additional setting.)
- Wants: if unit A wants the unit B, it means that if A is to be
started, B also has to be started; but A will be started regardless
of whether A fails to start or not (or even exists or not) - and
A will not be stopped if B is stopped. It's a weaker version of
Requires
- Conflicts: if unit B conflicts with unit B, unit B will be stopped
if unit A is to be started and vice-versa
- Before/After: declares ordering within a single transaction. Note
that ordering is orthogonal in systemd: Requires/Wants do NOT imply
ordering dependencies, those have to be made explicitly. (Note that
for targets Requires/Wants do sometimes result in implicit ordering,
see below for details; but all other unit types don't have that
logic, so service A requiring service B will NOT lead to an ordering
between them, unless you also specify that explicitly.)
So in the case of our before-decrypt.target, we have the following
dependencies:
- Requires=basic.target
This means that the basic system setup should be pulled in. This
is essential; if you leave that out the most basic startup things
will not be started if you try to boot into before-decrypt.target.
- Conflicts=rescue.service rescue.target
If you were dropped in an emergency shell at boot (or requested it
explicitly over the command line), this conflict will tell systemd
that once you do continue booting from the emergency shell, it
should stop the emergency shell automatically.
- After=basic.target rescue.service rescue.target
Make sure that before-decrypt.target is started only after the very
basic system initialization is complete.
But wait a minute: we copied this unit from multi-user.target, where
all the non-GUI services should all be running... Where are they? This
unit file appears to only pull in basic.target and nothing else.
Well, since you don't want to modify a unit file every time you install
a new daemon on your system, systemd provides ways to add additional
dependencies to units dynamically via symlinks.
There are two directories for multi-user.target:
/lib/systemd/system/multi-user.target.wants
/etc/systemd/system/multi-user.target.wants
Each directory contains symlinks to unit files that are automatically
added as Wants= type dependencies to the target (you can also have
.requires). The directory in /lib contains all native systemd services
that shouldn't be disabled by the administrator (and systemctl disable
won't work on them although you could manually do something), while the
latter contains all service that can easily be disabled by the
administrator (those include SSH, cron, MTAs, web servers, ...).
Typically one wants to still enable all units from the /lib directory
also in our target, so let's do the following:
mkdir /etc/systemd/system/before-decrypt.target.wants
cd /etc/systemd/system/before-decrypt.target.wants
for i in /lib/systemd/system/multi-user.target.wants/* ; do
ln -s /lib/systemd/system/$(basename $i) .
done
Then we additionally want to enable SSH and syslog (but no other
services):
cd /etc/systemd/system/before-decrypt.target.wants
ln -s /etc/systemd/system/syslog.service .
ln -s /lib/systemd/system/ssh.service .
A couple of notes:
- we take syslog.service from /etc because I don't know what your
favorite syslog daemon is and syslog.service is always a symlink
to the current one (although you may have to force it manually
because in Jessie there's a Debian packaging bug if you change
syslog daemons that the symlink doesn't get updated)
- both syslog and ssh are native services, which is why that works
- you can also enable additional services you might need in the same
way. There's no cron nor atd started here (because those might
require the encrypted partition to be mounted - or not, depending
on your system) and also no MTA
- note that these symlinks only work if you have native systemd
service files. You can also enable init scripts, but then you have
to explicitly specify Wants=[INITSCRIPTNAME].service, e.g.
Wants=exim4.service - since the service files for init scripts are
dynamically generated and you thus can't easily just create a simple
symlink (in the default case they are activated because systemd's
generator looks for /etc/rcX.d/S**INITSCRIPTNAME symlinks and then
creates Wants= dependencies for runlevelX.target dynamically - and
all runlevel[2-5].target are symlinks to graphical.target)
Then we should be nearly ready for a test boot to see if our minimal
system boots. To summarize what we have done so far:
- modify /etc/crypttab and /etc/fstab to add all required entries with
the 'noauto' option
- create /etc/systemd/system/before-decrypt.target (see above for
the contents)
- create directory /etc/systemd/system/before-decrypt.target.wants and
symlink all services in multi-user.target that we need before the
partition is decrypted
As a final step we can now make our new target the default boot target:
systemctl set-default before-decrypt.target
Now let's reboot the system. It will come up again and hopefully the
following will have happened:
- it didn't hang at boot (otherwise check noauto options)
- SSH was started
- other system daemons that were not included in the list we activated
were NOT started (e.g. no cron/atd/exim4)
Now for the final step: we want to be able to decrypt the partition and
enter the password. So let's create a file
/etc/systemd/system/decrypt.target that contains the following:
[Unit]
Description=Decrypted System
Requires=before-decrypt.target
After=before-decrypt.target
Conflicts=systemd-ask-password-console.path systemd-ask-password-console.service systemd-ask-password-plymouth.path systemd-ask-password-plymouth.service
Requires=systemd-cryptsetup@crypto.service srv.mount start-full-system.service
The interesting settings in that file are:
- Conflicts= lots of ask-password stuff:
This is to work around a quirk in systemd's handling of asking
passwords for encrypted partitions. See footnote [1] for an
explanation if you are really interested.
- Requires=...: this is the meat of the matter. Here we make sure we
pull in
- the service that decrypts the partition
- the mount for /srv
- and an additional service (see below) that will cause the rest
of the daemons to start (start-full-system.service)
This means that when one then starts this target, it will cause the
partition to be decrypted, the filesystem to be mounted and then all
daemons to be started.
Let's look at start-full-system.service:
[Unit]
Description=Start full system after decryption
After=decrypt.target
[Service]
Type=oneshot
ExecStartPre=/bin/systemctl is-active --quiet decrypt.target
ExecStart=/bin/systemctl --no-block start multi-user.target
The unit calls systemctl to start multi-user.target. The reason this
is done is because of ordering: we only want to start daemons once the
filesystem is mounted. But we don't want to add an explicit After=
dependency to each daemon (or list every daemon as a Before= depdency
here), so instead of pulling in multi-user.target directly in
decrypt.target and adding all those dependencies, we simply create a
small service here that is ordered AFTER decrypt.target that causes
systemd to start multi-user.target.
A word on service ordering: unless you set DefaultDependencies=no, a
target will always be ordered AFTER all of its other dependencies
(Requires=, Wants=, Conflicts=, ...). However, if a service already has
an explicit ordering dependency, this will not occur. So what we have
here is the following ordering:
decrypt.target has an implicit
After=systemd-cryptsetup@crypto.service srv.mount
BUT because start-full-system.service is ordered relative to
decrypt.mount explicitly already, it doesn't get an implicit ordering.
So that means we have the following order w.r.t. those 4 units:
systemd-cryptsetup@crypto.service srv.mount
| |
\-------------------+--------------/
|
v
decrypt.target
|
v
start-full-system.service
(In principle, one could also order srv.mount after the cryptsetup
service, but that is unnecessary, since mount units in systemd will
always wait for the corresponding devices to appear, so an explicit
ordering is not required here.)
If you are wondering what the ExecStartPre= is for: it's to make sure
that decrypt.target was successful before it continues. Otherwise, if
decrypting and/or mounting fails, it would still be executed, so add
an additional check that it only starts if decrypting was
successful.[2]
As a final thing w.r.t. start-full-system.service, let's talk about
the --no-block here: it causes systemctl to enqueue the start job for
multi-user.target but will return immediately. The reason for that is
that systemd serializes transactions to a certain extent, so that not
using --no-block here would cause the system to deadlock because
systemctl would wait for the multi-user.target transaction to finish
but systemd wouldn't begin the transaction until the transaction
containing start-full-system.service would be finished. (It's generally
a very bad idea to call systemctl without --no-block in service files.)
However, this means that the starting of the daemons will be initiated
by starting decrypt.target, but it will be done in the background.
Once this is done, let's do a systemctl daemon-reload to make sure
systemd has picked up the changes and now we can try to see if it
works:
systemctl start decrypt.target
You will now be asked for the encrypted partition's password. Once
you've entered that, the filesystem will be mounted, systemctl start
will return you to the command line and in the background all daemons
will be started automatically.
You can reboot the system, log in per SSH again, and use the command
again - and it should work.
Now wait a minute: how does this password prompt work exactly?
- systemd-cryptsetup@XXX.service will tell systemd that it needs a
password to decrypt the drive
- systemd has to ways of getting to such a passowrd: either via a
so-called 'ask-password agent' (typically used at boot or if the
administrator plugs in a known encrypted drive dynamically) - or,
if the password is asked as part of a transaction that was triggered
via systemct, systemctl will ask the password directly.
So this means that simply telling systemd to decrypt the drive will
then lead to a direct password prompt at that point. So in the end, one
needs only to enter one single command, then enter the password.
Everything will be decrypted and all services will be started then.
Neat, huh?
Summary of what we did:
- adjust /etc/crypttab and /etc/fstab : use noauto!
- create /etc/systemd/system/before-decrypt.target
[Unit]
Description=System before Decryption
Requires=basic.target
Conflicts=rescue.service rescue.target
After=basic.target rescue.service rescue.target
AllowIsolate=yes
- enable required services for before-decrypt.target
mkdir /etc/systemd/system/before-decrypt.target.wants
cd /etc/systemd/system/before-decrypt.target.wants
for i in /lib/systemd/system/multi-user.target.wants/* ; do
ln -s /lib/systemd/system/$(basename $i) .
done
ln -s /etc/systemd/system/syslog.service .
ln -s /lib/systemd/system/ssh.service .
- create /etc/systemd/system/decrypt.target
[Unit]
Description=Decrypted System
Requires=before-decrypt.target
After=before-decrypt.target
Conflicts=systemd-ask-password-console.path systemd-ask-password-console.service systemd-ask-password-plymouth.path systemd-ask-password-plymouth.service
Requires=systemd-cryptsetup@crypto.service srv.mount start-full-system.service
- create /etc/systemd/system/start-full-system.service
[Unit]
Description=Start full system after decryption
After=decrypt.target
[Service]
Type=oneshot
ExecStartPre=/bin/systemctl is-active --quiet decrypt.target
ExecStart=/bin/systemctl --no-block start multi-user.target
- reboot
- to decrypt: systemctl start decrypt.target, enter your password at
the prompt
A couple of comments:
1. This can be extended to multiple encrypted drives and also multiple
mounts. If you want to see what unit name systemd wants to use for
a given mount point (so you know what to set as Requires= in
decrypt.target), you can use 'systemctl status /mount/point' to see
how systemd mangles that into a unit name
2. If you install additional daemons, Debian will hook them up in
multi-user.target (either directly if they are native or indirectly
if they are sysv scripts), so if you install something that should
already be started before decryption, you need to manually add it.
On the other hand, since we don't modify multi-user.target itself,
any new daemon will always be hooked up there automatically, so
that if you install something, it will automatically be part of the
start-only-when-decrypted logic.[3]
3. If you have systemd installed, it's recommended (but not strictly
required) to have dbus installed. If you have, note that dbus is
the only service (other than systemd's internal ones and some
plymouth stuff that's not relevant for headless systems) that hooks
up in /lib/systemd/system/multi-user.target.wants (and not in
/etc, as all other services do!), and if it's installed it should
probably be activated already for the minimal system, so if you
install dbus only after copying all the symlinks to
before-decrypt.target.wants (see above), then you should copy that
one manually.
4. If you storage stack is a bit more complicated (e.g. you use LVM
on top of the encryption), you may or may not need to manually add
some services to activate that once the drive is decrypted. It's
definitely possible, and might even work out of the box (I haven't
tried it), but you may have to play around a bit there.
5. If you truly have a headless system, you should test this in a VM
first, especially when it comes to the boot process.
So there it is: how to solve your particular problem with systemd on
Jessie. I hope I could use this use-case to illuminate a couple of
concepts in systemd and give you enough information so that you have a
good starting point into the topic.
Regards,
Christian
[1] The Conflicts= is required since systemd's own service to notify
root of password prompts via wall(1) has the following command:
/bin/systemctl stop [... all units listed in Conflicts= here ...]
This is fine because it's typically started dynamically when the
administrator plugis in a drive dynamically or so (and is not
active during boot, there the other services are active, which
in turn don't really work after boot, that's why they are stopped).
The problem here is that we start this from within a transaction,
causing systemd to enqueue the stop of the units until after the
transaction for decrypting is done - but then we have a deadlock
again. Using Conflicts= here will simply tell systemd to always
stop those services as part of the transaction to start
decrypt.service, so nothing will deadlock.
[2] One could also turn it around and NOT make decrypt.target depend
on start-full-system.service, but the other way around (still with
Requires=), then the ExecStartPre would not be necessary, and one
could do systemctl start start-full-system.service... That's
probably a matter of taste...
[3] There are of course other options of overriding this; for example
one could leave the standard target alone, just override the
services for very few daemons and then hook those up with a new
target. There are advantages and disadvantages with either method.
Attachment:
signature.asc
Description: OpenPGP digital signature