On 06/23/2015 12:59 PM, Erwan David wrote: > Note that I use policy-rc.d to check whether the encrypted disk is > mounted for the daemons that need it (it allows not to change the init > files) That works? policy-rc.d should only affect invoke-rc.d, which shouldn't be relevant at boot, but only in maintainer scripts. (AFAIK at least.) > For what I need to know : I have a headless machine with an encrypted disk. > I cannot ask the password on console, so > 1) at boot I do not mount the encrypted disk, and start a minimal set > of daemons, among them the ssh daemon. > > 2) I ssh to the machine then mount encrypted disk and start remaining > daemons. > > How can I do this with systemd ? This is a great question because it presents a nice little problem that covers quite a few of topics regarding systemd. I've sat down and solved your little problem from a systemd perspective, and hopefully my solution will help you in understanding how systemd works. First of all a couple of very basic explanations: what is a unit? A unit is anything that systemd manages and/or monitors. Relevant for our case are the types service, target and mount, but systemd supports more than that (see man systemd.unit and the references therein for more details). A service unit is probably the simplest to understand, it corresponds to that which was previously provided by /etc/init.d scripts. I'm not going to go into much detail here about that. A mount unit represents a mount in the system. If you manually mount something via the mount(8) command, systemd will not interfere, but it will synthesize a dynamic mount unit for it. So for example, if you have: mkdir /mnt/a /mnt/b mount --bind /mnt/a /mnt/b Then you can also umount /mnt/b via the command: systemctl stop mnt-b.mount (umount /mnt/b will continue to work, of course) For manual mounts this is not really that relevant (I suspect even most systemd developers will use plain old umount in that case), but mount units are the method systemd uses to handle /etc/fstab: for every entry in /etc/fstab a mount unit is generated, that's how /etc/fstab is integrated into the boot process. (See below for further details.) A target is in some sense the simplest kind of unit: it can only have dependencies (and a description), but nothing else. You can achieve the same thing with a dummy /bin/true service unit. Targets are useful for grouping things together and also for providing synchronization points. For example, as is documented in man 7 bootup, there's a target called local-fs.target that has the semantics that every local filesystem mount in /etc/fstab will be ordered before it, and so that every service that orders after it can be sure that local filesystems are already mounted at that point. (Most services are implicitly ordered after local-fs.target because by default everything is ordered after basic.target, which itself is ordered after local-fs.target. You can override these kind of things if you need to, however.) How does systemd boot up a system? After doing some very basic initialization (such as setting the hostname from /etc/hostname and mounting some essential kernel filesystems such as /proc and /sys), it will try to start a specific unit, either the default (called 'default.target') or any unit specified on the kernel command line via the option systemd.unit=XXX. default.target is a symlink to graphical.target on Debian, but that can be overridden. (If you don't have a GUI installed, graphical.target is equivalent to multi-user.target, in analogy to the runlevels 2-5 on Debian with sysvinit.) graphical.target itself depends on a lot of things, and thus all services required for system startup are automatically pulled in. Now how does one solve the problem you have? There are multiple ways to do so, but for the sake of simplicity I'm going to show just one, the one I personally prefer (YMMV). The basic outline would be this: - tell systemd to NOT automatically decrypt the drive at boot and to NOT automatically mount it - create a new target unit that will serve as a new boot target, and it will only contain ssh + syslog as services (beyond the very basic early-boot things that should always be there) - create a second target unit that will pull in the encrypted drive and the mount and then ulitmately start the original multi-user.target, thus pulling in all other installed daemons on the system - that second target will be started manually by you after you log in and decrypt the drive Let's begin. First of all you need to add your drive to /etc/crypttab. I've done this in a KVM and the entry looks like this: crypto /dev/vda5 none luks,noauto (This means that /dev/vda5 is the device with the encrypted LUKS data on it and /dev/mapper/crypto will be the name of the virtual device with the plain text data.) Note that here there is a 'noauto' setting, which means that the drive should NOT be decrypted automatically at boot (otherwise systemd would wait for you to enter the password on the console before even mounting the local filesystems, so it would hang...). What happens internally here is that systemd uses a so-called generator to dynamically transform the contents /etc/crypttab into systemd units. In this case, it generates a service for each line, in our case it will generate email@example.com. If we had not put noauto here, the service would be started automatically at boot, asking you for your password at the console. But with 'noauto' here, the service will be synthesized by the generator, but not started automatically. You can now see if systemd picks up on that crypttab entry be running systemctl daemon-reload That does the following: - re-runs all generators (for reading /etc/crypttab, /etc/fstab, ...) - re-reads all static and generated unit files on the disk => so every time you change something, you need to run the above command for systemd to pick it up You can see that the unit was detected by systemd via: systemctl status firstname.lastname@example.org But it should also tell you it's inactive (i.e. hasn't been started so far). Now we add the entry for the filesystem in /etc/fstab: /dev/mapper/crypto /srv ext4 rw,noauto 0 0 Here we also have a noauto setting to make sure it's not automatically mounted at boot. This is really important here, since this is a local filesystem and systemd considers any local filesystem that couldn't be mounted at boot to be a fatal error. So if you leave out the noauto here, systemd will wait 90s by default for the device to appear at boot and (since it won't appear) will stop the boot process and start an emergency shell instead. With a headless system that's a really bad thing to happen, which is why you should definitely put 'noauto' here. Then systemd won't automatically try to mount it at boot at all. Now that that's done, let's create our target unit for booting the system with just a minimal set of daemons. We don't want to interfere with the early-boot process, so we only want to replace the default graphical.target, but nothing else. When editing static units, you should know that systemd knows two directories where those may be stored: /lib/systemd/system and /etc/systemd/system. /lib/systemd/system is for things that have been packaged, whereas /etc/systemd/system is the territory of the administrator. If files with the same name exist in both directories, the file in /etc/systemd/system completely overrides the file in /lib/systemd/system. So in this case, we want to edit things in /etc/systemd/system. Let's create a new file /etc/systemd/system/before-decrypt.target. That file is supposed to be our new default target, so let's put in the following: [Unit] Description=System before Decryption Requires=basic.target Conflicts=rescue.service rescue.target After=basic.target rescue.service rescue.target AllowIsolate=yes The contents (apart from Description= and Documentation=) is identical to the default /lib/systemd/system/multi-user.target. (graphical.target just pulls that in and since we have a headless system, let's skip the extra indirection.) To understand this file, we need to talk systemd dependencies for a moment. There are a lot of different types of dependencies in systemd, but in this case you need to understand only a couple (those are also the most important ones): - Requires: if unit A requires the unit B, it means that if A is to be started, B also has to be started (and if B fails to start, A will not start either). Also, if B is to be stopped, A will also be stopped. (Note that this only affects explicit actions, so A would not be stopped if B segfaults and just dies unexpectedly, unless you tell systemd to do that with an additional setting.) - Wants: if unit A wants the unit B, it means that if A is to be started, B also has to be started; but A will be started regardless of whether A fails to start or not (or even exists or not) - and A will not be stopped if B is stopped. It's a weaker version of Requires - Conflicts: if unit B conflicts with unit B, unit B will be stopped if unit A is to be started and vice-versa - Before/After: declares ordering within a single transaction. Note that ordering is orthogonal in systemd: Requires/Wants do NOT imply ordering dependencies, those have to be made explicitly. (Note that for targets Requires/Wants do sometimes result in implicit ordering, see below for details; but all other unit types don't have that logic, so service A requiring service B will NOT lead to an ordering between them, unless you also specify that explicitly.) So in the case of our before-decrypt.target, we have the following dependencies: - Requires=basic.target This means that the basic system setup should be pulled in. This is essential; if you leave that out the most basic startup things will not be started if you try to boot into before-decrypt.target. - Conflicts=rescue.service rescue.target If you were dropped in an emergency shell at boot (or requested it explicitly over the command line), this conflict will tell systemd that once you do continue booting from the emergency shell, it should stop the emergency shell automatically. - After=basic.target rescue.service rescue.target Make sure that before-decrypt.target is started only after the very basic system initialization is complete. But wait a minute: we copied this unit from multi-user.target, where all the non-GUI services should all be running... Where are they? This unit file appears to only pull in basic.target and nothing else. Well, since you don't want to modify a unit file every time you install a new daemon on your system, systemd provides ways to add additional dependencies to units dynamically via symlinks. There are two directories for multi-user.target: /lib/systemd/system/multi-user.target.wants /etc/systemd/system/multi-user.target.wants Each directory contains symlinks to unit files that are automatically added as Wants= type dependencies to the target (you can also have .requires). The directory in /lib contains all native systemd services that shouldn't be disabled by the administrator (and systemctl disable won't work on them although you could manually do something), while the latter contains all service that can easily be disabled by the administrator (those include SSH, cron, MTAs, web servers, ...). Typically one wants to still enable all units from the /lib directory also in our target, so let's do the following: mkdir /etc/systemd/system/before-decrypt.target.wants cd /etc/systemd/system/before-decrypt.target.wants for i in /lib/systemd/system/multi-user.target.wants/* ; do ln -s /lib/systemd/system/$(basename $i) . done Then we additionally want to enable SSH and syslog (but no other services): cd /etc/systemd/system/before-decrypt.target.wants ln -s /etc/systemd/system/syslog.service . ln -s /lib/systemd/system/ssh.service . A couple of notes: - we take syslog.service from /etc because I don't know what your favorite syslog daemon is and syslog.service is always a symlink to the current one (although you may have to force it manually because in Jessie there's a Debian packaging bug if you change syslog daemons that the symlink doesn't get updated) - both syslog and ssh are native services, which is why that works - you can also enable additional services you might need in the same way. There's no cron nor atd started here (because those might require the encrypted partition to be mounted - or not, depending on your system) and also no MTA - note that these symlinks only work if you have native systemd service files. You can also enable init scripts, but then you have to explicitly specify Wants=[INITSCRIPTNAME].service, e.g. Wants=exim4.service - since the service files for init scripts are dynamically generated and you thus can't easily just create a simple symlink (in the default case they are activated because systemd's generator looks for /etc/rcX.d/S**INITSCRIPTNAME symlinks and then creates Wants= dependencies for runlevelX.target dynamically - and all runlevel[2-5].target are symlinks to graphical.target) Then we should be nearly ready for a test boot to see if our minimal system boots. To summarize what we have done so far: - modify /etc/crypttab and /etc/fstab to add all required entries with the 'noauto' option - create /etc/systemd/system/before-decrypt.target (see above for the contents) - create directory /etc/systemd/system/before-decrypt.target.wants and symlink all services in multi-user.target that we need before the partition is decrypted As a final step we can now make our new target the default boot target: systemctl set-default before-decrypt.target Now let's reboot the system. It will come up again and hopefully the following will have happened: - it didn't hang at boot (otherwise check noauto options) - SSH was started - other system daemons that were not included in the list we activated were NOT started (e.g. no cron/atd/exim4) Now for the final step: we want to be able to decrypt the partition and enter the password. So let's create a file /etc/systemd/system/decrypt.target that contains the following: [Unit] Description=Decrypted System Requires=before-decrypt.target After=before-decrypt.target Conflicts=systemd-ask-password-console.path systemd-ask-password-console.service systemd-ask-password-plymouth.path systemd-ask-password-plymouth.service Requiresemail@example.com srv.mount start-full-system.service The interesting settings in that file are: - Conflicts= lots of ask-password stuff: This is to work around a quirk in systemd's handling of asking passwords for encrypted partitions. See footnote  for an explanation if you are really interested. - Requires=...: this is the meat of the matter. Here we make sure we pull in - the service that decrypts the partition - the mount for /srv - and an additional service (see below) that will cause the rest of the daemons to start (start-full-system.service) This means that when one then starts this target, it will cause the partition to be decrypted, the filesystem to be mounted and then all daemons to be started. Let's look at start-full-system.service: [Unit] Description=Start full system after decryption After=decrypt.target [Service] Type=oneshot ExecStartPre=/bin/systemctl is-active --quiet decrypt.target ExecStart=/bin/systemctl --no-block start multi-user.target The unit calls systemctl to start multi-user.target. The reason this is done is because of ordering: we only want to start daemons once the filesystem is mounted. But we don't want to add an explicit After= dependency to each daemon (or list every daemon as a Before= depdency here), so instead of pulling in multi-user.target directly in decrypt.target and adding all those dependencies, we simply create a small service here that is ordered AFTER decrypt.target that causes systemd to start multi-user.target. A word on service ordering: unless you set DefaultDependencies=no, a target will always be ordered AFTER all of its other dependencies (Requires=, Wants=, Conflicts=, ...). However, if a service already has an explicit ordering dependency, this will not occur. So what we have here is the following ordering: decrypt.target has an implicit Afterfirstname.lastname@example.org srv.mount BUT because start-full-system.service is ordered relative to decrypt.mount explicitly already, it doesn't get an implicit ordering. So that means we have the following order w.r.t. those 4 units: email@example.com srv.mount | | \-------------------+--------------/ | v decrypt.target | v start-full-system.service (In principle, one could also order srv.mount after the cryptsetup service, but that is unnecessary, since mount units in systemd will always wait for the corresponding devices to appear, so an explicit ordering is not required here.) If you are wondering what the ExecStartPre= is for: it's to make sure that decrypt.target was successful before it continues. Otherwise, if decrypting and/or mounting fails, it would still be executed, so add an additional check that it only starts if decrypting was successful. As a final thing w.r.t. start-full-system.service, let's talk about the --no-block here: it causes systemctl to enqueue the start job for multi-user.target but will return immediately. The reason for that is that systemd serializes transactions to a certain extent, so that not using --no-block here would cause the system to deadlock because systemctl would wait for the multi-user.target transaction to finish but systemd wouldn't begin the transaction until the transaction containing start-full-system.service would be finished. (It's generally a very bad idea to call systemctl without --no-block in service files.) However, this means that the starting of the daemons will be initiated by starting decrypt.target, but it will be done in the background. Once this is done, let's do a systemctl daemon-reload to make sure systemd has picked up the changes and now we can try to see if it works: systemctl start decrypt.target You will now be asked for the encrypted partition's password. Once you've entered that, the filesystem will be mounted, systemctl start will return you to the command line and in the background all daemons will be started automatically. You can reboot the system, log in per SSH again, and use the command again - and it should work. Now wait a minute: how does this password prompt work exactly? - systemd-cryptsetup@XXX.service will tell systemd that it needs a password to decrypt the drive - systemd has to ways of getting to such a passowrd: either via a so-called 'ask-password agent' (typically used at boot or if the administrator plugs in a known encrypted drive dynamically) - or, if the password is asked as part of a transaction that was triggered via systemct, systemctl will ask the password directly. So this means that simply telling systemd to decrypt the drive will then lead to a direct password prompt at that point. So in the end, one needs only to enter one single command, then enter the password. Everything will be decrypted and all services will be started then. Neat, huh? Summary of what we did: - adjust /etc/crypttab and /etc/fstab : use noauto! - create /etc/systemd/system/before-decrypt.target [Unit] Description=System before Decryption Requires=basic.target Conflicts=rescue.service rescue.target After=basic.target rescue.service rescue.target AllowIsolate=yes - enable required services for before-decrypt.target mkdir /etc/systemd/system/before-decrypt.target.wants cd /etc/systemd/system/before-decrypt.target.wants for i in /lib/systemd/system/multi-user.target.wants/* ; do ln -s /lib/systemd/system/$(basename $i) . done ln -s /etc/systemd/system/syslog.service . ln -s /lib/systemd/system/ssh.service . - create /etc/systemd/system/decrypt.target [Unit] Description=Decrypted System Requires=before-decrypt.target After=before-decrypt.target Conflicts=systemd-ask-password-console.path systemd-ask-password-console.service systemd-ask-password-plymouth.path systemd-ask-password-plymouth.service Requiresfirstname.lastname@example.org srv.mount start-full-system.service - create /etc/systemd/system/start-full-system.service [Unit] Description=Start full system after decryption After=decrypt.target [Service] Type=oneshot ExecStartPre=/bin/systemctl is-active --quiet decrypt.target ExecStart=/bin/systemctl --no-block start multi-user.target - reboot - to decrypt: systemctl start decrypt.target, enter your password at the prompt A couple of comments: 1. This can be extended to multiple encrypted drives and also multiple mounts. If you want to see what unit name systemd wants to use for a given mount point (so you know what to set as Requires= in decrypt.target), you can use 'systemctl status /mount/point' to see how systemd mangles that into a unit name 2. If you install additional daemons, Debian will hook them up in multi-user.target (either directly if they are native or indirectly if they are sysv scripts), so if you install something that should already be started before decryption, you need to manually add it. On the other hand, since we don't modify multi-user.target itself, any new daemon will always be hooked up there automatically, so that if you install something, it will automatically be part of the start-only-when-decrypted logic. 3. If you have systemd installed, it's recommended (but not strictly required) to have dbus installed. If you have, note that dbus is the only service (other than systemd's internal ones and some plymouth stuff that's not relevant for headless systems) that hooks up in /lib/systemd/system/multi-user.target.wants (and not in /etc, as all other services do!), and if it's installed it should probably be activated already for the minimal system, so if you install dbus only after copying all the symlinks to before-decrypt.target.wants (see above), then you should copy that one manually. 4. If you storage stack is a bit more complicated (e.g. you use LVM on top of the encryption), you may or may not need to manually add some services to activate that once the drive is decrypted. It's definitely possible, and might even work out of the box (I haven't tried it), but you may have to play around a bit there. 5. If you truly have a headless system, you should test this in a VM first, especially when it comes to the boot process. So there it is: how to solve your particular problem with systemd on Jessie. I hope I could use this use-case to illuminate a couple of concepts in systemd and give you enough information so that you have a good starting point into the topic. Regards, Christian  The Conflicts= is required since systemd's own service to notify root of password prompts via wall(1) has the following command: /bin/systemctl stop [... all units listed in Conflicts= here ...] This is fine because it's typically started dynamically when the administrator plugis in a drive dynamically or so (and is not active during boot, there the other services are active, which in turn don't really work after boot, that's why they are stopped). The problem here is that we start this from within a transaction, causing systemd to enqueue the stop of the units until after the transaction for decrypting is done - but then we have a deadlock again. Using Conflicts= here will simply tell systemd to always stop those services as part of the transaction to start decrypt.service, so nothing will deadlock.  One could also turn it around and NOT make decrypt.target depend on start-full-system.service, but the other way around (still with Requires=), then the ExecStartPre would not be necessary, and one could do systemctl start start-full-system.service... That's probably a matter of taste...  There are of course other options of overriding this; for example one could leave the standard target alone, just override the services for very few daemons and then hook those up with a new target. There are advantages and disadvantages with either method.
Description: OpenPGP digital signature