Problem: You have a computer running Debian 7.0 wheezy with an encrypted root filesystem. At startup you’d like to be able to unlock the
disk over ssh, maybe because it runs headless.


There are some existing solutions to this: early-ssh worked on previous Debian versions. There are other blog posts showing some methods but I found them unwieldy (ie a 35 second delay before you can type your passphrase.)

Here’s how I’m doing it. Unless otherwise mentioned, all of the following commands need to be run as root. The server name in the examples is “debian_headless”

At time of writing I was using versions Debian 7.0 Wheezy with packages cryptsetup 2:1.4.3-4, initramfs-tools 0.109.1 and dropbear 2012.55-1.3.

These steps may work on recent Ubuntu versions as well, please let me know if you try them out.

Overview

The very first thing a Debian system runs at startup is a minimal initramfs ramdisk which performs basic initialisation, mounts the main filesystem and starts the rest of the boot process.

This is the stage when an encrypted rootfs gets mounted. We need to customise it so we can ssh in and unlock the rootfs.

Install prerequisites

apt-get install dropbear initramfs-tools busybox

Dropbear is a slimline SSH server. This package installs into the initramfs and runs during early boot only, allowing you to log in remotely.

Authorise the user(s) you want to be able to login as root and unlock

Hopefully you’re already using SSH Key Authentication, otherwise have a read about it and why it’s a good idea. Set it up for the user(s) on the client systems by having them run ssh-keygen.

You have to add the public keys for these users to Dropbear’s list of authorized_keys for login.

For each client user you want to allow to unlock, take their public key file ~/.ssh/id_rsa.pub and append it to /etc/initramfs-tools/root/.ssh/authorized_keys

One way of doing it:

scp ~/.ssh/id_rsa.pub myuser@debian_headless:id_rsa.pub
ssh myuser@debian_headless
sudo sh -c "cat id_rsa.pub >> /etc/initramfs-tools/root/.ssh/authorized_keys"
rm id_rsa.pub

Alternative: Have a unique key for root/unlock

If you don’t want to use your usual user-level ssh key and passphrase to provide potential root-level access to your headless machine, generate a separate key (with a different passphrase) and then copy it across to authorized_keys instead:

ssh-keygen -f ~/.ssh/id_rsa_headlessunlock
scp ~/.ssh/id_rsa_headlessunlock.pub myuser@debian_headless:id_rsa.pub
(etc, as above.)

Add network hardware module

The initramfs doesn’t usually load network hardware modules, so you should add the name of the network hardware module directly to /etc/initramfs-tools/modules

You can find the module name via sysfs. For example:

$ grep DRIVER /sys/class/net/eth0/device/uevent
DRIVER=e1000

So to enable eth0 in this case, you’d add e1000 to /etc/initramfs-tools/modules.

(This may not be necessary on “normal” x86 machines with built-in ethernet, it’s definitely necessary using USB Ethernet on my ARM-based GK802 stick and it’s probably necessary on Raspberry Pi.)

Configure early boot IP Address

Initramfs needs to know the IP address to come up on. You could do this by passing the ‘ip=’ kernel command line parameter, but you can also just set it in an initramfs-tools config file:

Create /etc/initramfs-tools/conf.d/network_config with contents like this:

export IP=<client-ip>:<server-ip>:<gw-ip>:<netmask>:<hostname>:<device>:<autoconf>

<server-ip> can be left blank. Some possible examples:

export IP=dhcp

or

export IP=10.0.0.2::10.0.0.1:255.0.0.0:myserver:eth0

See the nfsroot kernel docs for full details. Remember if using DHCP that you’ll need a static DHCP entry configured so you can predict what IP your headless machine is going to have on startup.

Annoyance: Interface doesn’t reconfigure

It seems that after the main system boots, the network interface won’t automatically change to whatever is set in /etc/network/interfaces (because the interface is seen to be already “up” it gets left alone.) To work around this you could put a line ifdown eth0; ifup eth0 in /etc/rc.local, or just keep the initramfs network settings the same as the ones in /etc/network/interfaces.

Unlock Script

I’ve written a little initramfs-tools hook that generates scripts for easy unlocking of the rootfs.

Download this script file, then read it to make sure you’re not suspicious about what it’s going to do. Then copy it to /etc/initramfs-tools/hooks/mount_cryptroot and mark it executable:

chmod +x /etc/initramfs-tools/hooks/mount_cryptroot

There’s a section at the top of mount_cryptroot that allows you to set an ALLOW_SHELL variable if you want to be able to drop to a remote root shell for diagnostics. Note that this explicitly allows a remote root shell to anyone with an authorised SSH private key. I can’t guarantee it’s impossible to get to a root shell otherwise, but it’s at least not explicitly welcomed.

Update the initramfs

Update the initramfs so it includes all your changes:

update-initramfs -u

Add client SSH config entry

Because Dropbear has a different SSH host key to the normal system, OpenSSH will (rightly) complain if you try connecting to it on the same IP. The solution is to create a special host definition for the initramfs environment. A custom section in ~/.ssh/config:

Host headless_unlock
HostName <IP of headless>
User root
HostKeyAlias headless_unlock

Now you can log in to the unlock stage by simply typing ssh headless_unlock. (Replace headless with whatever name you want to use.)

If you authorized a unique private key to use for unlocking, you can add another line IdentityFile ~/.ssh/id_rsa_headless or similar.

Putting it all together

Here’s what it looks like to connect to my headless host and unlock the disk:

$ ssh headless_unlock

BusyBox v1.20.2 (Debian 1:1.20.0-7) built-in shell (ash)
Enter 'help' for a list of built-in commands.

Unlocking rootfs...
Enter passphrase: 
Success
Connection to 192.168.56.101 closed.

Easy!

Final word about security

When dealing with remote access and with encryption, it’s always worth thinking about security tradeoffs.

There’s the obvious risk that enabling Dropbear in the initramfs means allowing remote root logins to your system. Even with ALLOW_SHELL turned off there’s no guarantee that someone with access to an authorized SSH key (or Dropbear exploit) can’t log in, pop a root shell somehow, and cause havoc.

In addition, when I was looking for existing techniques to do this I came across a post exclaiming ssh unlocking was a terrible idea, because someone with physical access to your machine can then impersonate you remotely and you’ll never know.

I think this is worth considering, but it’s not as bad as it might seem. All encrypted disk access, with local or remote unlocking, relies on physical security of your device. Yes, if someone gets physical or root access they can steal your Dropbear host key and set up another server that pretends to be you and steals your keys. However, they already have physical or root access. They can just as easily insert malware in the initramfs that quietly captures your passphrase as you type it in in locally.

In either case, you have to be sufficiently sure noone is messing with you. With remote access that’s maybe more likely, but I’m not too worried. I’m using this on a USB stick PC that fits in my pocket, and I don’t want a burglar walking out my door with all my personal data. If some very dedicated person comes after my data, there would be easier ways to get it.

(Also, an obvious disclaimer – all the advice/scripts on this page is worth what you paid for it. I’m not a security professional. Please tell me if it seems broken/unsafe, but don’t blame me if you use it for something important and it isn’t safe enough.)

21 thoughts on “Unlocking an encrypted rootfs over ssh with Debian wheezy

  1. “I’m not a security professional.”

    Makes you think, doesn’t it… until someone finds a hole and patches it, there will always be a hole you don’t know about.

    No matter how much walls you put up.

  2. Many thanks for this article, it was very helpful.
    Just wondering if it’s possible to configure more than one early boot network interface with different IP address each. Say I have a server with two physical network cards: one is connected to local network, the other is connected to the Internet. The point is to be able to ssh and supply hard disk password not from local network only, but from other part of the world as well.
    Can I configure two interfaces in the “export IP=…” line for this purpose?

  3. Have you tried this on ubuntu 14.04 and busybox 1.21.1? For the life of me I can’t get this to work. I can shell in, but the busy box commands don’t work or it says, applet not found, or it just says “”Unlocking rootfs…”” and sits there.

  4. Or errors like this

    BusyBox v1.21.1 (Ubuntu 1:1.21.0-1ubuntu1) multi-call binary.

    Usage: ps

    Show list of processes

    w Wide output
    l Long output

    ps: invalid option — ‘o’

    • Hmm, that’s unfortunate. I just did some poking around my Ubuntu 14.04 laptop.

      On Debian, /bin/busybox is supplied to both the rootfs and the initramfs by the ‘busybox’ package.

      On Ubuntu, /bin/busybox is supplied on the rootfs by ‘busybox-static’ package but on the initramfs it’s supplied by ‘busybox-initramfs’ package. It’s a totally different busybox, you can find it at /usr/lib/initramfs-tools/bin/busybox. The busybox-initramfs seems to be stripped back with no -o option (and probably missing other non-essential applets), however the busybox-static busybo has -o options! Confusing, especially when they both identify themselves as the same binary! http://pastebin.com/raw.php?i=UkxH57B9 Oh Ubuntu, you’re so silly!

      There is actually a third package in Ubuntu simply named ‘busybox’, same name as Debian, but this package doesn’t replace busybox-initramfs just busybox-static. :(

      It’s possible that you could simply mutilate your system by copying /bin/busybox over /usr/lib/initramfs-tools/bin/busybox, run update-initramfs -u -k all, and then everything would just work. It’s equally possible doing this will break your system and prevent it from booting!

      Another option that doesn’t involve mutilating packages could be to try and override BUSYBOXDIR somewhere in /etc/initramfs-tools, because /usr/share/initramfs-tools/hooks/busybox looks to BUSYBOXDIR to find /bin/busybox…

      There may also be another way to get the full arguments of the “cryptsetup luksOpen” command, which is what the “-o args” argument to ps is for, but I can’t think of one off the top of my head.

      If you find a solution (elegant or not) to this problem, please let me know!

  5. Thank you very much for your guide.

    I’ve been hurting my head against walls to setup network on initramfs, a lot of websites are giving outdated or not working solutions but yours has done the trick!

  6. Thank you for your good guide. I have one observation:
    It is possible to share the same ssh host key between the main system with openssh and the initramfs with dropbear. In fact, dropbear already does this for its configuration on the root fs during package installation. We can copy over the keys from /etc/dropbear/ to /etc/initramfs-tools/etc/dropbear/ or we can reconvert the openssh host keys:

    # /usr/lib/dropbear/dropbearconvert openssh dropbear /etc/ssh/ssh_host_rsa_key /etc/initramfs-tools/etc/dropbear/dropbear_rsa_host_key
    # /usr/lib/dropbear/dropbearconvert openssh dropbear /etc/ssh/ssh_host_dsa_key /etc/initramfs-tools/etc/dropbear/dropbear_dss_host_key

    Note that dropbear doesn’t support ecdsa which is what ssh tries first. Therefore, if a client has previously connected to the host, we must first remove the ecdsa entry from .ssh/known_hosts and connect again with:
    ssh -o HostKeyAlgorithms=ssh-rsa
    After a new entry with an rsa key type has been added it is no longer necessary to specify HostKeyAlgorithm.

  7. Did you even try your script on Debian 7?

    You have errors. Try to create /etc/initramfs-tools/hooks/mount_cryptroot and add chmod +x to this file. When you end with update-initramfs -u command you will got errors.

  8. Yes, I’ve been using this script on a Debian Wheezy (7) box since shortly before I published the post, and that box has been kept up to date.

    The instructions for creating hooks/mount_cryptroot already say “… and mark it executable”.

    If anyone wants to politely provide an error message, rather than just being condescending, then I’d be happy to help them work it out.

  9. @Inferno
    When you get an error then:
    1) you copy/pasted the file instead of downloading it (and in conjunction with some wierdo behaviour editors like vim the file is messed up)
    2) your initramfs is messed up, e.g. missing drivers etc.

    as linux user you should know that stuff like “asking to ask” or “blabla does not work” are absolutly not helpful.
    at my jessie machine this tutorial also works, mostly.

    when restarting, i get “ip-config: no response after xxx – giving up.”, but this is more a bug in syslinux. will try to use a static ip instead.

  10. ssh works, but something is still not really working..

    [oli@localhost ~]$ ssh root@192.168.0.15

    BusyBox v1.22.1 (Debian 1:1.22.0-8) built-in shell (ash)
    Enter ‘help’ for a list of built-in commands.

    Unlocking rootfs…

    then it stops.

    • Hi olze,

      You might get some clues by enabling ALLOW_SHELL in the mount_cryptroot script, rebuild the initramfs, then use Ctrl-C to drop to a root shell? If that works then you can poke around a bit to see if the mount prompt command looks similar, and run /root/mount_cryptroot.sh manually (or with ‘sh -x’) to see exactly what goes wrong.

      If you figure out some changes that make it work for Jessie, let me know and I’ll set up a git repository for the scripts so we can keep track of required changes.

      Angus

    • seems like something changed in debian jessie. i do not have anything with plymouth:
      ps -o args | grep luksOpen | grep -v grep -> nothing
      your while loops are endless

      ps -o (interesting parts):
      {cryptroot} /bin/sh /scripts/local-top/cryptroot
      /lib/cryptsetup/askpass Please unlock disk md1_crypt:
      /sbin/cryptsetup -T 1 open –type luks /dev/disk/by-uuid/c04df9c1-2087-4f9e-8069-6fd99a9ce887 md1_crypt –key-file=-
      /sbin/dropbear

  11. got it working now, simply changing the grep luksOpen to “open” only will work. Unfortunately, this will maybe not work anymore if some application is running with “open” in its name :)
    but as workaround its ok

    • Nice work! Looks like the cryptsetup command line arguments have gotten more sophisticated.

      I expect we can probably change the line from this:

      > CMD=`ps -o args | grep luksOpen | grep -v grep`

      To this:

      > CMD=`ps -o args | grep cryptsetup | grep -i open | grep -v grep`

      … and it will work on both Wheezy and Jessie (and not likely catch any false positives). Any chance you could please try that out and let me know?

      – Angus

      PS The ‘plymouth’ parts are for when graphical boot environments are enabled, the while loop will either match that plymouth command or the direct cryptsetup one (which is for when a text boot environment is enabled).

      • Hi Angus

        I tried your suggestion and replaced

        > CMD=`ps -o args | grep luksOpen | grep -v grep`

        with

        > CMD=`ps -o args | grep cryptsetup | grep -i open | grep -v grep`

        on a fresh Jessie install. Did work without any issues.

        Thanks a lot?

  12. I always get the following message: “IP-Config: failed to set default route on eth0″
    In the /etc/network/interfacs have route entries in.
    post-up route add … dev eth0
    post-up route add default gw …
    post-down route del … dev eth0
    post-down route del default gw …

  13. Hi Daniel,

    Hmm. Can you use the standard ‘gateway xyz’ clause instead of the post-up/post-down commands, or is there a subtle difference in the way they work?

    I actually run mine with static DHCP so it’s possible even that doesn’t work though, I don’t know….

    Angus

  14. Hello Angus,

    the post-up rule is needed, otherwise I get no access inst Internet. And thus I can SSH into the box the Encrypted HDD not decrypt

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>