Opinions expressed here belong to your're mom
So the global rise in awareness of authoritarian police tactics finally has you encrypting your devices, huh?
You saw a fella get hit with a felony for using his duress password recently and figured "if it's encrypted you don't have to hit the duress". It's been like this for a long time but for some reason the news actually reports on it now.
Personal carry devices are the tip of the iceberg. Most of your day-to-day data lives on a server. Maybe you trust the provider. Maybe you are the provider. But just because your personal data is on a machine in a datacenter or a VM (or even in your own homelab) does not mean that it is safe from government overreach. They have been pulling these kinds of stunts likewise for years.
These might seem like edge cases, but whatever tactics have been used against others can be used against you. Curing the disease that leads to this kind of thing probably requires an arborist certification, but device and server encryption can help treat the symptoms. The only issue with server encryption is that you have to actually dial in a password, which is supremely inconvenient for a machine that you might not have physical access to and which you probably aren't sitting in front of at all times when it is powered on (that's kind of the whole point of a server).
This blogpost covers a few options for encrypting your server without requiring that you schedule onsite time at the datacenter for every reboot. All of the exact examples here are suited for Ubuntu server machines running 24.04. They will probably work for other recent LTS versions. They might need some adjustment for other distros.
If you want to decrypt your data it without physically going to the machine, there is no way to do it without placing your trust somewhere. By definition, decrypting your computer remotely allows your computer to be decrypted without your presence. The big question comes down to where you place that trust.
This trust can be vested in various places:
The choice of where you put your trust is not one that I can make for you. You have to weigh the options and consider your threat model. Most of the vulnerabilities here come from different varieties of an "evil maid" attack. A server is basically the most evil-maid-able type of computer imagineable. The only solace here is that (for now), cold boot, hot plug, and ANT-catalog-style tampering attacks are difficult, technically complex, and expensive. There are a limited number of teams that are capable of carrying them out, and they probably aren't being dedicated towards you, the reader of this post.
If you are the type of person who those resources would be dedicated to, then you'll need to engineer some custom solutions to keep anyone from getting physical access to your servers in an environment where other people have physical access to your servers. This is a fun mental game to play but is outside the scope for this blogpost. Maybe another day.
Most colocation deals that I have seen include some level of rudimentary "remote hands". This generally means that the employees of the datacenter will go over to your computer and do basic hardware swaps, push some buttons, and interact with with your machine on a crash cart.
Obviously, telling everyone at the datacenter what your decryption key is kind of defeats the point. This information would likely be stored in some CRM software and it would likely be read whenever your hardware is disappeared from the rack. Putting it in a tamper-proof envelope doesn't help much either, since this would have to be physically stored somewhere and it would probably disappear with your server.
The most sensible way to do this would probably be to have a pre-determined key that, using a complicated expect script on unlock, rotates the key on each boot. This way, you could provide the passphrase to the datacenter worker and have a false sense of confidence that it would only be used once.
This has some pretty obvious downsides and is certainly the most low-tech method for getting this done. Most apparent among them is that you don't actually need to go through a whole boot process just to type in a passphrase, and a nefarious worker could be lying to you, with your disk plugged into his computer.
This also doesn't work so great for homelab equipment where you are probably the worker in question and you want to decrypt your systems from another room. Additionally, is the datacenter even staffed 24/7? Probably not.
You know the "virtual console" button on the VPS that you rent for $5/mo? Physical hardware has that too, but as a spearate interface called a "remote console" or a "LOM" (Lights Out Management). This interface allows you to interact with the hardware of your system even when the OS is powered off. This is fantastic news, because you're one button away from having a virtual crash cart plugged into the machine.
This (usually) requires a separate network cable run to your machine into the LOM port. You'll also need your LOM to be configured to hop onto the network that you plug it into and you'll need a way to actually reach your LOM over the Internet. This probably looks like a special VPN tunnel from your provider so that you can conserve public IP addresses.
You're also trusting your LOM's security. Security updates here are rare, and applying them at all is occasionally tricky. For all its downsides, this is solution uses things as they already exist and is certainly the lowest effort way to set things up. You just type your decryption key into the remote console as if you were sitting at the machine itself.
Since your remote console comes from the hardware itself, if someone were to replace your disks and boot into an OS that looks identical to your installation but instead just records whatever decryption key you entered, they could trick you and effectively steal your decryption key.
If you can't reach your LOM (or don't trust it), you can install a minimal SSH daemon called "Dropbear" into your initial ramdisk. With this, you can remotely connect to the server before it is decrypted and manually type in your key.
You can hopefully be a little bit more confident in the security of Dropbear than you can your LOM because it is actually open source and auditable. Updating it is as easy as updating the rest of your system.
If you use your main host's SSH keys for Dropbear in the initrd, then these keys are just sitting there waiting to be taken. You can use fresh SSH keys exclusively for Dropbear to alleviate this, but you'll have to look past the standard "someone may be doing something nasty" message.
You first need some prereqs:
sudo apt isntall dropbear-initramfs busybox-initramfs
Then you'll need to set the config options in /etc/initramfs-tools/initramfs.conf:
...
BUSYBOX=yes
DROPBEAR=yes
...
That same file will also need the static configuration for your IP address.
IP=<V4_ADDRESS>::<V4_GATEWAY>:<SUBNET_MASK>:<HOSTNAME>:<DEVICE>:off
...example...
IP=10.0.0.2::10.0.0.1:255.0.0.0:great_server:eth0:off
I don't recommend relying on DHCP here in case DHCP is down. In general I don't think that hard-relying on DHCP for server IP addressing is a good idea. If you really want to use it though, you'd use this config:
IP=:::::eth0:dhcp
Next you'll need to configure Dropbear itself. Make the entire content of /etc/dropbear/initramfs/dropbear.conf look like:
DROPBEAR_OPTIONS="-I 1000 -j -k -p 22 -s -R"
Since password-based logins are disabled here, you need to copy in your SSH public key. Put it in the file /etc/dropbear/initramfs/authorized_keys the same as you would put it in any authorized_keys file.
Then to make this all take effect, run:
update-initramfs -ck all
To SSH in after a reboot, you'd just do ssh -o StrictHostKeyChecking=no root@address.
Once you're logged in, you can get your disk unlock prompt by running the command cryptroot-unlock.
So far all of these solutions have required you, a human who might be asleep, dead, or otherwise incapacitated, to manually provide a decryption key through some method.
Clevis and Tang create a totally automated system. Tang runs on a separate computer as a daemon, then Clevis runs on boot on the client. On boot, Clevis reaches out to Tang and gets the password. The exact mathematical details of the implementation exceed my understanding, but based on the Tang project's readme, this is much more secure than traditional key escrow systems.
Becuase Clevis and Tang use asymmetrical encryption, the whole thing can be done totally anonymously. The Tang server never knows the Clevis client's decryption keys. Since the keys never get transmitted unencrypted over the wire, you can run Clevis/Tang over the Internet. It uses existing JSOE and HTTP REST standards, so if you really want you then you can build your own infrastructure around it from scratch using off-the-shelf components.
Setting up the Tang server is super straightforward and simple. You just:
sudo apt install tang
The upstream default port is 9090/tcp and the default key storage location is /var/db/tang/, but on Ubuntu it's 80/tcp and /var/lib/tang/.
Once you've got Tang installed, the systemd socket tangd.socket is running and ready to start up a fresh Tang service for each request. If you look at the contents of /var/lib/tang/ then you'll find an empty directory until you make your first request, which you can do with:
$ clevis encrypt tang '{"url":"http://localhost:80"}' <<< "asdfasdfasdf"
The advertisement contains the following signing keys:
eoP7ntkPXxK5VQCnL7NHFlCb-v_NYLRJOUmzuwGyAaM
Do you wish to trust these keys? [ynYN]
This will give you some encrypted output on stdout, but importantly it will give you some keys in /var/lib/tang/. In my example it created these two:
The latter of which matches the key fingerprint in the above terminal output. Now that you can be sure of your fingerprint from your local machine, you can enroll a client Clevis into this Tang.
If your Tang server is also hosting any kind of web traffic, then you'll probably want to either proxy connections through a reverse proxy like Nginx or put it on a different port. The Nginx proxy setup is out of scope for this blogpost, but switching the port is as simple as editing the /usr/lib/systemd/system/tangd.socket file and changing the ListenStream port.
Install prereqs:
sudo apt install clevis clevis-luks clevis-initramfs
You might need to add the initramfs option to your /etc/crypttab file for the crypt device that you want to auto-decrypt.
Then, to enroll your Clevis client into your Tang system, make sure that your Tang is online and reachable from the Clevis system, and:
# Get the block device path (like /dev/sda1)
blkid -t TYPE=crypto_LUKS
# Enroll in tang
clevis luks bind -d [DEVICE] tang '{"url": "http://tang_server_address:80"}'
Clevis will prompt you to enter an existing password for the block device. Then it will ask you if you accept the fingeprint (check it against the previous output). After both of these steps are complete, Clevis will add its own Tang-derived password as well.
If you want, you can evern set up multiple Tang servers, with either of them being able to decrypt your machine:
clevis luks bind -d [DEVICE] sss '{"t":1,"pins":{"tang":[ {"url":"http://tang_server_address:80"}, {"url":"http://another_tang_server_address:8080"} ]}}'
Crypto nerds generally suggest that you occasionally rotate your keys. Ubuntu's tang distribution provides a simple way to do this. You just run:
/usr/libexec/tangd-rotate-keys -d /usr/lib/tang/
This program hides the old keys from advertisement (so newly enrolled clients won't use them) and creates new keys. The old keys will still be used, however, to satisfy requests coming in for them.
To remove the old key and use the new key on a Clevis client, simply run the regen (assuming you are using slot 1, which you are if you're following along here for the first time):
sudo clevis luks regen -d /dev/sda1 -s 1
If you need to get a slot list, you can see it with:
sudo clevis luks list -d /dev/sda1
The TPM is a modern Clipper Chip. It sits there on your motherboard, all proprietary and black-boxy, storing your decryption key and surely only releasing it under circumstances that you previously decided. There is certainly no way at all that a proprietary chip designed to hold your decryption key would ever betray you with a government-mandated backdoor. That kind of thing would never happen.
Even though you (sensibly, like good cattle) trust the TPM manufacturer's honesty and integrity to stand up for your privacy under any circumstances, there could still be ways to get the key out of the TPM.
Maybe your TPM setup from your distro doesn't actually do verification well enough for the system-to-be-unlocked. Maybe your OS implementation is fine but literally any other piece of software in the boot chain has a flaw.
Maybe all the software is flawless. A TPM chip still has to pass the information to the CPU anyways, and at this point it can be sniffed with a wide variety of hardware.
I don't believe in the security of TPM-bound encryption any more than I believe in the checking account of the tooth fairy.
What are you supposed to do if you've already deployed your machine without disk encryption? Or if you're working with a VPS that doesn't let you manually walk through the installer? There are various options out there for post-hoc encryption, but sadly no easy-to-use tool that lets you convert your existing non-encrypted disk to an encrypted disk.
Install and configure fscrypt:
sudo apt install fscrypt
# Follow the prompts
sudo fscrypt setup
# Get your filesytem device path, change if you're doing this on a non-root device
mount | grep 'on / '
# Turn on filesystem encryption, [DEVICE] is the path from your previous command, like /dev/dm-1
sudo tune2fs -O encrypt "[DEVICE]"
Simply create a directory that you want to encrypt and then run:
$ sudo fscrypt encrypt /encrypted
The following protector sources are available:
1 - Your login passphrase (pam_passphrase)
2 - A custom passphrase (custom_passphrase)
3 - A raw 256-bit key (raw_key)
Enter the source number for the new protector [2 - custom_passphrase]: 2
Enter a name for the new protector: really_great_password
Enter custom passphrase for protector "really_great_password":
Confirm passphrase:
"/encrypted" is now encrypted, unlocked, and ready for use.
Then, to unlock or lock the directory, just use either:
sudo fscrypt lock /encrypted/
# OR
sudo fscrypt unlock /encrypted/
NB: eCryptfs used to be the standard tool for this usecase, but it has become unmaintaned and may soon be deprecated. This is a shame because it is the only tool in this whole blogpost that has a cool mascot.
I haven't touched upon systemd's disk encryption at all in this blogpost. It can handle TPM-backed encryption if you want. However, the systemd-homed can manage your home directory with encryption. With this, you can encrypt your home directory as a filesystem with LUKS encryption. The usecase for this would be to create a service account and store the sensitive data in that account's homedir, managed by systemd-homed.
This isn't installed by default on Ubuntu, so to set it up you'll need to first install it:
sudo apt install systemd-homed
homectl list
If everything is working, you'll get an output that says No home areas. This is great because it means that systemd-homed is installed and running but that it didn't do anything with your existing home directory.
sudo homectl create username --storage=luks --disk-size=15G
Then you can unlock and lock the homedir (in this case /home/username/) with:
sudo homectl activate username
# OR
sudo homectl deactivate username
On Ubuntu the disk images are stored in /home/*.home.
Gee Bill, two blogposts in one week? This writeup was finished in preparation for my upcoming talk at FOSSY 2026 in Vancouver, BC. Come check it out!