But at least for the VM storage ZFS on LUKS would be very interesting so clustering makes sense without broken migration.
None Root Disks ZFS on LUKS is easy
Straight copied out of my wiki so sorry for the formating
// Base Setup - Once Per Host
// Optional: Fill device with random data (specially recommended with HDDs) for more details look checklist for disk preparation
dd if=/dev/urandom of=<device>
// Create Folder to store decryption keys of non root disks
mkdir /root/crypt/keys/
chmod -R 400 /root/crypt/
chown -R root:root /root/crypt/
// Create a script that automatically unlocks the drives after boot
// Create a text file that stores the unlock information of non root disks
touch /root/crypt/datadisk-cryptindex
chmod 600 /root/crypt/datadisk-cryptindex
chown root:root /root/crypt/datadisk-cryptindex
// Create the actual script that unlocks all the data crypt disks
nano /root/crypt/datadisk-unlock.sh
Code:
################# Datadisk / Non Root Unlock Script V4
# /root/crypt/datadisk-unlock.sh
# File path to the luks volume list
luksdevices_list="/root/crypt/datadisk-cryptindex"
# Check if the volumes file exist and stop if it doesnt
if [ ! -r "$luksdevices_list" ]; then
echo "$luksdevices_list doesnt exist."
exit 1
fi
# Read the content of the cryptindex and store them inside an array
declare -A volumes
while IFS= read -r line; do
LUKSNAME=$(echo $line | awk '{print $1}')
UUID=$(echo $line | awk '{print $2}')
volumes[$UUID]=$LUKSNAME
done < "$luksdevices_list"
# Debug Line / Echo content of the array
#echo "Volumes Array: ${volumes[@]}"
# Loop through all the disks
for UUID in "${!volumes[@]}"
do
LUKSNAME=${volumes[$UUID]}
echo "Importing Device:"
echo "LUKSNAME: $LUKSNAME"
echo "UUID: $UUID"
# Unlocks the volume with keyfile
cryptsetup luksOpen --key-file /root/crypt/keys/$LUKSNAME.key /dev/disk/by-uuid/$UUID $LUKSNAME --allow-discards
done
// Ensure that the permissions of the script are set correctly
chmod 700 /root/crypt/datadisk-unlock.sh
chown root:root /root/crypt/datadisk-unlock.sh
// Let the script run automatically directly after boot
// Add the script as systemd service
nano /etc/systemd/system/datadisk-unlock.service
Code:
[Unit]
Description=Automatically Import Luks devices
After=network.target
[Service]
Type=oneshot
ExecStart=/bin/bash /root/crypt/datadisk-unlock.sh
[Install]
WantedBy=multi-user.target
// Enable the service
systemctl enable datadisk-unlock.service
// For every disk
// Create Encryption Key File (each disks has its own)
dd if=/dev/urandom of=/root/crypt/keys/crypt-disk1.key bs=512 count=8
// Change Permission so only root can read the file
chmod 400 /root/crypt/keys/crypt-disk1.key
chown root:root /root/crypt/keys/crypt-disk1.key
// Format the drive and create a partition
// Clear all current partition tables of the disk & Create a new GPT partition table
sgdisk -og /dev/disk/by-id/diskyouarecurrentlyworkingon
// Create a new partition that will span accross the entire disk
sgdisk -n1:0:0 -t1:8309 /dev/disk/by-id/diskyouarecurrentlyworkingon
// Create the luks volume
cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha512 -i 6000 /dev/disk/by-id/diskyouarecurrentlyworkingon-part1 --key-file /root/crypt/keys/crypt-disk1.key
Parameters
--type luks2 ensure that luks2 is used (but should be default with cryptsetup 2.1 or newer)
-c aes-xts-plain64 Sets the cipher - in this case the aes-xts-plain64
-s 512 Sets the Key Size
-h sha256 Sets the hash algorithm that is used
–iter-time 3000 or -i 3000 How long it takes (in ms so 3000 means 3 seconds) to unlock on the system where it was originally created. The number of milliseconds to spend with PBKDF2 password processing.
// You can find more parameters, settings and explanations regarding luks here
https://wiki.archlinux.org/title/Dm-crypt/Device_encryption#Encryption_options_for_LUKS_mode
// Mount the luks device
cryptsetup luksOpen /dev/disk/by-id/whateverdiskyouarecurrentlyworkingon-part1 --key-file /root/crypt/keys/crypt-5452452.key crypt-5452452
// Add second decryption password for recovery purposes
cryptsetup luksAddKey /dev/disk/by-id/whateverdiskyouarecurrentlyworkingon-part1 --key-file /root/crypt/keys/crypt-5452452.key
// Add the drive id to the automatic unlock script index
nano /root/crypt/datadisk-cryptindex
// One Disk per line in the form luksname and UUID (you can get the UUID with blkid)
// Example like crypt-datatest1 {{disk uuid}}
// Repeat for the other disks
// Option 1 - Create ZFS Pool
Create ZFS Pool / Raid
// ALWAYS use the luks mapped devices as devices, NEVER the disks directly otherwise the data will be stored next to the luks container unencrypted!!
// If a pool has multiple vdevs data is distributed between than and they behave a little bit similar to a conventional raid 0 / stripe
// Different Parameters / arguments during pool creation
// General
-f force, might ignore potential warnings
// Pool parameters basic (lower case o) must be in front of each parameter if you add multiple ones
-o ashift=12 sets the ashift to 12 (which is correct for drives with 4k sectors ashift=13 would be for drives with 8K sectors)
// Pool parameters advanced (upper case o) must be in front of each parameter if you add multiple ones
-O autotrim=on|off Enables or disables the automatic trim feature for SSDs.
-O dedup=on|off Enables or disables deduplication for the pool.
-O compression=lz4 Enabled compression for the pool
-O encryption=on|off Enables or disables zfs native encryption for the pool. (we are already using LUKS in this guide, this is fully independent of that so leaf it disabled
-O logbias=throughput|latency Sets the write cache priority for the ZFS Intent Log (ZIL).
-O primarycache=all|none|metadata Sets the primary cache setting for the pool.
-O secondarycache=all|none|metadata Sets the secondary cache setting for the pool.
-O recordsize= Sets the size of data blocks in the pool.
-O redundant_metadata=most|all Specifies how redundant the metadata in the pool should be stored.
// you can see even more potential parameters with the zpool create -O help command
// Pool Name
<pool> Choose a name for the pool, following naming standard if applicable like dpool-01-nvme-a
// Devices
raidz1 <devices> the actual data vdev
cache <cache-device> Adds a layer2arc / ssd cache to a pool
log <log-device> Adds a log device
// You can find more informations under
https://openzfs.org/wiki/System_Administration#Pool_creation and
https://openzfs.github.io/openzfs-docs/Performance and Tuning/Workload Tuning.html
// Examples for most common configurations
Basic Raidz1 Pool
zpool create -o ashift=12 -O compression=lz4 <pool> raidz1 /dev/mapper/<device1> /dev/mapper/<device2> /dev/mapper/<device3>
Basic Raidz2 Pool
zpool create -o ashift=12 -O compression=lz4 <pool> raidz2 /dev/mapper/<device1> /dev/mapper/<device2> /dev/mapper/<device3> <device4>
Basic Raidz3 Pool
zpool create -o ashift=12 -O compression=lz4 <pool> raidz3 <devices>
Raidz3 Pool with de-duplication, a layer2arc cache and a log device
zpool create -o ashift=12 -O compression=lz4 -O dedup=on dpool-01 raidz3 <device1> <device2> <device3> <device4> <device5> cache <cache-device> log <log-device>
Basic Equivalent to Raid 10 (two vdevs that each contain two disks as a mirror / raid1)
zpool create -o ashift=12 -O compression=lz4 <pool> mirror <device1> <device2> mirror <device3> <device4>
Basic Equivalent to Raid 60 (stripe of two raidz2 / raid6)
zpool create -o ashift=12 -O compression=lz4 <pool> raidz2 <device1> <device2> <device3> <device4> raidz2 <device5> <device6> <device7> <device8>
// Create a vm-disk subdataset to be used by proxmox for vm-disks zfs create <pool>/subpoolname
zfs create dpool-01/virtual-disks
zfs create -O dedup=on dpool-01/virtual-disks-dedup
zfs create dpool-01/data
// Optional: List all zfs datasets
zfs list
zfs get all or zfs get all <pool/datasetname>
// Add the vm-disk pool to proxmox via the webui
// Go to the datacenter view then Storage > Add > ZFS and add the subpools we just created (NOT pool root)
// Verify that automatic unlocking, zfs import and vm autostart is working
