iSCSI storage encryption for a cluster

trent--

Member
Mar 19, 2021
19
0
6
Hello,
I am setting up a Proxmox cluster with 3 servers, and 1 shared iSCSI storage.
It seems that I can only access this storage in iscsi mode (I asked a question about this at https://forum.proxmox.com/threads/zfs-via-iscsi.74843/post-378849)
With this setup, what are my options for VM encryption ?
- I could encrypt the drives within each VM, but it would be a pain, as I would have to ssh into each VM to decrypt the drive after VM reboot
- ZFS encryption is not an option, as ZFS cannot be used in this setup, as far as I know
- Is there any other option ? Can I setup a huge LUKS volume on the iscsi storage, and setup LVM within that volume ? Will this not create problems with shared access ?
 
So, I setup Proxmox with iSCSI / LUKS / LVM like this :
  • iSCSI access added to Debian OS :
apt-get install open-iscsi vi /etc/iscsi/iscsid.conf [tune config and reboot]
  • iSCSI storage added to Proxmox using web-GUI
  • whole iSCSI device encrypted with LUKS :
cryptsetup -v luksFormat /dev/sdd cryptsetup luksOpen /dev/sdd cryptlvm
  • PV and VG created on this LUKS device :
pvcreate /dev/mapper/cryptlvm vgcreate datavg /dev/mapper/cryptlvm
  • make LVM available to Proxmox :
pvesm lvmscan
  • LVM storage added to Proxmox using web-GUI :
ID: LVM1 Base storage: "Existing volume groups" Volume Group: datavg "Shared" checkbox checked

Now when I create a VM, I use the LVM1 storage for its hard disk.
I tried this with a few VMs, and it seems to work so far.
Is this a supported setup ? Should I be worried about the LUKS part getting in the way ?
As a reminder, I am using a 3-node HA cluster, with iSCSI shared storage, and active VMs on each node.

Thanks in advance.
 
So, I setup Proxmox with iSCSI / LUKS / LVM like this :
  • iSCSI access added to Debian OS :
apt-get install open-iscsi vi /etc/iscsi/iscsid.conf [tune config and reboot]
  • iSCSI storage added to Proxmox using web-GUI
  • whole iSCSI device encrypted with LUKS :
cryptsetup -v luksFormat /dev/sdd cryptsetup luksOpen /dev/sdd cryptlvm
  • PV and VG created on this LUKS device :
pvcreate /dev/mapper/cryptlvm vgcreate datavg /dev/mapper/cryptlvm
  • make LVM available to Proxmox :
pvesm lvmscan
  • LVM storage added to Proxmox using web-GUI :
ID: LVM1 Base storage: "Existing volume groups" Volume Group: datavg "Shared" checkbox checked

Now when I create a VM, I use the LVM1 storage for its hard disk.
I tried this with a few VMs, and it seems to work so far.
Excellent! I'm toying with almost the same setup myself, except experimenting with ssh protocols in initramfs - so I could unlock the volumes remotely over VPN (and trying to avoid IPMI) before the system(s) boot.

This whole experiment has made me think about what needs to be encrypted, and what doesn't... The part I'm hung up on are the TLS certs (letsencrypt everywhere, also VPN) and ssh keys - i'd want those stored on encrypted volumes, but that means entire systems. So now I'm lowering the severity of TLS certs being stolen (only internal TLS certs) because when you think about it, the only reason to encrypt is in the event of theft - because the keys are already present in memory once unlocked and could be extracted via exploits or physical access. So the case of theft is the only reason to encrypt which my servers would defiantly get whole new TLS certs on a rebuild.

As far as ssh keys, I'm making a rule of no private keys on any servers and ssh forwarding everywhere - meaning, it all has to be unlocked (aka "Approved") from my personal (and secure) devices. Not a bad idea, just makes me a road block if the servers reboot in the middle of the day and the kids wants to watch their Plex.

Is this a supported setup ? Should I be worried about the LUKS part getting in the way ?

The feedback I've received here is basically anything you do outside of the GUI is unsupported (unless it's proxmox CLI commands).

But "Unsupported" just means Proxmox's support/staff isn't obligated to help or resolve issues in any official capacity, including subscription support. It's just normal Debian under the hood, and I'm guessing with some special startup scripts that updates common files on boot (e.g. /etc/network/interfaces).

I have to prefix by saying I myself am new to proxmox as well and experimenting. But so far, Proxmox only does what it needs and stops. You are free to setup Debian any way you see fit - just Proxmox staff isn't obligated to resolve issues if you go out-of-bounds and the GUI won't show it. :)

This is why I am setting up an grafana + promethus + influx with alerts setup via a 2nd ISP (a SIM card in one of my SoC devices). It will ingest everything from Proxmox to Debian to containers and VMs and apps into a central place to "monitor." That way, Proxmox GUI just turns into a way to "manage Proxmox-related stuff" and nothing more.

Humm... There's a Feature Request: Plugin Support for Proxmox GUI. Would allow whole 3rd party genres of tools one could install! Oh wells...
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!