Newbie: a couple of storage-related questions

[gctwnl]

I'm a newbie to Linux management (I've been doing low level (macOS) Unix for years, so I'm not totally new to unix-like stuff) as well as managing my own virtualisation setup. I am reading, searching and learning while setting up my new server environment on an Intel NUC with PVE and Ubuntu clients. My apologies for the long post/questions but I thought it best to give as much information to prevent a lot of misunderstandings.

So, far I've been able to install PVE and setup an initial Ubuntu client, purely on the internal NVMe of the NUC. I've also been able to add a USB RAID enclosure and have it encrypted and mounted (at boot) in PVE's Debian. That means that my block devices now look like this:

root@pve:~# lsblk
NAME                                          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                                             8:0    0   1.7T  0 disk 
└─sda1                                          8:1    0   1.7T  0 part 
  └─luks-fa1483bd-f599-4dcf-9732-c09069472150 253:7    0   1.7T  0 crypt /mnt/RNAStore4
nvme0n1                                       259:0    0 931.5G  0 disk 
├─nvme0n1p1                                   259:1    0  1007K  0 part 
├─nvme0n1p2                                   259:2    0   512M  0 part  /boot/efi
└─nvme0n1p3                                   259:3    0   931G  0 part 
  ├─pve-swap                                  253:0    0     8G  0 lvm   [SWAP]
  ├─pve-root                                  253:1    0    96G  0 lvm   /
  ├─pve-data_tmeta                            253:2    0   8.1G  0 lvm   
  │ └─pve-data-tpool                          253:4    0 794.8G  0 lvm   
  │   ├─pve-data                              253:5    0 794.8G  1 lvm   
  │   └─pve-vm--100--disk--0                  253:6    0    32G  0 lvm   
  └─pve-data_tdata                            253:3    0 794.8G  0 lvm   
    └─pve-data-tpool                          253:4    0 794.8G  0 lvm   
      ├─pve-data                              253:5    0 794.8G  1 lvm   
      └─pve-vm--100--disk--0                  253:6    0    32G  0 lvm

So, I've reached my first goal: run PVE with a Ubuntu client and make sure all data is encrypted when the system is turned off (data protection against hardware theft). When the NUC boots I need to enter a hardware password (twice, once for the NUC, once for the internal SSD) then everything boots. As I have set up LUKS on PVE's Debian, I do not have to mess around with disk encryption on each client OS, so far so good.

My next goal is to (create and) mount a 500GB volume from PVE's /dev/sda on the Ubuntu client and make sure that the external data volume is backed up to the internal disk I will add remote backup later). Step one is creating the volume and having it mounted on the Ubuntu client. I assume my steps are to create a VG in PVE, create a LVM and have that mounted as a 'data' volume in the Ubuntu client OS by adding the volume as a 'hard disk' to the VM in PVE.

I am getting stuck because of my unfamiliarity with the toolset, so I have a couple of simple questions that show my ignorance and I could need some help/tips with. These are:

1. How much of this stuff can I do in the PVE GUI? Should I assume the GUI is no use for my stuff and do everything in the underlying Debian CLI? The whole LUKS stuff for the external RAID I did in the CLI on PVE's Debian. Should I avoid the GUI or can it make life simpler for me yet??
2. PVE Install created an LVM of 100GB and a LVM-thin of 900GB
   

   root@pve:~# lvs
     LV            VG  Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
     data          pve twi-aotz-- <794.79g             0.56   0.26                            
     root          pve -wi-ao----   96.00g                                                    
     swap          pve -wi-ao----    8.00g                                                    
     vm-100-disk-0 pve Vwi-a-tz--   32.00g data        13.83

   As I understand it, 'nvme0n1p3' is a partition and that is the same as a physical volume (PV). PVE's Debian has a single volume group (VG) called 'pve' that is made up of that single PV. And VG 'pve' contains two logical volumes, a standard one of 100GB that is called 'root' and a thin one that is called 'data'. The PVE installer has put the Ubuntu ISO on 'LVM:root' and the Ubuntu install created a bootable 32GB VM ('vm-100-disk-0') image from it on 'LVM-Thin:data'. I have two questions (and one unrelated third):
   1. How do I create a VG on /dev/mapper/luks-fa1483bd-f599-4dcf-9732-c09069472150? Create a VG, add /dev/mapper/luks-fa1483bd-f599-4dcf-9732-c09069472150 or /dev/dm-7 to it as PV? Something is not right here, as I have created an ext4 file system on that device and I need it as block device. So, I must stop mounting it in PVE's Debian, I guess and adding /dev/mapper/luks-fa1483bd-f599-4dcf-9732-c09069472150 to the new VG will wipe that file system, correct?
   2. If I have a VG on /dev/mapper/luks-fa1483bd-f599-4dcf-9732-c09069472150, I should be able create a 500GB logical volume in it. What escapes me still is how I then in Ubuntu client OS mount that logical volume.
   3. Because I have a file system on the LUKS-encrypted external drive I could perform a small write speed test:
      

      root@pve:~# dd if=/dev/zero of=/mnt/RNAStore4/tempfile bs=128k count=32k
      32768+0 records in
      32768+0 records out
      4294967296 bytes (4.3 GB, 4.0 GiB) copied, 4.72204 s, 910 MB/s

      But I wonder, is this a reasonable test?

 

[gctwnl]

I can add a part of the answer myself:

# Create the data object in the LVM of PVE's Debian for a PV (overwrites the existing file system on the LUKS block device):
pvcreate /dev/mapper/luks-fa1483bd-f599-4dcf-9732-c09069472150
# Create a VG from that PV:
vgcreate rna-mepdm-1 /dev/mapper/luks-fa1483bd-f599-4dcf-9732-c09069472150
# Create a 500GB LV named "MinIO" on that VG
lvcreate --size 500G -n MinIO rna-mepdm-1

The GUI only sees/recognises part of what I do so far. For instance, when I try to add hardware to my Ubuntu Client OS, the GUI doesn't see the rna-mepdm:MinIO LV.



But the VG is seen:



My guess is that I have to add the new VG to PVE somehow. Probably in /etc/pve/storage.cfg but I don't understand that format (yet)

 

[shanreich]

1. Setting up your LUKS partitions is currently not supported in any way by the UI so you have to resort to managing it via the CLI. The same goes for creating your PV. Creating the VG and LV should be doable through the proxmox GUI, but it is also just as easy to create it from the CLI. Depends on what you prefer.

2.1/2 It depends on how you want to set it up, there are several options which are described in detail in the Arch Wiki [1]. Generally I would suggest trying LVM on LUKS [2], which is something that should be easily doable with your current setup. It also is the easiest to start with and the best general option if you have no specific use case (imho).

What you have setup right now is an ext4 file system on the LUKS partition. What you need to do before creating the ext4 partition is creating a physical volume from your LUKS partition (this is described in great detail in the Arch wiki article I linked) and add it to a volume group. Then afterwards you can create a logical volume from your volume group. This logical volume can then be formatted as any filesystem you like (ext4).

2.3 This is how I also have tested the throughput of encrypted disks personally, so I think it should be fine.

Something that might be interesting in the future for you and worth looking into is using ZFS as your filesystem, since it also supports encryption. [3]

[1] https://wiki.archlinux.org/title/dm-crypt/Encrypting_an_entire_system#Overview
[2] https://wiki.archlinux.org/title/dm-crypt/Encrypting_an_entire_system#LVM_on_LUKS
[3] https://pve.proxmox.com/wiki/ZFS_on_Linux#zfs_encryption

 

[shanreich]

My guess is that I have to add the new VG to PVE somehow. Probably in /etc/pve/storage.cfg but I don't understand that format (yet)

You should be able to this if you click on Datacenter > Storage. If you want a thin pool you need to specify this when creating your LV. This can be done with the -T parameter of the lvcreate command. If you don't want a thin pool, then it is sufficient to create the volume group and add it as a regular LVM storage.

 

[gctwnl]

You should be able to this if you click on Datacenter > Storage. If you want a thin pool you need to specify this when creating your LV. This can be done with the -T parameter of the lvcreate command. If you don't want a thin pool, then it is sufficient to create the volume group and add it as a regular LVM storage.

Thank you. I had created a VG in the cli, but I did not find a way to add it. Using Datacenter->Storage I was able to create a new storage there. This created a new 500GB (I was unable to somehow reuse the LVM I had already created). So now I have this:

root@pve:~# vgs
  VG          #PV #LV #SN Attr   VSize    VFree 
  pve           1   4   0 wz--n- <931.01g  15.99g
  rna-mepdm-1   1   2   0 wz--n-   <1.75t 788.36g
root@pve:~# lvs
  LV            VG          Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data          pve         twi-aotz-- <794.79g             0.56   0.26                           
  root          pve         -wi-ao----   96.00g                                                   
  swap          pve         -wi-ao----    8.00g                                                   
  vm-100-disk-0 pve         Vwi-aotz--   32.00g data        13.83                                 
  MinIO         rna-mepdm-1 -wi-a-----  500.00g                                                   
  vm-100-disk-0 rna-mepdm-1 -wi-ao----  500.00g

I can now remove LV MinIO and I can add rna-mepdm-1:vm-100-disk-0 to Ubuntu where it shows up as /dev/sbd

What confused me is that I am adding a LV in Datacenter, but I was expecting to either be able to tell PVE about the VG or tell PVE to use the LV I had created. But I had to create the LV via Datacenter in the GUI. Probably some sort of pvesm command would have worked as well.

 

[gctwnl]

Everything works now. I've partitioned, formatted and mounted the LV on the Ubuntu client OS.

Returning to my performance question. The LUKS-encrypted external RAID did 910 MB/s write speed when mounted on PVE's Debian. On the Ubuntu clientOS I only keep roughly half of that:

$ sudo dd if=/dev/zero of=/mnt/MinIO/tempfile bs=128k count=32k
32768+0 records in
32768+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 8.86554 s, 484 MB/s

I would expect a bit of a performance loss, but not 50%. Maybe worth a separate question later.

 

[Dunuin]

Also keep in mind that your swap LV isn't encrypted. So data of the encrypted enclosre might be stored unencrypted in RAM and the RAM is then stored unencrypted on the swap LV. This might leak your data.

 

[gctwnl]

Also keep in mind that your swap LV isn't encrypted. So data of the encrypted enclosre might be stored unencrypted in RAM and the RAM is then stored unencrypted on the swap LV. This might leak your data.

Nope. My root disk is hardware encrypted. Any swap remainders on the disk will be encrypted as well (when the machine is turned off). That is why I wanted to have the encryption at the lowest level possible,

 

[shanreich]

Everything works now. I've partitioned, formatted and mounted the LV on the Ubuntu client OS.

Returning to my performance question. The LUKS-encrypted external RAID did 910 MB/s write speed when mounted on PVE's Debian. On the Ubuntu clientOS I only keep roughly half of that:

$ sudo dd if=/dev/zero of=/mnt/MinIO/tempfile bs=128k count=32k
32768+0 records in
32768+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 8.86554 s, 484 MB/s

I would expect a bit of a performance loss, but not 50%. Maybe worth a separate question later.

Maybe something to do with the blocksize? Just a wild guess though, would have to try myself.