[SOLVED] Proxmox-ZFS Configuration

[gkummer]

Hi.
I'm an absolute beginner in the proxmox and ZFS matter - just start to migrate from hyper-v to proxmox.
I bought a brand-new server with proxmox pre-installed. It contains 2 smaller disks for the OS, which should be mirrored
and 3 larger disks (connected as RAID-Z1) for the vm's.
Now I'm not sure, that I really got the configuration I ordered (and got no answer from the seller about my question).
First, I'm not sure if the 2 smaller disks are really mirrored, and especially I'm worried if the vm's are really stored on the larger disks.
I created a first VM.
On the GUI it seems the storage is ok - but if I open a shell and look at the FS - I don't find the VM's at the location I would expect -
i.e at /vm_nvme i.e. the larger disks and instead via links in /dev/zvol/vm_nvme on /dev/

For infos from the GUi I attached files and below some system info

~# cat /etc/pve/storage.cfg
dir: local
path /var/lib/vz
content iso,backup,vztmpl

zfspool: local-zfs
disable
pool rpool/data
content images,rootdir
sparse 1

zfspool: vm_nvme
pool vm_nvme
content rootdir,images
mountpoint /vm_nvme
nodes praxis-kummer
sparse 1

i.e. pool vm_nvme is at /vm_nvme but
--------

~#df -h
...
vm_nvme 3.4T 128K 3.4T 1% /vm_nvme
...
and ls /vm_nvme -all shows only . and .. and no data
--

BUT th vm seems to be

ls /dev/zvol/vm_nvme -all
drwxr-xr-x 2 root root 120 Jul 15 12:06 .
drwxr-xr-x 3 root root 60 Jul 10 16:09 ..
lrwxrwxrwx 1 root root 9 Jul 10 16:09 vm-100-disk-0 -> ../../zd0
lrwxrwxrwx 1 root root 11 Jul 10 16:09 vm-100-disk-0-part1 -> ../../zd0p1
lrwxrwxrwx 1 root root 11 Jul 10 16:09 vm-100-disk-0-part2 -> ../../zd0p2
lrwxrwxrwx 1 root root 11 Jul 10 16:09 vm-100-disk-0-part3 -> ../../zd0p3

ls /dev/zd* -all
brw-rw---- 1 root disk 230, 0 Jul 10 16:09 /dev/zd0
brw-rw---- 1 root disk 230, 1 Jul 10 16:09 /dev/zd0p1
brw-rw---- 1 root disk 230, 2 Jul 10 16:09 /dev/zd0p2
brw-rw---- 1 root disk 230, 3 Jul 10 16:09 /dev/zd0p3

------

~# lsblk

zd0 230:0 0 32G 0 disk
├─zd0p1 230:1 0 1M 0 part
├─zd0p2 230:2 0 513M 0 part
└─zd0p3 230:3 0 31.5G 0 part
nvme3n1 259:0 0 223.6G 0 disk
├─nvme3n1p1 259:1 0 1007K 0 part
├─nvme3n1p2 259:2 0 1G 0 part
└─nvme3n1p3 259:3 0 222.6G 0 part
nvme4n1 259:4 0 223.6G 0 disk
├─nvme4n1p1 259:5 0 1007K 0 part
├─nvme4n1p2 259:6 0 1G 0 part
└─nvme4n1p3 259:7 0 222.6G 0 part
nvme1n1 259:8 0 1.7T 0 disk
├─nvme1n1p1 259:11 0 1.7T 0 part
└─nvme1n1p9 259:12 0 8M 0 part
nvme2n1 259:9 0 1.7T 0 disk
├─nvme2n1p1 259:13 0 1.7T 0 part
└─nvme2n1p9 259:15 0 8M 0 part
nvme0n1 259:10 0 1.7T 0 disk
├─nvme0n1p1 259:14 0 1.7T 0 part
└─nvme0n1p9 259:16 0 8M 0 par

 

[g777]

Could you got to <your virtual machine name> -> Hardware and see what "Hard disk" starts with?
Is should be something like vm_nvme:vm-123-disk-0,iothread=1,size=32G
If not:

• stop VM
• click Hard disk
• chose disk action
• move storage
• choose where you want to move it.

 

[gkummer]

yes it is at vm_nvme. What I not understand is that the dir /vm_nvme is empty!
The vm seems to be at /dev/zvol/vm_nvme linked to /dev/zdoXXX and not at /vm_nvme

 

[gkummer]

Again to clarify: the storage name vm_nvme should be point to FS - folder /vm_nvme - as it looks for me.
But in spite of what GUI says -- (as I would interpret)!

 

[g777]

What does zfs mount says?

 

[gkummer]

looks also ok
rpool/ROOT/pve-1 /
vm_nvme /vm_nvme
rpool /rpool
rpool/var-lib-vz /var/lib/vz
rpool/ROOT /rpool/ROOT
rpool/data /rpool/data

 

[gkummer]

and also zfs list lokks good.

rpool 9.54G 421G 104K /rpool
rpool/ROOT 2.41G 421G 96K /rpool/ROOT
rpool/ROOT/pve-1 2.41G 421G 2.41G /
rpool/data 96K 421G 96K /rpool/data
rpool/var-lib-vz 7.09G 421G 7.09G /var/lib/vz
vm_nvme 24.6G 3.35T 12.3G /vm_nvme
vm_nvme/vm-100-disk-0 12.3G 3.35T 12.3G -

Is it possible, that ls not "sees" the block devices? But why it lists the vm at /dev/zd*

 

[LnxBil]

yes it is at vm_nvme. What I not understand is that the dir /vm_nvme is empty!
The vm seems to be at /dev/zvol/vm_nvme linked to /dev/zdoXXX and not at /vm_nvme

That's normal.

Is it possible, that ls not "sees" the block devices? But why it lists the vm at /dev/zd*

The block DEVices are there... in /dev where they belong.

 

[gkummer]

ok, but what means this physically.? It's treated like an own device/disk (it's of course a vm-disk) - but
it has to be stored on some physically disk (or pool). If it would be just a ( vhd-) file I would expect to find it at /vm_nvme.
I just wanna get sure, that these are really stored on the large disks.

 

[gkummer]

in short again:
df -h says: vm_nvme is mounted on /vm_nvme with Size = 3,4 TB and Used = 65 GB
proxmox-gui: Storage says: vm_nvme has 3.71 TB and 96.34 GB in use
If I install a VM used size of /vm_nvme keeps constant, where vm_nvme storage increases.

It's a difference of vm_nvme on proxmox vs. basic linux level...

If my server crashes, but I can rescue my disks and put it in another machine. Where I will find the vm's?

 

[g777]

If my server crashes, but I can rescue my disks and put it in another machine. Where I will find the vm's?

You could do a little experiment.

• download livecd/liveusb with zfs support;
• boot server with liveusb;
• ensure zfs kernel module is present;
• see zfs stats for disks and pool;
• if you feel confident enough. Mount zfs and see what's inside. If it's new task for you, better just check stats without altering file system in any way.

I think it's better to find quirks now, rather when system will be up and running in production.

 

[LnxBil]

ok, but what means this physically.? It's treated like an own device/disk (it's of course a vm-disk) - but
it has to be stored on some physically disk (or pool). If it would be just a ( vhd-) file I would expect to find it at /vm_nvme.
I just wanna get sure, that these are really stored on the large disks.

zvols are block devices, so that they're accessed in a block fashion. They are the files (everything in Linux/Unix is a file), yet those are not files you know from Windows. Files are not the most efficient way to do things, therefore VMs are stored on block devices, where each block device is already a disk (which is also just a block device).

Please read about zfs list and zpool list to inspect what you have.

 

[gkummer]

Thanks - yes I understand that I win file is different, but anywhere it has to be physically some place on my disk, where data are stored.
I read about zfs, but not really found the info I searched.
But now I found another thread in this forum from 2021 (how insert link to a thread?) called "VM Disk location (Can't find the VM disk)". Someone else seems to has had the same problem as I ... And I start to understand (and mark it as solved).
... again thanks to all.