New LVM Showing as Full

[GGodden]

Hi People,

Im having a bit of a weird issue where I have created a striped Logical Volume using the CLI but when I add it to the storage.cfg or use pvesm add lvm <STORAGE_ID> --vgname <VGNAME> to add it to the cluster it shows up as full. Im sure I am doing somthing really silly but I can't seem to figure it out.

For reference my setup is 4 proxmox nodes connected to a SAN via ISCSI, there are 2 LUNs on the SAN and each proxmox can see both LUNs. I have setup multipathing so that should a controller go down the nodes will still be able to get to the storage. I have then created a Volume Group including both LUNs using vgcreate, from there I have created a striped Logical Volume using lvcreate and have used mkfs to create an xfs file system. I have also tried without having a file system on there but still no luck.

When I try and get proxmox to actually use the LV it shows that all the space is used regardless of the method I use to add it to the nodes. I can see it isn't full as when I mount it on a directory and do df -h it shows 1% usage, I can also then add that directory to the nodes via the GUI and see the volume is empty.

I can't use it as a directory though as I need it to work with HA and when I have tried with it being a directory the other nodes can't see eachothers vm disks and it makes sense to me to just use the LV how its meant to be used.

Thank you for all the help in advance

George

 

[bbgeek17]

I have created a striped Logical Volume using lvcreate and have used mkfs to create an xfs file system

XFS is not a cluster aware filesystem. Do not use it as shared storage.

When I try and get proxmox to actually use the LV it shows that all the space is used regardless of the method I use to add it to the nodes. I can see it isn't full as when I mount it on a directory and do df -h it shows 1% usage, I can also then add that directory to the nodes via the GUI and see the volume is empty.

Proxmox addresses the Volume Group, not Logical Volume. By creating LV - you took all the space.
https://blog.mohsen.co/proxmox-shared-storage-with-fc-san-multipath-and-17a10e4edd8d

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[GGodden]

XFS is not a cluster aware filesystem. Do not use it as shared storage.

Proxmox addresses the Volume Group, not Logical Volume. By creating LV - you took all the space.
https://blog.mohsen.co/proxmox-shared-storage-with-fc-san-multipath-and-17a10e4edd8d

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Hi bbgeek17,

Thats really helpful thank you. Are you able to point me in the direction of a tutorial that shows how to do this with a striped LV? The one you've linked doesn't take that into account and theres no option in the GUI but I would really like to not leave all that performance on the table.

Cheers,

George

 

[bbgeek17]

Thats really helpful thank you. Are you able to point me in the direction of a tutorial that shows how to do this with a striped LV?

I know of no such tutorial. And I don't think you can do what you envisioned. You can only stripe LV, but PVE needs an empty VG. You can concatenate the disks in VG, but not stripe them.

The recommendation is to create one big LUN vs 2 smaller ones.
Striping would only make sense if you have two independent RAID groups on your storage and split the LUN ownership across controllers. If both of your LUNs are backed by the same RAID group, you are not gaining much, if anything, even if you were able to stripe.

Another option is to have two storage pools, each with its own VG backed by a dedicated LUN.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[GGodden]

I know of no such tutorial. And I don't think you can do what you envisioned. You can only stripe LV, but PVE needs an empty VG. You can concatenate the disks in VG, but not stripe them.

The recommendation is to create one big LUN vs 2 smaller ones.
Striping would only make sense if you have two independent RAID groups on your storage and split the LUN ownership across controllers. If both of your LUNs are backed by the same RAID group, you are not gaining much, if anything, even if you were able to stripe.

Another option is to have two storage pools, each with its own VG backed by a dedicated LUN.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

okay in that case is there a way to make this work by mounting a striped lv and then using a directory instead of the lv directly? My setup has 2 Luns on independant raid sets and when I run it how it is now I get roughly half the performance. When I had it setup as a directory I had full performance and both controllers were being used but for some reason the other nodes couldn't see eachothers vm disks.

Thank you for all your help so far

 

[bbgeek17]

okay in that case is there a way to make this work by mounting a striped lv and then using a directory instead of the lv directly?

You need a Cluster Aware Filesystem. There are only two viable options at this time: OCFS and GFS2. Neither are directly supported by PVE. You need to research, install, configure and support them on your own.
Search the forum, there have been many conversations about this topic. I also think there were some tutorials floating around.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox