NVMe/tcp volume group discovery

R

r88

New Member
Aug 26, 2025
6
0
1
I have a proxmox cluster of 5 hosts with a shared SAN. The SAN exposes a number of volumes over NVMe/tcp and it all works fine.
The issue I have is when I add a new volume on the SAN, I initialise the volume group on one pve node, then the other nodes do not see the new vg without a reboot.

e.g.
Create the VG on pve-t1.
Bash: 
root@pve-t1:~# ls -l /dev/nvme-eui.e86fe7a15954b9a98ccf096800240d12
lrwxrwxrwx 1 root root 7 Aug 29 10:12 /dev/nvme-eui.e86fe7a15954b9a98ccf096800240d12 -> nvme1n5
root@pve-t1:~# vgcreate nvg5 /dev/nvme-eui.e86fe7a15954b9a98ccf096800240d12
  Physical volume "/dev/nvme-eui.e86fe7a15954b9a98ccf096800240d12" successfully created.
  Volume group "nvg5" successfully created
root@pve-t1:~# pvs
  PV                                             VG       Fmt  Attr PSize    PFree   
  /dev/nvme-eui.807c891bc81207058ccf0968000ccee0 nvg4     lvm2 a--    <5.00g   <5.00g
  /dev/nvme-eui.9ef11edc03865ebb8ccf0968002eff51 nvg2     lvm2 a--  <500.00g <348.00g
  /dev/nvme-eui.e86fe7a15954b9a98ccf096800240d12 nvg5     lvm2 a--   <10.00g  <10.00g
  /dev/nvme-eui.e966eea5c41c19f08ccf0968000b3657 nvme_vg1 lvm2 a--  <100.00g    7.99g
  /dev/nvme-eui.f35464bb446a77ef8ccf096800325af1 nvg3     lvm2 a--  <100.00g  <68.00g
  /dev/sda3                                      pve      lvm2 a--  <135.70g   16.00g
root@pve-t1:~# vgs
  VG       #PV #LV #SN Attr   VSize    VFree   
  nvg2       1   2   0 wz--n- <500.00g <348.00g
  nvg3       1   1   0 wz--n- <100.00g  <68.00g
  nvg4       1   0   0 wz--n-   <5.00g   <5.00g
  nvg5       1   0   0 wz--n-  <10.00g  <10.00g
  nvme_vg1   1   3   0 wz--n- <100.00g    7.99g
  pve        1   3   0 wz--n- <135.70g   16.00g


Then on other nodes I can see the raw volume but they do not detect it as a volume group.
Bash: 
root@pve-t2:~# ls -l /dev/nvme-eui.e86fe7a15954b9a98ccf096800240d12
lrwxrwxrwx 1 root root 7 Aug 29 10:12 /dev/nvme-eui.e86fe7a15954b9a98ccf096800240d12 -> nvme1n5
root@pve-t2:~# pvs
  PV                                             VG       Fmt  Attr PSize    PFree   
  /dev/nvme-eui.807c891bc81207058ccf0968000ccee0 nvg4     lvm2 a--    <5.00g   <5.00g
  /dev/nvme-eui.9ef11edc03865ebb8ccf0968002eff51 nvg2     lvm2 a--  <500.00g <348.00g
  /dev/nvme-eui.e966eea5c41c19f08ccf0968000b3657 nvme_vg1 lvm2 a--  <100.00g    7.99g
  /dev/nvme-eui.f35464bb446a77ef8ccf096800325af1 nvg3     lvm2 a--  <100.00g  <68.00g
  /dev/sda3                                      pve      lvm2 a--  <135.70g   16.00g
root@pve-t2:~# vgs
  VG       #PV #LV #SN Attr   VSize    VFree   
  nvg2       1   2   0 wz--n- <500.00g <348.00g
  nvg3       1   1   0 wz--n- <100.00g  <68.00g
  nvg4       1   0   0 wz--n-   <5.00g   <5.00g
  nvme_vg1   1   3   0 wz--n- <100.00g    7.99g
  pve        1   3   0 wz--n- <135.70g   16.00g
root@pve-t2:~#

If I add the storage in the pve cluster datacenter it is added to all nodes however shows up as unknown until a reboot of the node. I expect this is because the node doesn't detect the new vg.

Have I missed a step to get this working correctly? I don't really want to have to go into each node every time I add a new volume, especially as our cluster grows to more nodes.
 
pvscan picks it up but it doesn't then appear under pvs etc.

Bash: 
root@pve-t2:~# pvscan
  PV /dev/nvme-eui.9ef11edc03865ebb8ccf0968002eff51   VG nvg2       lvm2 [<500.00 GiB / <348.00 GiB free]
  PV /dev/nvme-eui.e966eea5c41c19f08ccf0968000b3657   VG nvme_vg1   lvm2 [<100.00 GiB / 7.99 GiB free]
  PV /dev/nvme-eui.e86fe7a15954b9a98ccf096800240d12   VG nvg5       lvm2 [<10.00 GiB / <10.00 GiB free]
  PV /dev/nvme-eui.807c891bc81207058ccf0968000ccee0   VG nvg4       lvm2 [<5.00 GiB / <5.00 GiB free]
  PV /dev/nvme-eui.f35464bb446a77ef8ccf096800325af1   VG nvg3       lvm2 [<100.00 GiB / <68.00 GiB free]
  PV /dev/sda3                                        VG pve        lvm2 [<135.70 GiB / 16.00 GiB free]
  Total: 6 [<850.68 GiB] / in use: 6 [<850.68 GiB] / in no VG: 0 [0   ]
root@pve-t2:~# pvs
  PV                                             VG       Fmt  Attr PSize    PFree   
  /dev/nvme-eui.807c891bc81207058ccf0968000ccee0 nvg4     lvm2 a--    <5.00g   <5.00g
  /dev/nvme-eui.9ef11edc03865ebb8ccf0968002eff51 nvg2     lvm2 a--  <500.00g <348.00g
  /dev/nvme-eui.e966eea5c41c19f08ccf0968000b3657 nvme_vg1 lvm2 a--  <100.00g    7.99g
  /dev/nvme-eui.f35464bb446a77ef8ccf096800325af1 nvg3     lvm2 a--  <100.00g  <68.00g
  /dev/sda3                                      pve      lvm2 a--  <135.70g   16.00g
 
bbgeek17 said:
You can try "vgscan --mknodes"

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...
This didn't seem to help.

Bash: 
root@pve-t2:~# vgscan --mknodes
  Found volume group "nvg2" using metadata type lvm2
  Found volume group "nvme_vg1" using metadata type lvm2
  Found volume group "nvg5" using metadata type lvm2
  Found volume group "nvg4" using metadata type lvm2
  Found volume group "nvg3" using metadata type lvm2
  Found volume group "pve" using metadata type lvm2
root@pve-t2:~# pvs
  PV                                             VG       Fmt  Attr PSize    PFree   
  /dev/nvme-eui.807c891bc81207058ccf0968000ccee0 nvg4     lvm2 a--    <5.00g   <5.00g
  /dev/nvme-eui.9ef11edc03865ebb8ccf0968002eff51 nvg2     lvm2 a--  <500.00g <348.00g
  /dev/nvme-eui.e966eea5c41c19f08ccf0968000b3657 nvme_vg1 lvm2 a--  <100.00g    7.99g
  /dev/nvme-eui.f35464bb446a77ef8ccf096800325af1 nvg3     lvm2 a--  <100.00g  <68.00g
  /dev/sda3                                      pve      lvm2 a--  <135.70g   16.00g
root@pve-t2:~# vgs
  VG       #PV #LV #SN Attr   VSize    VFree   
  nvg2       1   2   0 wz--n- <500.00g <348.00g
  nvg3       1   1   0 wz--n- <100.00g  <68.00g
  nvg4       1   0   0 wz--n-   <5.00g   <5.00g
  nvme_vg1   1   3   0 wz--n- <100.00g    7.99g
  pve        1   3   0 wz--n- <135.70g   16.00g
root@pve-t2:~
 
The command "pvscan --cache" seems to have done the trick. Bit of a pain having to run this on every pve node in a cluster. Anyone aware of another way to avoid this?
 
We just use one volume group for everything. On the SAN is everything already on all disks, so it makes for us no sense to do it otherwise.

Why do you use more than one? What is your usecase?
 
LnxBil said:
We just use one volume group for everything. On the SAN is everything already on all disks, so it makes for us no sense to do it otherwise.

Why do you use more than one? What is your usecase?
Click to expand...
We want to control the permissions of each storage area separately so team A can not see the storage area of team B. We can then also assign quotas such as 1TB to team A and 2TB to team B. The only way I could see to do this was via multiple volume groups. I would be interested if there is another way as it would simplify things.
 
Would be keen to know if there is a better way of doing this. Anyone got any thoughts? Thanks
 
You must log in or register to reply here.
Top Bottom