Cluster Templates ( NFS )

[CloudHoster]

Hello

We are having an issue with templates.

We use NFS to store KVM templates however it appears that eac node in our cluster requires a separate templates to be 'housed' there.

Is it possible to have KVM templates living on NFS on one node and each cluster node be able to deploy from that NFS storage.

At the moment we use Hostbill but we receive errors when deploying templates from NFS to a node that the KVM templates do not live on.

It is no a Hostbill issue as the whole point of shared NFS is all nodes should be able to manage NFS shared storage.

The difficulty is we need to have 2 sets of tempaltes living on both nodes even though they both use the same templates. on NFS

 

[layer7.net]

Hi,

as far as i can see, thats not possible with proxmox.

But it would be definitly an interesting feature that templates can be shared between hosts / clusters.

 

[MeneM]

To me this seems strange.

So each node needs to have these templates installed? And also with unique VMID's?

Then what is the point of making templates when they need to be unique on each node?

 

[bbgeek17]

What you described works perfectly fine in our every day CI/CD workflow. All templates are kept on shared NFS storage. In fact it is shared across multiple independent clusters...

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[MeneM]

Do you mean the container templates? Those are fine indeed.

I mean the KVM type templates. even though we save them on shared NFS storage, only the node that made the templates "sees" them. As in: the config is in /etc/pve/ but not accessible to the rest?

Do you perhaps have an example?

 

[bbgeek17]

root@pve7test1:~# qm config 2000
boot:  
memory: 512
meta: creation-qemu=7.2.0,ctime=1685298891
name: vm2000
onboot: 0
scsi0: bbnas:2000/vm-2000-disk-0.raw,size=112M
scsihw: virtio-scsi-pci
smbios1: uuid=31534635-9c3a-4f50-8e3a-179d0419eef2
sockets: 1
vmgenid: 21f7a7bb-4353-481e-8299-992fd99d9e87

qm template 2000
root@pve7test1:~# qm config 2000
boot: 
memory: 512
meta: creation-qemu=7.2.0,ctime=1685298891
name: vm2000
onboot: 0
scsi0: bbnas:2000/base-2000-disk-0.raw,size=112M
scsihw: virtio-scsi-pci
smbios1: uuid=31534635-9c3a-4f50-8e3a-179d0419eef2
sockets: 1
template: 1
vmgenid: 21f7a7bb-4353-481e-8299-992fd99d9e87

root@pve7test1:~# pvesm status
Name             Type     Status           Total            Used       Available        %
bbnas             nfs     active       100773504        21473792        79299712   21.31%

root@pve7test1:~# qm clone 2000 101 --target pve7test2
create linked clone of drive scsi0 (bbnas:2000/base-2000-disk-0.raw)
clone 2000/base-2000-disk-0.raw: images, vm-101-disk-0.qcow2, 101 to vm-101-disk-0.qcow2 (base=../2000/base-2000-disk-0.raw)
Formatting '/mnt/pve/bbnas/images/101/vm-101-disk-0.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=117440512 backing_file=../2000/base-2000-disk-0.raw backing_fmt=raw lazy_refcounts=off refcount_bits=16

root@pve7test2:~# qm list
      VMID NAME                 STATUS     MEM(MB)    BOOTDISK(GB) PID       
       101 Copy-of-VM-vm2000    stopped    512                0.00 0

root@pve7test2:~# qm config 101
boot: 
memory: 512
meta: creation-qemu=7.2.0,ctime=1685298728
name: Copy-of-VM-vm2000
onboot: 0
smbios1: uuid=b45d3edf-4e11-4fb1-995c-2905a463d7a1
sockets: 1
vmgenid: 4e1bdf56-f791-4589-87aa-50ee1012e3f5
root@pve7test2:~# qm config 101
boot: 
memory: 512
meta: creation-qemu=7.2.0,ctime=1685298891
name: Copy-of-VM-vm2000
onboot: 0
scsi0: bbnas:2000/base-2000-disk-0.raw/101/vm-101-disk-0.qcow2,size=112M
scsihw: virtio-scsi-pci
smbios1: uuid=455b5ab1-2e40-42ef-bf6e-52f11cfb5dc4
sockets: 1
vmgenid: 6958cc66-ca33-4226-a9e2-0b24d55aefbc

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[MeneM]

Ok awesome, and I see the resulting clone is on the same NFS share as the template?

I think that is the thing, I can get that to work as well, but I want to go from the NFS share to the lvm-thin or zfs or even local folder of any node.

But that looks awesome, thank you very much for the example!

 

[bbgeek17]

perhaps your issue is:

root@pve7test1:~# qm clone 2000 101 --target pve7test2 --storage local-lvm
can't clone to non-shared storage 'local-lvm'

root@pve7test1:~# pvesh create /nodes/pve7test1/qemu/2000/clone -newid 102 -ful 1 -format raw -storage local-lvm -target pve7test2
can't clone to non-shared storage 'local-lvm'

In this case you need to use shared storage as a target, i.e. Ceph or Blockbridge:

pvesh create /nodes/pve7test1/qemu/2000/clone -newid 102 -ful 1 -format raw -storage blockbridge  -target pve7test2
create full clone of drive scsi0 (bbnas:2000/base-2000-disk-0.raw)
transferred 0.0 B of 112.0 MiB (0.00%)

root@pve7test2:~# qm config 102
boot: 
memory: 512
meta: creation-qemu=7.2.0,ctime=1685298891
name: Copy-of-VM-vm2000
onboot: 0
scsi0: blockbridge:vm-102-disk-0,size=112M
scsihw: virtio-scsi-pci
smbios1: uuid=a0cb7146-1ae9-481e-8013-721a8c427903
sockets: 1
vmgenid: 36fee2da-cd23-4fc5-81b6-027c2d4aa81b

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[MeneM]

Yeah I guess this makes sense indeed. local storage is not a logical destination to "cluster-wide/shared" kvm templates