[SOLVED] Multi node cluster, local ZFS per node

[tjk]

Hi,

What is the correct way to setup local ZFS Pools on each node? I am not looking to share them/ha across nodes, just want to use a local pool on each node for VM's, and used my shared storage for HA (NFS) across the pool.

I have the cluster setup, and when I add a zpool to a node, it looks good, I add a new zpool to another node in the cluster and the other zpool disappears.

I feel like I'm missing a step since I don't want HA/shared zpools across the cluster.

Edit - Running latest commercial 7.2.7.

Thanks!

 

[oguz]

hi,

I have the cluster setup, and when I add a zpool to a node, it looks good, I add a new zpool to another node in the cluster and the other zpool disappears.

where did you add/create the zpool?

could you post your /etc/pve/storage.cfg file?

 

[tjk]

Thanks, info below. The error is different this time, it won't let me create "tank" on more than one node, it gives me an error 500.

Node 1:

Gui -> Node 1 -> ZFS - Create ZFS -> TANK -> Raid10 -> Add Storage Checked

Datacetner -> Storage -> TANK -> Edit -> Thin Provision Checked

Node 2:

Gui -> Node 1 -> ZFS - Create ZFS -> TANK -> Raid10 -> Add Storage Checked ->
Error Storage ID Tank already defined (500).

root@node1:~# cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content vztmpl,iso,backup

zfspool: local-zfs
        pool rpool/data
        content rootdir,images
        sparse 1

zfspool: tank
        pool tank
        content rootdir,images
        mountpoint /tank
        nodes node1
        sparse 1

root@node1:~#

root@node1:~# pveversion -v
proxmox-ve: 7.2-1 (running kernel: 5.15.35-2-pve)
pve-manager: 7.2-7 (running version: 7.2-7/d0dd0e85)
pve-kernel-5.15: 7.2-4
pve-kernel-helper: 7.2-4
pve-kernel-5.15.35-2-pve: 5.15.35-5
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph-fuse: 15.2.16-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-3
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-3
libpve-storage-perl: 7.2-5
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.12-1
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.3-1
proxmox-backup-file-restore: 2.2.3-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-1
pve-container: 4.2-1
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.4-2
pve-ha-manager: 3.3-4
pve-i18n: 2.7-2
pve-qemu-kvm: 6.2.0-11
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.4-pve1
root@node1:~#

I assume I need unique zpool names per node if I am not clustering them?

 

[oguz]

it won't let me create "tank" on more than one node, it gives me an error 500.

in that case you should use a different name for the pool, or you can activate the storage for the other node if you want to use the same name

 

[tjk]

Thanks, unique names per node worked. I assume if I want to cluster/share/ha the zpool's and use replication etc I need to use the same zpool name on each node in the cluster?

 

[aaron]

As a follow up especially regarding HA: the pools need to have the same name.

If you create the pool via the GUI on the first node, you can leave the "Add Storage" checkbox enabled. On the second node, you will need to disable that checkbox for now and then adjust the nodes list for that storage when you edit it under Datacenter->Storage.

To remedy that now, you can remove the second storage from the Datacenter->Storage config and rename the pool on the second node. To do so, first export it

zpool export <name>

and then import it:

zpool import <name> <new name>

Then check the Proxmox VE Storage config and make sure that the node is also selected for that storage. Then it should show up for both and things like replication and HA should work.

I just sent a patch to make this easier in the future

 

[tjk]

Thank you both! Not sure what the initial issue was, but adding the zpools for local use (non clustered, non HA) worked once I gave each pool a unique name. Thank you!!!