[SOLVED] 7.4 cluster with mixed storage (XFS and ZFS)

[cybc]

I have issues since on my Proxmox cluster since I added a new node with ZFS (RAIDZ2) storage.

I had a 3-node cluster (version 7.3) where each nodes (names: node1, node2 and node3) had a single disk using XFS filesystem, I added a new v7.4 node (name: node4) having 8 disks ZFS RAIDZ2.

Note: After this I upgraded each 7.3 node to version 7.4-3.

As an example of issue: I can not migrate an existing VM from old XFS nodes to the new ZFS node because it can not find the "pve" VG, error is:

Volume group "pve" not found

I guess the migration process is using LVM features to copy data from one node to another and the new node lacks LVM so it fails.

I managed to migrate an existing VM having no disk from an old XFS node to the new ZFS but, from this new location, I cannot add a disk to it, I get the following error:

failed to update VM 114: no such logical volume pve/data (500)

The /etc/pve/qemu-server/114.conf does not seems to reference any leftover storage configuration:

boot: order=ide2;net0
cores: 2
ide2: none,media=cdrom
machine: pc-i440fx-7.1
memory: 4096
meta: creation-qemu=7.1.0,ctime=1682077154
name: foo.example.com
net0: e1000=1A:D1:B7:83:19:0D,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsihw: virtio-scsi-single
smbios1: uuid=848a35f6-6ca9-4bb0-8ca8-030ed9ad1ab7
sockets: 1
vmgenid: 9c11d306-2ee3-4683-bb6b-9c8f26f2009b

I also cannot, on the new ZFS node, create a VM with a disk (on the only available storage which is "local-lvm") as I get similar error:

TASK ERROR: unable to create VM 118 - no such logical volume pve/data

Creating a VM without disk works fine, but I canot delete it:

TASK ERROR: no such logical volume pve/data

(I had tested VM-with-disk creation on the new ZFS node prior to adding it to the cluster: it worked just fine)

Is my cluster setup doomed to not working because one cannot mix LVM and ZFS like this? Or is there a solution/config I failed to find?

More details:

The "Storage" of the cluster only list those 2 (no ZFS) storages:

• local
• local-lvm

(should I add a ZFS here?")

And the "Disks" section of each nodes are as follow:

• For the old XFS nodes (1 disk):
  • LVM: pve (/dev/sda3)
  • LVM-Thin: data (in "pve" VG)
  • Directory: (empty)
  • ZFS: (empty)
• For the new ZFS node (8 disks):
  • LVM: No VGs found
  • LVM-Thin: No thin-Pool found
  • Directory: (empty)
  • ZFS: rpool

 

[mira]

Since the `local-lvm` storage is not available on the 3rd node, you have to limit it to the first 2 nodes under Datacenter -> Storages.
Since it is not limited currently, it assumes the storage is available on the 3rd node as well.

Live migration will then require an explicit target storage set when migrating between the 3rd and the other nodes.

 

[cybc]

(Small note: the new ZFS node is the 4th node -node4- of the cluster as I had a cluster with 3 nodes: node1, node2 and node3)

OK, I get it.

I guess the local-lvm storage can not be configured added on the 4th node, because it seems weird to have "LVM" stuff over ZFS filesystem.

I applied the suggested settings of limiting the local-lvm to nodes 1-3 and since then the "Add disk" wizard list no storage on the storage dropdown menu for node4.

I don't remember what where the storages my new ZFS node ad before joining the cluster, but I guess it did had a configured storage which was removed when joining the cluster (probably named "local-zfs")?

So if I want to be able to migrate VMs from storages that are physically on the nodes (no external one like NFS, CIFS, …) I can either:

• Go for consistency: Re-install nodes 1-3 with the ZFS setting during installation wizard?
• Re-add/configure the missing ZFS storage on node4 which will then be available as target when migrating
• Add a "Directory" type of storage, available to each nodes and store my disk images and containers there? But would it loose the benefits of LVM or ZFS features?

 

[mira]

The simple approach would be to add the ZFS storage via Datacenter -> Storages and limit it to node4.
Then you can create new disks on that node and (live) migrate the VM to any of the nodes again, specifying the target storage (node1/node2/node3 <-> node4).

 

[cybc]

I have re-created a storage (at Datacenter level) with the following settings (taken from a fresh ZFS installation of Proxmox on a test VM):

• ID: local-zfs
• ZFS Pool: rpool/data
• Content: Disk image, Container
• Notes: node4
• Enable: [x] (checked)
• Thin provision: [x] (checked)
• Block Size: 8

I can now:

• Add hard drives to existing VMs on node4
• Live migrate VM from nodes1-3 to node4

Offline migration with GUI is not working (Error "Storage (local-lvm) not available on selected target. Start VM to use live storage migration or select other target node") and qm migrate 666 node4 --targetstorage local-zfs does not work:

2023-05-02 17:36:57 starting migration of VM 666 to node 'node4' (192.168.0.104)
2023-05-02 17:36:57 found local disk 'local-lvm:vm-666-disk-0' (in current VM config)
2023-05-02 17:36:57 copying local disk images
2023-05-02 17:36:57 ERROR: storage migration for 'local-lvm:vm-666-disk-0' to storage 'local-zfs' failed - cannot migrate from storage type 'lvmthin' to 'zfspool'
2023-05-02 17:36:57 aborting phase 1 - cleanup resources
2023-05-02 17:36:57 ERROR: migration aborted (duration 00:00:01): storage migration for 'local-lvm:vm-666-disk-0' to storage 'local-zfs' failed - cannot migrate from storage type 'lvmthin' to 'zfspool'
migration aborted

Looks similar to cannot migrate from storage type 'lvmthin' to 'zfspool' but I have no cloudinit drive.

 

[mira]

That's a limitation of offline migration.
In offline migration we have to export it from the source storage and import it on the target storage.
Some storages are incompatible.

Online migrations works differently, via QEMU, and works in every case.

 

[cybc]

Thanks for the confirmation: it's good enough for me. I just wanted to be sure I was not in face of another issue a simple configuration setting could fix.

Thanks for the debug/help.