Cluster with both zfs and lvm

[Patrik Stenfeldt]

Hi all.

I just found proxmox a few days ago and since i wanted to move away from esxi for a while i installed a machine with it.

SInce then i have retired one of my esxi hosts to run proxmox on it and have a ton of issues (but im saving that for another thread).

It is now up and running and im trying to start a cluster. The test machine only have a single hdd and the next machine if fitted with a raid which i decided to configure as zfs.

If i simply connect it to the cluster the local-zfs dissapear.

How do i make a mixed file-system cluster?

 

[wolfgang]

Hi,

Proxmox Storage is cluster-wide.
When you add a node to a cluster you will get the storage conf of the cluster and loos your local one.
In your case, you have to add the local-zfs again and restrict it to the node where it exists.

see
https://pve.proxmox.com/wiki/Storage

 

[Patrik Stenfeldt]

Hi,

Proxmox Storage is cluster-wide.
When you add a node to a cluster you will get the storage conf of the cluster and loos your local one.
In your case, you have to add the local-zfs again and restrict it to the node where it exists.

see
https://pve.proxmox.com/wiki/Storage

Great. Thanks for the reply.

 

[rioga]

Hi!

I'm on the same situation as PAtrik and I can't find a solution. I'm trying to add a node with ZFS-local storage to a cluster with LVM-thin storage. When I add the ZFS node, the local-zfs storage from the node just dissapear and it's replaced with the local-lvm storage from the cluster (up to this point, normal behaivor). If I try to add a ZFS storage to that node as wolfang said there is no ZFS pool selectable fom the menu in datacenter storage.

I'm new in proxmox, what am I missing?

Thanks!

 

[rioga]

Here is the solution if someone is in the same situation:
Once the node with ZFS storage is member of the cluster, open a shell on any node and edit the /etc/pve/storage.cfg configuration file:
you have to add:

zfspool: name_of_the pool
pool rpool/data
content rootdir,images
mountpoint /rpool/data
nodes nodename_which_will_mount_ZFS_Storage
sparse 0


for example, for mounting a pool named pvo-pool-ssd on server name pvo:

zfspool: pvo-pool-ssd
pool rpool/data
content rootdir,images
mountpoint /rpool/data
nodes pvo
sparse 0

Cheers!

 

[timdonovan]

Thank you @rioga!

 

[shabbirh]

Here is the solution if someone is in the same situation:
Once the node with ZFS storage is member of the cluster, open a shell on any node and edit the /etc/pve/storage.cfg configuration file:
you have to add:

zfspool: name_of_the pool
pool rpool/data
content rootdir,images
mountpoint /rpool/data
nodes nodename_which_will_mount_ZFS_Storage
sparse 0


for example, for mounting a pool named pvo-pool-ssd on server name pvo:

zfspool: pvo-pool-ssd
pool rpool/data
content rootdir,images
mountpoint /rpool/data
nodes pvo
sparse 0

Cheers!

Thank you; this helped me hugely!!

 

[noahpugsley]

Here is the solution if someone is in the same situation:
Once the node with ZFS storage is member of the cluster, open a shell on any node and edit the /etc/pve/storage.cfg configuration file:
you have to add:

zfspool: name_of_the pool
pool rpool/data
content rootdir,images
mountpoint /rpool/data
nodes nodename_which_will_mount_ZFS_Storage
sparse 0


for example, for mounting a pool named pvo-pool-ssd on server name pvo:

zfspool: pvo-pool-ssd
pool rpool/data
content rootdir,images
mountpoint /rpool/data
nodes pvo
sparse 0

Cheers!

Just worked for me. Thanks!
Let me migrate over so now I can re-do the original also zfs raid1 and have it all kosher.

Cheers.

 

[Frunk]

Hello,

so this happened to me as well. I had a 3 node cluster and wanted to replace a node. So I went and deleted one. I followed the instructions in the documentation
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_remove_a_cluster_node
it did not say anything about removing storage.
When I added a new node with ZFS root filesystem the local-lvm storage stopped functioning. I could migrate a a VM onto the node but could not migrate away. It posted the following error

ERROR: Problem found while scanning volumes - no such logical volume pve/data

so as mentioned here I went to edit the file /etc/pve/storage.cfg
(Interestingly I found mentions of storages from my old node. I deleted them)
Then I modified

lvmthin: local-lvm
thinpool data
vgname pve
content images,rootdir

such that I added the line with node names of the rest of the nodes(the problem was with the node proxmox2)

lvmthin: local-lvm
thinpool data
vgname pve
content images,rootdir
nodes proxmox1,proxmox3

and then the not functioning storage disappeared from the node and I could also migrate VMs.
The rest of the nodes do not have ZFS and use LVM instead as per basic installation.

PS: This could be possibly achieved also in datacenter view in the storage area where for each storage you should be able to choose for which node they should be present.