ZFS Sync

[lordhanuman]

Hi I'm testing a 3 node cluster, before I venture into ceph, where I already know I'm going to cry, I'm testing zfs replication.
there are 3 nodes with an SSD where there is the operating system and a second dedicated to the ZFS.
If I try to insert a zfs with the same name in the cluster it won't let me, but when I replicate the VPS it gives me an error saying it can't find the destination disk. Here is the error from the log

2022-01-12 22:51:00 400-0: start replication job
2022-01-12 22:51:00 400-0: guest => VM 400, running => 0
2022-01-12 22:51:00 400-0: volumes => ssd1:vm-400-disk-0
2022-01-12 22:51:01 400-0: (remote_prepare_local_job) storage 'ssd1' is not available on node 'sec'
2022-01-12 22:51:01 400-0: end replication job with error: command '/usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=ger' root@10.10.10.1 -- pvesr prepare-local-job 400-0 ssd1:vm-400-disk-0 --last_sync 0' failed: exit code 255

What am I missing?
Thanks

 

[fabian]

the storage.cfg entries are shared across the cluster - so you only need one there. but of course the underlying zpool / dataset needs to exist on each node where you want to use it.

if you do everything over the GUI - only tick the 'add storage' box when creating the zpool once (e.g., for the first or the last node) and leave it unchecked for the rest - but use the same pool name on each node. if you do it manually - create the pools/datasets on all nodes with the same name, then add a single storage.cfg entry.

 

[lordhanuman]

ok i managed to show zfs volumes on all nodes.
I first created the volume on the first node and placed it in the pool.
then I created the other volumes in the other nodes without checking to put them in the datastore.
when I gave everything from the cluster management I entered the names of the volumes on the other nodes, in this way I saw them.
After that I inserted some replication processes.
the test VPS that I inserted, however, in the has simulating the failure of a node does not migrate to another node and remains in a state of freeze.
the logs from the first node say this. but I can't understand why the VPS was not migrated

an 13 16:50:13 fra systemd[647026]: Reached target Exit the Session.
Jan 13 16:50:13 fra systemd[1]: user@0.service: Succeeded.
Jan 13 16:50:13 fra systemd[1]: Stopped User Manager for UID 0.
Jan 13 16:50:13 fra systemd[1]: Stopping User Runtime Directory /run/user/0...
Jan 13 16:50:13 fra systemd[1]: run-user-0.mount: Succeeded.
Jan 13 16:50:13 fra systemd[1]: user-runtime-dir@0.service: Succeeded.
Jan 13 16:50:13 fra systemd[1]: Stopped User Runtime Directory /run/user/0.
Jan 13 16:50:13 fra systemd[1]: Removed slice User Slice of UID 0.
Jan 13 16:50:13 fra systemd[1]: user-0.slice: Consumed 1.059s CPU time.
Jan 13 16:50:13 fra pve-ha-crm[1218]: service 'vm:400': state changed from 'started' to 'freeze'

 

[fabian]

how did you simulate the node failure? see https://pve.proxmox.com/pve-docs/chapter-ha-manager.html , especially https://pve.proxmox.com/pve-docs/chapter-ha-manager.html#ha_manager_shutdown_policy

 

[chrispage1]

before I venture into ceph, where I already know I'm going to cry

We've started using Ceph ourselves and I have to say how good it is. The particular benefit is in the case of failover it'll pick up right where it left off, the cluster is self healing and VM migrations are wickedly fast as there's no disk to migrate. In our test environment I tried my best to break it but really couldn't!

You do however use the ability to use replication - but then that makes sense.

Just be careful with the hardware you're running as you'll need good hard disks and a decent amount of memory/compute.

 

[lordhanuman]

Hello. I put it into production, and to avoid problems I use the ssd nvme in order to have troughput.
I went crazy setting up OVH vracks decently.
When I finish everything maybe I put the configuration here so that it can be useful to someone without them going crazy like I did.