PVE Replication with ext4

[insel09]

Hello.

I have an existing PVE node, which will be integrated into a PVE cluster. Installation is based on ext4 as filesystem. I want to use replication. Is this (really) not possible with ext4 / PVE? (It seems, replication is not possibe with Proxmox Backup Server.)

Thanks in advance,
Sebastian

 

[Johannes S]

No, since it needs a filesystem-specific feature which ext4 or XFS simply doesn't have. In theory ut would je possible with btrfs but at the moment ProxmoxVE storage replication only works with ZFS.

I would backup all vms/lxcs on your node ( just to be safe), migrate them to the cluster with Proxmox Datacenter Manager or the remote-migrate feature of the qm/pct cli tools and afterwards reinstall the nodecwith ZFS before joining it to the cluster. Afterwards migrate/restore the vms/lxcs back to the node.

 

[insel09]

Thank you, @Johannes S. Next step is to verify, that the hardware raid controller can be used with ZFS. So far it seems, that the HPE Smart Array P816i-a SR Gen10 can be used in "HBA-Mode", so ZFS will see the disks without the HW RAID in between. That could have gone wrong.

 

[waltar]

Don't think that's the truth as you want replication for your virtual machines and if they sit on zvols of option disks replication should work fine even with pve on ext4 installed ... so where does that expression come from "isn't possible" ?!

 

[Johannes S]

Don't think that's the truth as you want replication for your virtual machines and if they sit on zvols of option disks replication should work fine even with pve on ext4 installed ... so where does that expression come from "isn't possible" ?!

Of course OP could install PVE on an ext4 or xfs formatted boot drive and ZFS on another storage which he could use for storage replication. But for this to work OP need a zfs pool in the first place. If I understood @insel09 correct his server has a RAID controller in RAID mode for all his disks thus no ZFS pool at the moment

 

[alexskysilk]

Installation is based on ext4 as filesystem. I want to use replication.

Your root file system doesnt really matter for the purposes of this discussion. only the vm storage does. Assuming you intend to use the same filesystem for your OS and payload, you cant use ZFS replication- but that doesnt mean you cant accomplish this anyway- just maybe not the way you think.

(It seems, replication is not possibe with Proxmox Backup Server.)

Why not? just backup and restore. if you'd like to skip the middle part where a backup file is created, you can create a snapshot and send it to the other node although this needs some scripting.

 

[insel09]

TL;DR: I will re-install PVE on top of plain zfs.

To give you an update of missing / unclear settings: The HW RAID handles all disks (at the moment), presenting only one disk to PVE. PVE installation and VM storage is on this single disk with ext4 as filesystem. No zfs for VM storage. (As all disks are handled by the HW RAID controller this - an additional zfs pool on top of the HW RAID - would be a bad idea anyway.)

Next steps will be: Remove the HW RAID, present all disks directly to the OS. Re-Install PVE on top of zfs RAID 10. PVE and VM images will be on zfs.

 

[insel09]

@alexskysilk Actually i was thinking of "Hey, create a backup with PBS and trigger a restore on the target as soon as the backup is done". But i think, the replication feature of PVE is the more common and "correct" way in doing replication from A to B with Proxmox.

I was looking for an integrated replication feature on PBS because we use a different backup sollution for a different virtualization system [1] at the moment. There you can create a replication job: I want this VM to be replicated to this server. I thought, this could/should be possible with PBS too. Maybe it would be a nice feature for PBS. Just for all, who can not use zfs as VM storage for whatever reason.

[1] I don't know the policy of nameing competition products/vendors, but i'm sure you have an idea, what i'm talking about.

 

[alexskysilk]

@alexskysilk Actually i was thinking of "Hey, create a backup with PBS and trigger a restore on the target as soon as the backup is done". But i think, the replication feature of PVE is the more common and "correct" way in doing replication from A to B with Proxmox.

I think this is the part of the conversation where we investigate the WHY. If your intention is to live migrate without shared storage, any CoW backed storage (zfs, qcow, even lvm with snapshot) would work and doesnt require any additional steps (cloning manually.) If the reason is something else, cloning disks to another server without a shared file system is enough of a edge case not to warrant the dev team investing effort in it. Perhaps you can explain so they see the value.