[SOLVED] ZFS rpool names and Storage Replication

[Matt Kelley]

I’m trying to figure out the preferred hardware / software setup for proxmox in my case . I’d like to be able to use the awesome storage replication features as well as be able to freely use migration across cluster nodes using the GUI.


My hardware:

2x 8-bay Dell R520s (already own)

Planned setup – 2x SSD - ZFS Raid 1 for OS, 6x 4TB HDD - ZFS Raid Z2 for VMs

2x 4-bay Dell R420s (already own)

Planned setup – 4x HDD - ZFS Raid Z2 for OS and VMs


The problem:

The Proxmox installer gives both the R520 SSDs and R420 HDDs zfs pool names rpool but my 2nd ZFS zpool for my R520s (where I’d like to store the VMs) doesn’t match the storage ID or zpool name so I can’t get Storage Replication or migration working from the R520s to the R420s.


R520 – rpool (OS) vm_tank (VM Images)

R420 – rpool (OS and VM Images)


I realize I could just go with all HDDs in the R520 and do Raid ZFS Z2 but I really like the idea of isolating the OS hard drives where its hardware feasible.

Thoughts?

Am I missing something obvious?

I've read
https://pve.proxmox.com/wiki/Storage_Replication
https://pve.proxmox.com/wiki/Storage
and searched the forums and haven't been able to figure it out.

 

[wolfgang]

Hi,

replication only works on the same zfs pool.
There are no settings where you can change the storage.

 

[Matt Kelley]

Is there a preferred way then between the two? Should I just Raid Z2 everything and keep OS and VM images together or should I get more 8 drive servers and pursue the OS and VM pool separation?

 

[wolfgang]

I think it is always better to separate the OS from the Data.
Also for performance and reliability,
the Guest is not able to interfere with the OS by huge Disk usage.

 

[Matt Kelley]

Thanks wolfgang! Last question - What's the preferred way for me to do the OS partition? I thought ZFS mirroring on SSD was a good idea but I have seen some people suggest that it's not since ZFS doesn't support trim yet. But now my raid card is in IT mode so I can't use it to mirror the drives in hardware and proxmox only supports ZFS for software raid.

 

[guletz]

Hi,

Nice problem to solve, ....

For the botth servers:

- install pmx using only one ssd with zfs (raid0)
- add the hdd's as you planed
- create on each server on any hdd 2 partion (A = the same size as ssd, B = rest of hdd)

Now you have only rpool on ssd, an the hdd's.
- create on both servers using B partions your desired raidz2 (pool name = data-pool for example )
- create on both servers a new boot zfs pool (name like rpool2) using A partions (mirror)

Now you have the same pools on both servers. The trick is then you can use zfs send/recive from rpool to rpool2. Also you can install grub on all of yours hdd's.

And if any ssd is broken, then you could boot with pmx live cd, import your zfs rpool2, rename rpool2 to rpool, and reboot your server. I will boot using rpool from hdd's!

Good luck!

 

[wolfgang]

but I have seen some people suggest that it's not since ZFS doesn't support trim yet

Trim is overrated.
All SSD controller in the last 5 years is capable to do this autonomy.
And in the worse case, you loos a bit performance what is unimportant for the OS.
Anyway, trim will come next ZFS release.

create on each server on any hdd 2 partion (A = the same size as ssd, B = rest of hdd)

I know this is only for failover but ZFS should use the hole partition.
If you have both pools on this disk in use the performance will drop.

 

[Matt Kelley]

Just an update if anyone is curious:
I realized that if I really wanted OS and data separation I needed to get the SSDs to mount inside my R420 and not take up any of the four front hotswap bays.

I wound up with two of these: https://www.amazon.com/gp/product/B07GDDVKX8/
connected to my motherboard with: https://www.amazon.com/gp/product/B01KFEVQ3K/
Using Intel SATA M.2 SSD drives we already had.
The PCIE card here is really just providing power and a place to mount the SSDs

Important thing to note: Don't get NVME drives. Proxmox can't install the OS drive on mirrored NVME drives. Use SATA M.2 drives.

Once I got proxmox installed this way I was able to have my zpool names match and get storage replication rocking away.

 

[guletz]

Thx. Matt.