New Proxmox setup - Disk setup

[int-tech01]

Hello all,

I have been lurking on here for quite some time, and have setup Proxmox on a few nodes already. I am about to install 2 new PM nodes and have a few questions.

I will have 2 nodes. They EACH will have enough storage and RAM to handle all workloads from both nodes. In other words, for maintenance purposes for proxmox (updates, etc), I want to be able to migrate all workloads from 1 node to another. A blip or a few seconds of downtime at the end of the migration is perfectly fine.

I would like to be able to manage these from 1 web interface. (cluster, without shared storage) In order to do a migration from 1 node to another node, do I need to have ZFS as the storage type on both nodes? Or can I do migrations simply using LVM?

Speaking of ZFS: Traditionally I have used Dell servers with RAID cards for all of my workloads. I have read around on the forums and in the admin guide for Proxmox that you should not use hardware RAID with ZFS. These 2 nodes will be going into a data center where I can utilize Remote Hands if there is a problem with the node such as a failed disk. With traditional hardware RAID, I can easily tell Remote Hands to go and rip and replace a specific drive. How is this handled if I would choose ZFS? I assume that replacing a failed drive with ZFS is not as straigh forward?

Thanks in advance. Looking forward to learning more about Proxmox and supporting the project.

 

[ubu]

Use ZFS, configure storage replication. It will regularly snapshot your running vms and sync the difference to the other side. So the migration will cause minimum downtime and your data is replicated.
Without the storage replication at migration you will have to copy the complete data, which will take a long time

 

[niteshadow]

ZFS won't work with hardware RAID.
If you are not comfortable with ZFS, you can use hardware RAID, but you can't then setup replication.
But hardware controller might need software setup, for managing raid/disks, so there is a tradeoff.
You can stick to what you know best, but replacing disk is not hard, but maybe not as straightforward as what you know.

ZFS replication speeds up migration, but I think it's not as reliable as shared storage.

If uptime is a high concern, I guess backup is also very important: what is your backup strategy ?

Also Proxmox updates don't usually require reboot.
So how often do you anticipate rebooting servers ?

 

[int-tech01]

I'd rather stay away from shared storage, as it just adds another layer of complexity. It can absolutely be great, but unnecessary for this workload.

I am OK with learning ZFS. I've used it in the past, but only for home projects and I have not had to deal with failed drives with ZFS. So I would just need to learn it. My backup strategy is to use Proxmox Backup server. It works fanstastic!

So to be clear, without ZFS, you cannot even migrate to another node? You mentioned replication, but are you stating that you are not able to do even a migration to another node without doing a backup/restore?

 

[niteshadow]

Without ZFS replication live migration will be very slow, I think it pauses or stops VM, copies data across, and starts remote VM.
There is no replication, so it heavily depends on VM size.
To be honest I have not tried it recently; last time I tried I concluded it was much faster to backup and restore, especially with a fast Proxmox Backup server.
I think it's all in what's acceptable downtime.

 

[int-tech01]

We will be hosting various workloads such as a couple websites, a SQL server and a couple of other items. I would like to minimize downtime as much as possible. A couple of minutes in the late evening is perfectly acceptable. It sounds like zfs is the way to go. I don't know enough about ZFS on top of hardware RAID to even make comments here, but I've read on another forum that someone setup their disks on hardware raid, then in Proxmox, created a 1 disk ZFS, and presented that hardware raid "disk" to Proxmox. They were then able to do replication/migration, but also had the benefits of being able to still do blind swaps, etc with hardware raid. What are your thoughts on that approach?

 

[ubu]

If you do not have onboard Sata ports you can use the raid Controller in JBODMode, so it presents Single disks to the os but it it not good because you do not what caching the Controller is doing, it might destroy your data

 

[ubu]

Replacing a failed zfs drive is easy, i will post my instructions tomorrow.

 

[int-tech01]

Replacing a failed zfs drive is easy, i will post my instructions tomorrow.

Thank you

 

[niteshadow]

ZFS on top of hardware RAID (ZFS single-drive RAID 0) is probably OK, but not as reliable as regular ZFS.
It's probably more reliable than using EXT4, but you miss some important features:
https://openzfs.github.io/openzfs-d...uning/Hardware.html#hardware-raid-controllers

 

[ubu]

cat documentation/proxmox/zpool-replace-disk.md

Get Zpool Status:

zpool status


Copy Partitions from working to new disk:

sfdisk -d /dev/nvme2n1 |sfdisk /dev/nvme3n1

Replace Disk, give ZFS Partition
zpool replace zp_pve 1414275488282 /dev/disk/by-id/nvme-SAMSUNG-part3


Check status, should resilver:
zpool status


Rewite Bootloader:
proxmox-boot-tool format /dev/disk/by-id/nvme-SAMSUNG-part2
proxmox-boot-tool init /dev/disk/by-id/nvme-SAMSUNG-part2
proxmox-boot-tool status

Clean /etc/kernel/proxmox-boot-uuids

proxmox-boot-tool status
proxmox-boot-tool refresh
proxmox-boot-tool clean

 

[int-tech01]

Thank you for the instructions!