New Proxmox setup - Disk setup

int-tech01

New Member
Oct 24, 2023
11
0
1
Hello all,

I have been lurking on here for quite some time, and have setup Proxmox on a few nodes already. I am about to install 2 new PM nodes and have a few questions.

I will have 2 nodes. They EACH will have enough storage and RAM to handle all workloads from both nodes. In other words, for maintenance purposes for proxmox (updates, etc), I want to be able to migrate all workloads from 1 node to another. A blip or a few seconds of downtime at the end of the migration is perfectly fine.

I would like to be able to manage these from 1 web interface. (cluster, without shared storage) In order to do a migration from 1 node to another node, do I need to have ZFS as the storage type on both nodes? Or can I do migrations simply using LVM?

Speaking of ZFS: Traditionally I have used Dell servers with RAID cards for all of my workloads. I have read around on the forums and in the admin guide for Proxmox that you should not use hardware RAID with ZFS. These 2 nodes will be going into a data center where I can utilize Remote Hands if there is a problem with the node such as a failed disk. With traditional hardware RAID, I can easily tell Remote Hands to go and rip and replace a specific drive. How is this handled if I would choose ZFS? I assume that replacing a failed drive with ZFS is not as straigh forward?

Thanks in advance. Looking forward to learning more about Proxmox and supporting the project.
 
Use ZFS, configure storage replication. It will regularly snapshot your running vms and sync the difference to the other side. So the migration will cause minimum downtime and your data is replicated.
Without the storage replication at migration you will have to copy the complete data, which will take a long time
 
  • Like
Reactions: int-tech01
ZFS won't work with hardware RAID.
If you are not comfortable with ZFS, you can use hardware RAID, but you can't then setup replication.
But hardware controller might need software setup, for managing raid/disks, so there is a tradeoff.
You can stick to what you know best, but replacing disk is not hard, but maybe not as straightforward as what you know.

ZFS replication speeds up migration, but I think it's not as reliable as shared storage.

If uptime is a high concern, I guess backup is also very important: what is your backup strategy ?

Also Proxmox updates don't usually require reboot.
So how often do you anticipate rebooting servers ?
 
I'd rather stay away from shared storage, as it just adds another layer of complexity. It can absolutely be great, but unnecessary for this workload.

I am OK with learning ZFS. I've used it in the past, but only for home projects and I have not had to deal with failed drives with ZFS. So I would just need to learn it. My backup strategy is to use Proxmox Backup server. It works fanstastic!

So to be clear, without ZFS, you cannot even migrate to another node? You mentioned replication, but are you stating that you are not able to do even a migration to another node without doing a backup/restore?
 
Without ZFS replication live migration will be very slow, I think it pauses or stops VM, copies data across, and starts remote VM.
There is no replication, so it heavily depends on VM size.
To be honest I have not tried it recently; last time I tried I concluded it was much faster to backup and restore, especially with a fast Proxmox Backup server.
I think it's all in what's acceptable downtime.
 
We will be hosting various workloads such as a couple websites, a SQL server and a couple of other items. I would like to minimize downtime as much as possible. A couple of minutes in the late evening is perfectly acceptable. It sounds like zfs is the way to go. I don't know enough about ZFS on top of hardware RAID to even make comments here, but I've read on another forum that someone setup their disks on hardware raid, then in Proxmox, created a 1 disk ZFS, and presented that hardware raid "disk" to Proxmox. They were then able to do replication/migration, but also had the benefits of being able to still do blind swaps, etc with hardware raid. What are your thoughts on that approach?
 
If you do not have onboard Sata ports you can use the raid Controller in JBODMode, so it presents Single disks to the os but it it not good because you do not what caching the Controller is doing, it might destroy your data
 
cat documentation/proxmox/zpool-replace-disk.md

Get Zpool Status:

zpool status


Copy Partitions from working to new disk:

sfdisk -d /dev/nvme2n1 |sfdisk /dev/nvme3n1

Replace Disk, give ZFS Partition
zpool replace zp_pve 1414275488282 /dev/disk/by-id/nvme-SAMSUNG-part3


Check status, should resilver:
zpool status


Rewite Bootloader:
proxmox-boot-tool format /dev/disk/by-id/nvme-SAMSUNG-part2
proxmox-boot-tool init /dev/disk/by-id/nvme-SAMSUNG-part2
proxmox-boot-tool status

Clean /etc/kernel/proxmox-boot-uuids

proxmox-boot-tool status
proxmox-boot-tool refresh
proxmox-boot-tool clean
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!