Advice on Setting Up Redundant RAID with SATA SSD for Proxmox Server

[Bravestarr]

Hello everyone,

I'm currently running a Proxmox server that has been operating smoothly with a 1 TB NVMe SSD. I'm considering enhancing my server's data safety by adding a new 1 TB SATA SSD and setting it up in a redundant RAID configuration. Given the differences in performance between NVMe and SATA SSDs, I'm seeking advice on the best way to achieve this.

My main goals are to ensure data redundancy and to improve the overall resilience of my server setup without significantly compromising performance. Here are a few specific questions I have:

1. RAID Configuration: What RAID level would you recommend for achieving redundancy with these two different types of SSDs (NVMe and SATA)? I'm leaning towards RAID 1 for its simplicity and mirroring capabilities, but I'm open to suggestions if there's a better approach.
2. Performance Concerns: How significant is the performance impact when pairing an NVMe SSD with a SATA SSD in a RAID setup? I understand that the slower speed of the SATA SSD could potentially limit the overall performance, but I'm curious about real-world experiences and how much of a bottleneck this might actually be.
3. Best Practices: Are there specific considerations or best practices I should follow when setting up a RAID array with mixed SSD types on Proxmox? For example, are there settings or configurations that can help minimize performance losses or ensure smoother operation?
4. Alternatives: If a redundant RAID with these two SSDs is not advisable, what other strategies might I consider for enhancing data safety and server resilience? I'm open to exploring other hardware or software solutions that achieve similar goals.

I appreciate any insights, experiences, or advice you can share. My priority is to make an informed decision that balances data safety with maintaining good server performance. Thank you in advance for your help!

 

[Dunuin]

RAID Configuration: What RAID level would you recommend for achieving redundancy with these two different types of SSDs (NVMe and SATA)? I'm leaning towards RAID 1 for its simplicity and mirroring capabilities, but I'm open to suggestions if there's a better approach.

Not great because the NVMe SSD will be slowed down to SATA performance. But there is no other option with only 2 disk when you want redundancy.

Performance Concerns: How significant is the performance impact when pairing an NVMe SSD with a SATA SSD in a RAID setup? I understand that the slower speed of the SATA SSD could potentially limit the overall performance, but I'm curious about real-world experiences and how much of a bottleneck this might actually be.

Every write will have to be written to and acknowledged by both SSDs. If the NVMe SSD finishes this first, all has to wait for the SATA SSD to finish too.

Best Practices: Are there specific considerations or best practices I should follow when setting up a RAID array with mixed SSD types on Proxmox? For example, are there settings or configurations that can help minimize performance losses or ensure smoother operation?

Best practise would be to buy a platform that supports enough disks of the same performance class in the first place.

Alternatives: If a redundant RAID with these two SSDs is not advisable, what other strategies might I consider for enhancing data safety and server resilience? I'm open to exploring other hardware or software solutions that achieve similar goals.

I would prefer a slow raid1 over a fast single disk for everything you actually care about.

 

[Bravestarr]

Okay, thank you! What will happen when I install the SATA SSD? Will Proxmox ask me what to do and guide me on how to add it to the RAID 1?

 

[leesteken]

Not great because the NVMe SSD will be slowed down to SATA performance.

I my experience, it's not that bad (with recent ZFS versions). I've run some quite uneven mirrors in the past, like consumer SATA SSD + laptop HDD and enterprise NVMe + SATA SSD, where (random) reads appear to go to mostly to the fastest of the two.

Okay, thank you! What will happen when I install the SATA SSD? Will Proxmox ask me what to do and guide me on how to add it to the RAID 1?

Not at all; you'll have to do it yourself. But it's not really Proxmox specific, so you can probably follow other Linux guides (or the ZFS examples in the Proxmox manual).

 

[Bravestarr]

Can you please point me to a tutorial I can follow? Is it hard and can it be done without formatting the existing NVMe?

 

[sg90]

Can you please point me to a tutorial I can follow? Is it hard and can it be done without formatting the existing NVMe?

You would need to wipe and start from fresh as the underlying setup of the Filesystem would need to change from what ever your currently running to ZFS.

You'd need to backup your VM's wipe and run through the installer to setup ZFS: https://pve.proxmox.com/wiki/ZFS_on_Linux

 

[Bravestarr]

Oh no. And Proxmox can not do this on its own? But this is really bad news. I have a backup but will all VMs have the same IP addresses?

 

[Bravestarr]

And is there a monitoring when one of the Ssds fail?

 

[leesteken]

Can you please point me to a tutorial I can follow? Is it hard and can it be done without formatting the existing NVMe?

Not really, it depends on whether you used ZFS, Btrfs, LVM, Ext4, XFS, etc. As an enterprise systems administrator (for which Proxmox is intended) or someone who is interested to use Proxmox as a hobby, I expect you to do a little more searching yourself. But maybe someone else here will do the work for you.

 

[Bravestarr]

Hi, this is a Hobby project: https://marc.tv/proxmox-synology-homelab/

 

[sg90]

Oh no. And Proxmox can not do this on its own? But this is really bad news. I have a backup but will all VMs have the same IP addresses?

This is impossible to answer without knowing your setup. But there is 0 reasons why you wouldn’t be able to use the same IP.

However going by your limited expirence and fact this is a hobby project ZFS may not be the best option due to the extra complexity you will need to learn and manage.

However if the hobby project means there is no issues if things break and your using it to learn then the best way to learn is trial and error.

Backup your VM’s download the Proxmox installer and try and find.

 

[Bravestarr]

Okay. Thanks. I read this here: https://pve.proxmox.com/wiki/ZFS_on_Linux#_installation_as_root_file_system It seems like the installer will prompt me to use the two disks as RAID-1 if I am correct and that's it.

"The installer automatically partitions the disks, creates a ZFS pool called rpool, and installs the root file system on the ZFS subvolume rpool/ROOT/pve-1."

I read here that you need to install a Debian first and on top of that PVE: https://forum.proxmox.com/threads/raid-1-einrichten.101821/ But this seems to be outdated, right? So if I use the backup (NFS in on a NAS) this will be ... easy. Right?

 

[Dunuin]

Okay. Thanks. I read this here: https://pve.proxmox.com/wiki/ZFS_on_Linux#_installation_as_root_file_system It seems like the installer will prompt me to use the two disks as RAID-1 if I am correct and that's it.

Yes, installing raid1 using ZFS is simple. The harder part is to properly administrate it afterwards. What are you doing in case a disk dies? Are you able to optimize the ZFS option to match your hardware and workload? Are you sure your hardware can handle ZFS (you usually use ZFS if you care about your data and then you want to follow the recommendations and use lots of ECC RAM, Enterprise SSD with PLP and lots of TBW/DWPD to compensate the overhead, ...)? Do you really understand the concepts of datasets, vdevs, zvols and pools? Are you able to set up proper monitoring?

I read here that you need to install a Debian first and on top of that PVE: https://forum.proxmox.com/threads/raid-1-einrichten.101821/ But this seems to be outdated, right?

Thats about mdadm raid1 and not a ZFS mirror.

So if I use the backup (NFS in on a NAS) this will be ... easy. Right?

Depends. Your VM/LXC backups only contain the VMs/LXCs. All your hosts configs, optimizations and customizations will be lost. So the more you customize your server (firewall rules, monitoring agents, UPS agents, log collecting, security software, backup clients, ...whatever else you should install to increase security/reliability) the more painful it will be to set up PVE again.

 

[esi_y]

So I do not want to discourage the OP, but if I was doing the same I would try to answer (to myself) the following 2 questions:
1) Can I tolerate having this whole thing crumble for me given my current limited experience? Also a good reminder to have backups.
2) Do I plan to take up on learning more about ZFS internals, etc. as @Dunuin mentions in the post above or not at all.

If my score is not 2, I would use no mirror at all. I would have backups and run a simple ISO install. In fact I would not even run ZFS at all. And I would not run LVM either.

 

[Bravestarr]

Okay, after I read more about this I have a few questions. When I install Proxmox using USB, the installer prompts me to use a RAID. I thought it would not be possible with SATA and NVMe but it seems to be possible. When Proxmox is using this software RAID, it has to use ZFS. I am now on LVM:



So, if I want redundancy, I need to add a SATA because there is no 2nd NVMe slot available. You say this I need to learn about ZFS if I go down this route. I don't really understand why when Proxmox nudges me to make a RAID1 and even warns me during the installation when I only format one SSD. So what is precisely the problem when I use ZFS on my two SSDs? What do I have to learn? I thought Proxmox will inform me when the SMART values are bad and then I swap the disk.

 

[Dunuin]

I thought Proxmox will inform me when the SMART values are bad and then I swap the disk.

There is more than bad SMART values. ZFS has to do data integrity checks and the pool might degrade if there are too much read, write or checksum errors while SMART will report that everything is fine. And you for example can't swap a disk via GUI. This has to be done manually via CLI. And there won't be a notification when a disk starts to fail unless you set up some monitoring yourself like smartd, zfs-zed, zabbix or whatever.

 

[esi_y]

There is more than bad SMART values. ZFS has to do data integrity checks and the pool might degrade if there are too much read, write or checksum errors while SMART will report that everything is fine. And you for example can't swap a disk via GUI. This has to be done manually via CLI. And there won't be a notification when a disk starts to fail unless you set up some monitoring yourself like smartd, zfs-zed, zabbix or whatever.

I would set up an NVMe for my system drive and keep backups at all times. When your drive fails with no SMART telling you beforehand, you will appreciate them. If you get silent corruption (data that have changed despite you didn't get any sign they have), you will appreciate it. If you run RAIDZ and you drop an accidental bad command into root shell, you will also be happy you had backups. I do not believe you need redundancy as you do not appear to have business continuity requirements and your understanding of "data safety" by the means of RAIDZ is incorrect. It does not mean that you put a drive in, maybe one day replace with a spare, and sleep like a baby. One more important thing, if you do not understand the underlying ZFS concepts and you have that situation (e.g. drive replacement), you may accidentally destroy the whole pool at that moment instead, so you will need ... backups.

@Dunuin And then there's block-cloning with ZFS and other such topics.

 

[Bravestarr]

Thanks. I was not aware that I get problems without a SMART warning. Okay, so you say it is better to stick with the current NVMe and the existing backups via NFS to my NAS everyday? This only backs up the VMs but not the Proxmox configuration. So in the case that the NVMe fails, i order a new one and reinstall proxmox and then, one by one, I restore the VMs?

 

[Dunuin]

So in the case that the NVMe fails, i order a new one and reinstall proxmox and then, one by one, I restore the VMs?

Yes.

 

[Bravestarr]

One question regarding this @Dunuin

"If you run RAIDZ and you drop an accidental bad command into root shell, you will also be happy you had backups."

I thought that the Proxmox System itself is not part of my backups. I seems like it is not possible to include the proxmox system in a backup. Is this correct? Look at my backup configuration pls:



Or does this contain the node itself? if so, how can I use this backup during installation?