should I store my VM on a RAID?

B

boris2000

New Member
Feb 15, 2025
4
0
1
Hi,

I have my home lab where I deploy my applications in VM and plan to extend that to do web hosting.
I currently purchase 4 x 2TB 980 SSD NVMe to install on a PCI hub interface, 1TB Crucial SSD MX500 for the OS. Here is my plan:

  1. Install the OS on the 1TB drive
  2. use the 2TB SSD for storage. So I will have 4 storage volumes.
  3. purchase a 16TB HDD for storing other stuff that needs space, for example, configure it as a NAS.
Since I plan to start using it commercially, here are my questions:
  • What did you think of the volume setup? Any advice, please?
  • Is it worth using RAID 10 on the 4x2TB? that will reduce the usable storage to 4TB for VM disks.
  • Should I consider other storage options?

Server Spec: Dell Precision Tower 5810, 98GB RAM, 1 Intel Xeon CPU - 16 cores. Will upgrade if the business goes well. ;)
 
I would advise going with a better SSD for the OS if you're planning on running 24/7. MX500 is consumer-level and may wear out faster than you'd expect, or be I/O or temperature constrained after a certain amount of data transfer. Proxmox recommends Enterprise-level SSD, but for homelab you can get away with a high TBW rating.

ZFS "RAID10" (striped mirrors) would definitely be recommended for the VM backing storage.

UdoB

[TUTORIAL] Thread 'FabU: Can I use ZFS RaidZ for my VMs?'

Assumption: you use at least four identical devices for that. Mirrors, RaidZ, RaidZ2 are possible - theoretically.

Technically correct answer: yes, it works. But the right answers is: no, do not do that! The recommendation is very clear: use “striped mirrors”. This results in something similar to a classic Raid10.

(1) RaidZ1 (and Z2 too) gives you the IOPS of a single device, completely independent of the actual number of physical devices. For the “four devices, mirrored” approach this will double --> giving twice as many Operations per Second. For a large-file...
  • Like
 
  • Like
Reactions: boris2000 and Johannes S
Kingneutron said:
I would advise going with a better SSD for the OS if you're planning on running 24/7. MX500 is consumer-level and may wear out faster than you'd expect, or be I/O or temperature constrained after a certain amount of data transfer. Proxmox recommends Enterprise-level SSD, but for homelab you can get away with a high TBW rating.

ZFS "RAID10" (striped mirrors) would definitely be recommended for the VM backing storage.

UdoB

[TUTORIAL] Thread 'FabU: Can I use ZFS RaidZ for my VMs?'

Assumption: you use at least four identical devices for that. Mirrors, RaidZ, RaidZ2 are possible - theoretically.

Technically correct answer: yes, it works. But the right answers is: no, do not do that! The recommendation is very clear: use “striped mirrors”. This results in something similar to a classic Raid10.

(1) RaidZ1 (and Z2 too) gives you the IOPS of a single device, completely independent of the actual number of physical devices. For the “four devices, mirrored” approach this will double --> giving twice as many Operations per Second. For a large-file...
  • Like
Click to expand...
Great thanks.

Will go with raidz-2 and with time, I will see how I can add another 4x2TD SSD.

Hope the live extensions will not be destructive for the VM.
 
Raid z2 is going to slow you down. In my opinion best practice would be to put your OS on a small set of mirrored SSD or NVME drives. put your VMs on a larger mirrored pair if SSDs or NVMEs, and store your data on another set of drives. Raid or mirror is appropriate for the data. For a production environment you want your boot/OS drives, and VM drives to be on enterprise grade disks with power loss protection and error correction. Data can go on SSD, NVMe or spinning rust.
 
  • Like
Reactions: Kingneutron and Johannes S
mx500 as ext4 for PVE OS will be fine.
it's ZFS will wearout SSD due to write amplification.
ZFS on 980 will be slow and quickly wearout.
ZFS only for datacenter drives.

I would use as 2 Lvmthin volumes storage + the 2 others as backups as PBS ext4 datastores, with PBS installed alongside PVE as it's homelab (if you know what you do, it's fast and robust).
 
louie1961 said:
Raid z2 is going to slow you down. In my opinion best practice would be to put your OS on a small set of mirrored SSD or NVME drives. put your VMs on a larger mirrored pair if SSDs or NVMEs, and store your data on another set of drives. Raid or mirror is appropriate for the data. For a production environment you want your boot/OS drives, and VM drives to be on enterprise grade disks with power loss protection and error correction. Data can go on SSD, NVMe or spinning rust.
Click to expand...
Hi, thank for the help. Just to clarify:

1- by mirrored disk, you means RAID 1 ? So the VM should go on a large pair of raid 1 disk.?
2. By entreprise grade disk, you means SAS ?
3. Why do raidz 2 will slow me down ?
 
_gabriel said:
mx500 as ext4 for PVE OS will be fine.
it's ZFS will wearout SSD due to write amplification.
ZFS on 980 will be slow and quickly wearout.
ZFS only for datacenter drives.

I would use as 2 Lvmthin volumes storage + the 2 others as backups as PBS ext4 datastores, with PBS installed alongside PVE as it's homelab (if you know what you do, it's fast and robust).
Click to expand...
I’m new to proxmox, I have been using libvirt before. So please help me understand what you are sharing.

1. So the 980 are not good enough for ZFS ? Which disk will you recommend?
2. Why lvmthin ?
3. What are you backing up in PBS ? The data or the vms disk entirely?
 
1/ indeed IMO. please search and read the forum, there is plenty of topics for advices.
2/ Because it's the default if not ZFS.
ZFS isn't a simple filesystem.
The learning curve is steeper than you expect.
3/ all data (VMs included) must have a backup.
 
boris2000 said:
Hi, thank for the help. Just to clarify:

1- by mirrored disk, you means RAID 1 ? So the VM should go on a large pair of raid 1 disk.?
2. By entreprise grade disk, you means SAS ?
3. Why do raidz 2 will slow me down ?
Click to expand...
Mirrored is mirrored. It will say that in the Proxmox installer. It is analogous to raid 1, but in ZFS its not really the same. By enterprise I do not necessarily mean SAS. There are SATA enterprise SSDs. These are just one example (not an endorsement): https://www.newegg.com/p/N82E16820318074

Enterprise drives have features like power loss protection, error correcting code, end-to-end data protection, and SIGNIFICANTLY higher endurance (drive writes per day or terabytes written) than consumer drives.

RaidZ is simply known to be slower than mirrored drives in ZFS. It has to do with things like computing parity and block sizes, etc.
https://www.raidz-calculator.com/raidz-types-reference.aspx
 
  • Like
Reactions: Kingneutron and _gabriel
uberarchiver said:
And PLP SSDs also lose much of their meaning with SLOG device.
Click to expand...

Does it? I am not convinced. But yeah, a slow SSD is usually (but not always and in any case) faster than a HDD - if that's what you mean.

An SLOG helps only with SYNC writes. The very most of my VMs (or the services inside them) do write asynchronous. (Exception: databases.)

On the other hand PLP helps a lot to increase actual IOPS. And this is what running multiple VMs needs to run smoothly. (Guaranteed data integrity is only one feature of PLP.)
 
  • Like
Reactions: Kingneutron
Unexpected power loss during trim (when the SSD is shuffling data around) can cause data to disappear completely. Unexpected power loss during a write can cause other data to get lost (as the SSD is shuffling data around). A mirror of SSDs don't fix this issue if they are both trimming or otherwise doing the exact same thing.
SSDs handle power loss much worse than HDDs. It's not just the data in flight but also the data that is (supposed to be) at rest. PLP solved these kinds of problems and give a nice durability and performance boost (but mostly with sync writes) as a bonus.
 
  • Like
Reactions: Retro1982, Johannes S, UdoB and 1 other person
All of this speculation about PLP and data loss is pretty much moot considering the topic of the "home lab" the writer did write in the first post though.

I do not know about your specific workloads in your homelabs but I for one do not handle any critical data in my homelab.

Rgds,

;)
 
UdoB said:
Does it? I am not convinced. But yeah, a slow SSD is usually (but not always and in any case) faster than a HDD - if that's what you mean.

An SLOG helps only with SYNC writes. The very most of my VMs (or the services inside them) do write asynchronous. (Exception: databases.)

On the other hand PLP helps a lot to increase actual IOPS. And this is what running multiple VMs needs to run smoothly. (Guaranteed data integrity is only one feature of PLP.)
Click to expand...
Considering that I did not (mean to) comment on anything other than the write amplification part, I still think that my comment is spot on. Using SLOG *will* help with exactly nullifying the write amplification concern the original writer had.
 
uberarchiver said:
All of this speculation about PLP and data loss is pretty much moot considering the topic of the "home lab" the writer did write in the first post though.

I do not know about your specific workloads in your homelabs but I for one do not handle any critical data in my homelab.
Click to expand...

This depends on what your "homelab" is. Do you just use it for learning stuff and not having anything critical on it? Then of course than anything goes (this is the situation for my two-node + qdevice cluster: I don't have anything important on it, even the two pi-holes are not really needed, because I can always switch back to my ISPs DNS). If however you have beloved holiday photos in a nextcloud or immich instance and your tax files in a paperless-ngx container the stuff get's way more important. For this reason my important stuff is on my NAS which is a VM on a single-node ProxmoxVE installation without any relation to the cluster (except that the clusters qdevice is a VM on the same single-node install as my NAS) ;)

And another thing to consider: Used enterprise sata or sas ssds are quite affordable and last way longer than consumer ssds so in the long run I wouldn't be so sure what's more feasible from a economical perspective.

I myself use enterprise SATA ssds in ZFS mirrors together with consumer M2 2280 NVMEs. The reason is simple: My mini-pcs only have one SATA and one 2280 slot and sadly there ain't any 2280 NVME for a reasonable price on ebay. So I decided to live with planning to replace the NVMEs more often. The ZFS mirror should still allow to keep the hassle to a reasonable limit (aka turning off the node, replacing the old nvmes, turning node back on, telling ZFS to use the new device).
 
Last edited:
  • Like
Reactions: UdoB, Retro1982 and boris2000
So let's dig deeper into the rabbit hole?

Consider this: I have very bad experiences with consumer SSDs. Of all that have failed on me, all have gone without any notification ahead of time.
So, I would not wager a bit on them doing a good job preserving my data in the long run.

That's one thing we seem all to agree on then: Consumer SSDs are pretty much crap.

I have 3 enterprise class flash based drives: 1 pcie nvme and 2 SAS SSDs. Of those I can not say much, yet.

The bulk of my storage is 12 over 10 year old Seagate HDDs. They have had SMART reporting faulty or reallocated sectors. Multiple times.
The state seems to linger for a while and then it goes away. I think its because of the CoW part of ZFS. And yeah, I use ZFS.

My setup is 3x4HDD RAIDZ1, 3xSSD special device in triple mirror (no small block stuff), SLOG (WD RED 1TB M.2, for now, 8GB used), 2 SAS SSDs for L2ARC and a spare HDD.

I have a dedicated Proxmox Backup Server where I run daily backups of nearly all my VMs and stuff. It stores backups in a mirror on HDDs.

Because of how RAID and stuff works, I came to the conclusion that if I were to use SSDs for the bulk of storage (Data Drives) I could very possibly run into a case were ALL of those drives would run out of their TBW endurance at the same time. I did not like that thought. Not one bit.

So I am using those crappy old drives that seem to just work and boost their performance by other means.

And this is because I myself have very crappy experiences with flash based storage. Probably not with (in the future) enterprise grade flash.

The 3 SSDs as triple mirror are Kingston DC600Ms.

And I have "backups" on Dropbox AND Onedrive of my family photos in addition to the PBS.

And I will not say what exactly anyone should do or not do. That is the rabbit hole for every DIYer to explore.

And still. The original poster did not mention any critical stuff in the original post. Although if going commercial is his/hers/its intentions I would advice running the PBS for backups. You will love it.
 
  • Like
Reactions: UdoB, Johannes S and boris2000
You must log in or register to reply here.
Top Bottom