[SOLVED] Choice of storage type for VMs (NVME, SATA SSD, HDD)

[xellos]

Hello all,

I am pretty new here and I am building my first ever Homelab to run proxmox.

My main use cases will be:
- Running development VMs for my personal projects: mainly Backend Development and Machine Learning
- A Plex server: I plan to attach my library from a NAS to a VM to keep storage separated from compute, so my media library won't be on the VM neither managed Proxmox, it's physically separated.
- Videos and photos processing: making proxies of 4k videos, same for pictures by making different resolution

So I don't need a lot of storage, 200gb per VM is more than enough for me. So I am wondering what storage to use to get the best performance possible for my VMs.

Right now I am looking at NVMEs as they are generally much much faster, not talking about dramless and QLC but high end consumer drives like the WD Black 850X and the Samsung 980 Pro or Crucial P5 Plus. I am thinking to build a pure NVME proxmox node, and keep large files on the NAS connected over a 2.5Gbit network.

Do you guys think these kind of drives are the best to get? Correct me if I am wrong but I read that random r/w and iops are the golden metrics when it comes to virtualization, do these drives have great random performance? Are there any old (affordable) enterprise SSDs that I can get and perform better?

If you have any recommendations of drives I'd be grateful.

Also, is it possible to passthrough a drive as a boot drive of a VM? Not sure if it's a good practice but just wondering if it's doable in proxmox.

Thank you guys!

 

[Dunuin]

Hello all,

I am pretty new here and I am building my first ever Homelab to run proxmox.

My main use cases will be:
- Running development VMs for my personal projects: mainly Backend Development and Machine Learning
- A Plex server: I plan to attach my library from a NAS to a VM to keep storage separated from compute, so my media library won't be on the VM neither managed Proxmox, it's physically separated.
- Videos and photos processing: making proxies of 4k videos, same for pictures by making different resolution

So I don't need a lot of storage, 200gb per VM is more than enough for me. So I am wondering what storage to use to get the best performance possible for my VMs.

Right now I am looking at NVMEs as they are generally much much faster, not talking about dramless and QLC but high end consumer drives like the WD Black 850X and the Samsung 980 Pro or Crucial P5 Plus. I am thinking to build a pure NVME proxmox node, and keep large files on the NAS connected over a 2.5Gbit network.

Keep in mind that nested filesystem, virtualization, raid and copy-on-write filesystems like ZFS/qcow2/btrfs will cause a lot of overhead. And this overhead is't adding, it's multiplying and so the write amplification will grow exponentially. If you plan to write a lot of data (and 4K videos sounds like that) or you are using a lot of sync writes (running databases and so) it might be way cheaper to get enterprise SSDs. Don't only look at the cost per TB of capacity. Also have a look at the cost per TB of TBW (so price per TB that you can write to the SSD before losing the warranty because you caused too much wear).

And yes, SSD would be recommended, because IOPS are needed. And don't only look at the numbers. I can bring a super fast 6000MB/s consumer NVMe SSD down to single digit MB/s when hitting it hard with 4K random sync writes, where a old slow enteprise SATA SSD might perform way better, because it can cache sync writes because of the powerloss protection.

Also, is it possible to passthrough a drive as a boot drive of a VM? Not sure if it's a good practice but just wondering if it's doable in proxmox.

With NVMes you could use PCI Passthrough: https://pve.proxmox.com/wiki/Pci_passthrough

 

[xellos]

Hello @

Keep in mind that nested filesystem, virtualization, raid and copy-on-write filesystems like ZFS/qcow2/btrfs will cause a lot of overhead. And this overhead is't adding, it's multiplying and so the write amplification will grow exponentially. If you plan to write a lot of data (and 4K videos sounds like that) or you are using a lot of sync writes (running databases and so) it might be way cheaper to get enterprise SSDs. Don't only look at the cost per TB of capacity. Also have a look at the cost per TB of TBW (so price per TB that you can write to the SSD before losing the warranty because you caused too much wear).

And yes, SSD would be recommended, because IOPS are needed. And don't only look at the numbers. I can bring a super fast 6000MB/s consumer NVMe SSD down to single digit MB/s when hitting it hard with 4K random sync writes, where a old slow enteprise SATA SSD might perform way better, because it can cache sync writes because of the powerloss protection.


With NVMes you could use PCI Passthrough: https://pve.proxmox.com/wiki/Pci_passthrough

Hello Duniun,

Thanks a lot for the reply and the advices.

I have a question regarding the overhead you mentioned, do you mean the general virtualization overhead? Or you meant something specific to my use cases that I need to be aware of? And will that overhead eliminated with passthrough (not saying I will do thay for everything but just curious to know).

For consumer NVMEs vs. Used Enterprise SSDs, do you think the Pro versions like the Samsung 980 Pro are also not thay great? Also, do you have any affordable recommendations of Enterprise SSDs? I actually also considered Intel DC S3510 as I found 800Gb ones for 50€ and they still have +95% of life in them, what do you think about something like that?

Thanks again! Cheers

 

[Dunuin]

Hello @

Hello Duniun,

Thanks a lot for the reply and the advices.

I have a question regarding the overhead you mentioned, do you mean the general virtualization overhead? Or you meant something specific to my use cases that I need to be aware of? And will that overhead eliminated with passthrough (not saying I will do thay for everything but just curious to know).

PCI Passthrough will only prevent the virtualization Overhead. You might still get heavy overhead from filesystems like ZFS or raid.

For consumer NVMEs vs. Used Enterprise SSDs, do you think the Pro versions like the Samsung 980 Pro are also not thay great?

No, the prosumer SSDs are the same as consumer SSDs. Atleast for Samsung Pros which don't use any MLC NAND anymore. But all prosumer SSDs are still missing the powerloss protection, sync write performance and most of the durability.

Lets have a look at the rated write durability (TBW) of different grades of NVMe SSDs:

	TBW per 1TB capacity:	Price per 1TB capacity:	Price per 1TB TBW:
Cheap QLC Consumer (Crucial P3 1TB):	220 TB	71 €	0,322727 €
Normal TLC Consumer (Samsung 980 1TB):	600 TB	79 €	0.131667 €
Modern TLC Prosumer (Samsung 980 Pro 1TB):	600 TB	104 €	0.173333 €
End-of-Life MLC Prosumer (Samsung 970 Pro 1TB):	1,200 TB	220 €	0,183333 €
Cheap TLC Enterprise (Micron 7450 PRO 960GB)	1,771 TB	196 €	0.110672 €
Normal TLC Enterprise (Micron 7450 MAX 800GB):	5,375 TB	271 €	0.050419 €
Good MLC Enterprise (Intel P3700 800GB):	18,250 TB	1549 €	0.084877 €
High-end SLC Enterprise (Intel Optane P4801X 375GB):	109,333 TB	3296 €	0.030146 €

So basically...the more you pay for your SSD, the cheaper that SSD actually will be on the long run, because it is more durable and will survive longer. Only exception are the prosumer/NAS SSDs, which are just more expensive...So you don't have to wonder why ZFS is killing SSDs so fast, when buying cheap consumer SSDs with just a rated durability of 220 TB when there are good enterprise SSD that are rated to survive 109333 TB of writes.
Compare those numbers and you will realize how crappy those cheap consumer SSDs are, and that it might be a good idea to buy something more durable even if you have to pay a multiple for it. I would never buy a QLC SSD. Either buy a consumer TLC SSD (and be ready to replace it often, to have sufficient redundancy and a UPS) or better an enterprise SSD. Prosumer SSDs only made sense a few years ago when they were using the faster and more durable MLC NAND. But these days they also use the cheaper and less durable TLC NAND only.

People often don't get the big differences between SSD technologies. They think a SSD is a SSD so why buy one that costs 3 or 10 times as much. Only thing they care about is, if it is a SATA or a NVMe SSD, as NVMe SSDs are advertised with way higher throughput numbers and they think the more MB/s the better. But no one looks at the terrible TBW numbers or if that SSD got a powerloss protection.
But they don't think a car is a car, and wouldn't buy a cheap compact car to plow a field. If you have heavy work to do like plowing a field, buy something maybe slower but more powerful and durable like a tractor, even if that tractor costs a multiple. It's the same with SSDs. You get what you pay for and cheap SSDs just might not be up to the job.

Also, do you have any affordable recommendations of Enterprise SSDs? I actually also considered Intel DC S3510 as I found 800Gb ones for 50€ and they still have +95% of life in them, what do you think about something like that?

Yes, second hand enterprise SSDs are great if you know the wear because the seller is showing the smart values before buying.

 

[xellos]

PCI Passthrough will only prevent the virtualization Overhead. You might still get heavy overhead from filesystems like ZFS or raid.

No, the prosumer SSDs are the same as consumer SSDs. Atleast for Samsung Pros which don't use any MLC NAND anymore. But all prosumer SSDs are still missing the powerloss protection, sync write performance and most of the durability.
Lets have a look at the durability of different grades:

	TB TBW per 1TB capacity:	Price per 1TB storage:	Price per 1TB TBW:
Cheap QLC Consumer (Crucial P3 1TB):	220 TB	71 €	0,322727 €
Normal TLC Consumer (Samsung 980 1TB):	600 TB	79 €	0.131667 €
Prosumer TLC (Samsung 980 Pro 1TB):	600 TB	104 €	0.173333 €
Cheap Enterprise (Micron 7450 PRO 960GB)	1,771 TB	196 €	0.110672 €
Normal Enterprise (Micron 7450 MAX 800GB):	5,375 TB	271 €	0.050419 €
Good Enterprise (Intel Optane P4801X 375GB):	109,333 TB	3296 €	0.030146 €

So basically...the more you pay for your SSD, the cheaper that SSD actually will be on the long run, because it is more durable and will survive longer. Only exception is the prosumer SSD, which is just more expensive...So you don't have to wonder why ZFS is killing SSDs so fast, when buying cheap consumer SSDs with just a rated durability of 220 TB if there are good enterprise SSD that are rated to survive 109333 TB of writes.
Compare those numbers and you will realize how crappy those cheap consumer SSDs are, and that it might be a good idea to buy something more durable even if you have to pay a multiple for it.

Yes, second hand enterprise SSDs are great if you know the wear because the seller is showing the smart values before buying.

Thanks for the reply!

The table is amazing and it's a good thing to consider. I would hate it to have to replace SSDs every now and then. Ill definitely go for higher endurance drives now but I'll mix them with consumer SSDs as I already have some 2Tb brand new ones from Samsung laying around.

One last question from my side, how can I know if a drive has a good "sync write" performance? Is it determined by IOPs?

Thanks!

 

[Dunuin]

Edited my post above adding some more SSD grades. I forgot to add a MLC enterprise SSD and a MLC prosumer SSD.

One last question from my side, how can I know if a drive has a good "sync write" performance? Is it determined by IOPs?

If it got no powerloss protection (and no consumer/prosumer/NAS SSD got that), it will have a bad sync write performance. As only SSDs with a powerloss protection can use the SSDs internal RAM cache to cache sync writes. That's because all cached data in RAM is immediately lost on an power outage. Because of that enterprise SSDs got capacitors that can power the SSD for a few seconds, even when the PSU fails or there is an power outage, so it can quickly dump the data from the volatile RAM cache to the non-volatile SLC cache. That makes it possible to also cache sync writes in RAM-cache, which could only be cached in the slower SLC-cache otherwise. And if you can't cache those in RAM, sync writes will be terrible slow and you get a terrible write amplification increasing the SSD wear, as write operations can't be optimized to reduce wear. Keep in mind that RAM will not wear when writing to it. All SLC/MLC/TLC/QLC NAND cells will wear, also writing to the SLC cache.

 

[xellos]

Edited my post above adding some more SSD grades. I forgot to add a MLC enterprise SSD and a MLC prosumer SSD.


If it got no powerloss protection (and no consumer/prosumer/NAS SSD got that), it will have a bad sync write performance. As only SSDs with a powerloss protection can use the SSDs internal RAM cache to cache sync writes. That's because all cached data in RAM is immediately lost on an power outage. Because of that enterprise SSDs got capacitors that can power the SSD for a few seconds, even when the PSU fails or there is an power outage, so it can quickly dump the data from the volatile RAM cache to the non-volatile SLC cache. That makes it possible to also cache sync writes in RAM-cache, which could only be cached in the slower SLC-cache otherwise. And if you can't cache those in RAM, sync writes will be terrible slow and you get a terrible write amplification increasing the SSD wear, as write operations can't be optimized to reduce wear. Keep in mind that RAM will not wear when writing to it. All SLC/MLC/TLC/QLC NAND cells will wear, also writing to the SLC cache.

Thanks for all the information! I'll mark my question as "solved".