Nvmes, SSDs, HDDs, Oh My!

[Carl J]

So, I'm trying to plan out a new Proxmox server (or two) using a bunch of spare parts that are lying around.

Whether I go with a single or two Proxmox servers (it's deciding whether or not to have an Internal server, for Media, Backups, Git/Jenkins, and a separate External Server for Web, DBs, and Cloud), the layout I was considering would be pretty much the same.

But, from what I've read, I may be going about this the wrong way

- OS/VM Drives
--- 2 x 1TB Nvmes (I could go up to 4 x 1TB if I decide to go with a single server) - Would RAID them

- Databases/Image/file storage for web server
--- 2 x 2TB SSDs - RAID 1

- VM Backup Drive
- 4TB (or more) HDD

Then other HDDs for media or overall backup server (for other computers on the network), ....

But from what I've been reading, it seems like Nvmes and SSDs aren't the best option for booting off because of the amount of data that's written (and the block size being bigger than what may be needed). Is this still the case? Or is this only if certain features are being used?

Would the following make more sense?

- OS/VM Backups
--- 4TB (Parititioned for Proxmox and for VM Backups)

-- VMs
- 2 x 1TB (or 4 x 1TB) Nvmes

- Databases/Image/file storage for web server
--- 2 x 2TB SSDs - RAID 1

- Other drives for media/backups/....

 

[Dominic]

As reference, I use a Samsung 860 QVO since 2 years for the PVE workstation and it is at 2% wearout. So using cheap SSDs for PVE can go well. However, if you have some write-heavy VMs running 24/7 on it, this can look different pretty quick. This is the reason why we generally recommend enterprise SSDs.

 

[Dunuin]

For the OS drive it doesn't matter what you use, as long as it is no USB stick or SD card. Proxmox itself will only write around 30GB per day so thats no problem for HDDs or consumer SSDs.

For the VM storage you really want SSDs. HDDs just can't handle all the IOPS. And like Dominic already said it is recommended to use enterprise grade SSDs. Especially if you want to run some DBs doing sync writes because they are much faster (for example 50x) for that workload compared to consumer SSDs. And you may pay 2-3 times the price but you get up to 30x more durability compared to consumer SSDs. So in the long term they are much cheaper.

For the backups that depends...HDDs will do the job but if you want to use the Proxmox Backup Server (and I would recommend that because it is way faster and saves alot of space) HDDs are really slow because everything will be deduplicated using thousands or millions of chunks so you get alot of IOPS that HDDs can't handle well. Specially if you want to use "stop" instead of "snapshot" backup mode, so the backups are more save because the VMs is backuped in a defined state, a slow backup storage is really annoying because the VMs are offline all the time while backing them up.

And using raid1 or raid10 is a good idea for everything. Try not to use raid5/raid6 for your VM storage because this will increase the latency.

 

[spirit]

As reference, I use a Samsung 860 QVO since 2 years for the PVE workstation and it is at 2% wearout. So using cheap SSDs for PVE can go well. However, if you have some write-heavy VMs running 24/7 on it, this can look different pretty quick. This is the reason why we generally recommend enterprise SSDs.

if you use ceph or zfs, qvo just sucks for the journal syncronous writes. (something like 200-400fsync/s).

 

[Carl J]

Thanks everyone.

As for the enterprise SSDs, I wish I had some. I'm just using what I currently have available (from a bad CHIA experiment). So I have several NVMEs (Sabrent Rocket Max Performance and some Seagate Firecudas) and quite a few SSDs (Samsung EVO 850/870s and some cheaper Kingstons)

For backing up, I guess I'll have to look into that more. Was thinking of using some Seagate Ironwolfs for backing up the VMs (once a week or so). And using UrBackup for backing up my main Windows machine as well as the files within the VMs. Still trying to figure out the best way/options to do all of this.

By the way, this is for a home server (or two), nothing too fancy. Looking to have something to host some Wordpress and .Net sites on one (as well as NextCloud), and then a Dev/Build/DB server. They'll probably sit more idle than anything.

@spirit I think this is what I was wondering about, ZFS on consumer grade NVMEs and SSDs. So don't use it if I'm using either of those types of drives?

 

[Dunuin]

Thanks everyone.

As for the enterprise SSDs, I wish I had some. I'm just using what I currently have available (from a bad CHIA experiment). So I have several NVMEs (Sabrent Rocket Max Performance and some Seagate Firecudas) and quite a few SSDs (Samsung EVO 850/870s and some cheaper Kingstons)

Especially there enterprise SSD are recommended so they live more than some weeks.

By the way, this is for a home server (or two), nothing too fancy. Looking to have something to host some Wordpress and .Net sites on one (as well as NextCloud), and then a Dev/Build/DB server. They'll probably sit more idle than anything.

Dont underestimate the write amplification. I'm also running only a homeserver that is most of the time idleing but it still writes nearly 1TB per day and most of that are just logs/metrics the VMs are creating themself while idleing...

@spirit I think this is what I was wondering about, ZFS on consumer grade NVMEs and SSDs. So don't use it if I'm using either of those types of drives?

Peformance isn't that bad if you got TLC consumer SSDs. Atleast for async writes. More problematic is the durability.

 

[Dunuin]

2021-07-09T07:01:38+02:00: Original data usage: 9.11 TiB
2021-07-09T07:01:38+02:00: On-Disk usage: 125.95 GiB (1.35%)
2021-07-09T07:01:38+02:00: Deduplication factor: 74.05

That is why I like PBS. 9.11TB of VM disks compressed and deduplicated down to 126GB on the backup drive. Using vzdump it still wouldn't be 9.11TB because vzdump is also compressing, so empty VM disk space isn't a thing, but it still would use 2-3TB on the backup storage because it can't handle incremental backups.

If you got 2 server you could install PBS on both of them, give both a dedicated backup disk and they could backup each other. So if anyone of them is failing you got all VMs backuped on the other one and so its easy to restore them.

 

[Carl J]

That is why I like PBS. 9.11TB of VM disks compressed and deduplicated down to 126GB on the backup drive. Using vzdump it still wouldn't be 9.11TB because vzdump is also compressing, so empty VM disk space isn't a thing, but it still would use 2-3TB on the backup storage because it can't handle incremental backups.

If you got 2 server you could install PBS on both of them, give both a dedicated backup disk and they could backup each other. So if anyone of them is failing you got all VMs backuped on the other one and so its easy to restore them.

Interesting. Thanks for sharing this.

After I read it, I did start to set up a backup server on another little machine that I have.

The original idea was to have a dedicated drive on the Proxmox server for VM backs, and would have some other (wasn't sure which one) server on the smaller machine. Now, I will still have some other backup server for my home machines and a Proxmox one on it.

Just re-read your second paragraph, and not sure why, never thought about having a Proxmox Backup server on the Proxmox machine itself as well. Was just going to let the host manage the backups.

Two questions:
- Which file system should I use for the backup drives themselves? Does it matter?
- Can I get away with just having a Proxmox Backup Server on the host to copy to a dedicated drive, and then having some other single backup server on the other machine, and just copy the backed up files to it? Or is there an advantage of having two Proxmox Backup servers?

Thanks again

 

[Dominic]

There is a built-in function to sync backups between two PBS installations.

Which file system should I use for the backup drives themselves? Does it matter?

PBS offers some options in the GUI to manage this. Depends a little on the system around the drives (RAID controller, available memory).

 

[Dunuin]

- Which file system should I use for the backup drives themselves? Does it matter?

It isn't that important because PBS is doing replication, compression and checksums integety checks on file level. So stuff like ZFS wouldn't benefit that much of its features because you want to disable stuff like block level compression, block level deduplication and so on anyways so it isn't done twice.

- Can I get away with just having a Proxmox Backup Server on the host to copy to a dedicated drive, and then having some other single backup server on the other machine, and just copy the backed up files to it? Or is there an advantage of having two Proxmox Backup servers?

You can't restore your VMs (and later you will be able to backup the PVE host itself too) if no PBS is running. So if PBS is running on the same machine that is also running PVE you can't use the PBS to restore stuff if the server fails and needs to be reinstalled.
If you just sync the datastore (the folder containing all your backup data) to another host your backups should be save, you could reinstall a fresh PVE+PBS, sync back your datastore and then restore the VMs. But that ways it is much more complicated and you won't be able to restore the complete host if that feature is added in the future.

So if both servers are running Proxmox it would be best to install PBS on both of them and keep them synced. So no matter which of the 2 servers will fail, there will be always a PBS running to restore then failed server.

 

[Carl J]

It isn't that important because PBS is doing replication, compression and checksums integety checks on file level. So stuff like ZFS wouldn't benefit that much of its features because you want to disable stuff like block level compression, block level deduplication and so on anyways so it isn't done twice.

You can't restore your VMs (and later you will be able to backup the PVE host itself too) if no PBS is running. So if PBS is running on the same machine that is also running PVE you can't use the PBS to restore stuff if the server fails and needs to be reinstalled.
If you just sync the datastore (the folder containing all your backup data) to another host your backups should be save, you could reinstall a fresh PVE+PBS, sync back your datastore and then restore the VMs. But that ways it is much more complicated and you won't be able to restore the complete host if that feature is added in the future.

So if both servers are running Proxmox it would be best to install PBS on both of them and keep them synced. So no matter which of the 2 servers will fail, there will be always a PBS running to restore then failed server.

Awesome, thank you.
I guess with backing up/restoring, backing up is only half of the solution. Didn't think so much about the restore part