Proxmox VE best practices

[amasnur]

Hi, noob here, please be gentle. I have:

• 2 x 800GB SSDs,
• 2 x 73GB SASs
• 8 x 960GB SSDs

I want to setup a single PVE for multiple production application and DB VMs.

What will be the best practice for the drives I have? My initial plan was to use as follows:

• 1 x 73GB SAS for Proxmox VE OS
• 8 x 960GB enterprise SSDs as RAIDZ1 for VM drives

For the OS:

1. Will there be noticeable benefit to use the 800GB SSD instead? I think it is a great overkill (in size and speed) as I imagine there will not be much going on on the OS drive.
2. Will I be better off using two drives as RAID1 instead of creating an offline clone as backup? If yes, how do I do this during fresh install? (If no, I intend to do the offline clone using clonezilla).

For the VMs:

• I've read that RAIDZ1 will impact SSD health, and as such, am better off set as LVM-Thin pool. The DB and applications are new, and I cannot predict which will grow and need additional storage, so I would prefer them as a pool which I can allocate as needed later.

Your opinions are very much appreciated.

 

[leesteken]

Regarding RaidZ1 for VMs, please read this: https://forum.proxmox.com/threads/fabu-can-i-use-zfs-raidz-for-my-vms.159923/

 

[amasnur]

Regarding RaidZ1 for VMs, please read this: https://forum.proxmox.com/threads/fabu-can-i-use-zfs-raidz-for-my-vms.159923/

Using mirrored stripe setup for VM will leave me half the original space (3.9TB), as compared to n-1 capacity (6.7TB) or less because of the block size padding, but not to the extent of the mirrored drive.
The stripe won't give much advantage as SSDs are fast enough, I think.

 

[leesteken]

Using mirrored stripe setup for VM will leave me half the original space (3.9TB), as compared to n-1 capacity (6.7TB) or less because of the block size padding, but not to the extent of the mirrored drive.
The stripe won't give much advantage as SSDs are fast enough, I think.

It's your choice. Test it and in case it does not provide enough IOPS, you'll know what might be done about it. I just wanted you to be aware of that thread since a lot of people are disappointed by RaidZ1/2/3.

You say you are going to run "a single PVE for multiple production application and DB VMs". Are you a seasoned system administrator for a company of 100 employees or are these home automation VMs for your home lab. It's hard to tell from your post and advise from people here might depend on details like that.

 

[UdoB]

For the OS:

1. Will there be noticeable benefit to use the 800GB SSD instead? I think it is a great overkill (in size and speed) as I imagine there will not be much going on on the OS drive.
2. Will I be better off using two drives as RAID1 instead of creating an offline clone as backup? If yes, how do I do this during fresh install? (If no, I intend to do the offline clone using clonezilla).

1.: probably not

2.: yes, definitely. What happens when that single drive fails? You have to investigate and restore from backup. How old is it in the case of "offline clone"?

If it is a mirror you need to to... nothing, except of course to replace the damaged drive. No downtime, no investigation under stress - just "zpool replace ...". Live, while the system continues to work...

During installation you can choose to create a ZFS mirror (or raid1, maybe it is mis-named). This will be the "rpool" later on.

 

[amasnur]

It's your choice. Test it and in case it does not provide enough IOPS, you'll know what might be done about it. I just wanted you to be aware of that thread since a lot of people are disappointed by RaidZ1/2/3.

You say you are going to run "a single PVE for multiple production application and DB VMs". Are you a seasoned system administrator for a company of 100 employees or are these home automation VMs for your home lab. It's hard to tell from your post and advise from people here might depend on details like that.

I am running assets management, store management, and students management systems for my education based institution. Internal users count up to 1000+ as we have over 200 branches, but the loads are seasonal, so 300 concurrent users is expected most of the time.

Do you have any views about the health impact of RAIDZ on SSDs? My current setup is hardware RAID5, but I read many posts warning against reliance on those, especially with out-of-warranty hardware.

 

[amasnur]

1.: probably not

2.: yes, definitely. What happens when that single drive fails? You have to investigate and restore from backup. How old is it in the case of "offline clone"?

If it is a mirror you need to to... nothing, except of course to replace the damaged drive. No downtime, no investigation under stress - just "zpool replace ...". Live, while the system continues to work...

During installation you can choose to create a ZFS mirror (or raid1, maybe it is mis-named). This will be the "rpool" later on.

Thanks. I am considering to put the SSDs for other better use. Your feedback strengthens this.

 

[UdoB]

I've read that RAIDZ1 will impact SSD health, and as such, am better off set as LVM-Thin pool.

A minute ago I have posted something I had in my pipeline - for admins asking questions regarding dropping ZFS: https://forum.proxmox.com/threads/f...y-a-few-disks-should-i-use-zfs-at-all.160037/

 

[louie1961]

If it were my machine, and it is a production machine as you say, I would want the Proxmox OS on a set of mirrored drives. Depending on the number of VMs, you could put the OS on the two 800gb SSDs and use the rest of the space for VMs. I am not a fan of storing any data in the VMs (or more accurately the virtual drives that the VM runs from). I like to keep VMs as small as possible. I would use the 73GB drives for ISOs and possibly backup storage, but they are awfully small. I might not use them at all. I would set up the 8 960GB SSD drives in a striped mirror. I would have 4 mirrored VDEVs, yielding 3.8TB (as you noted above) pool of very fast storage for use in storing the databases and other application data. I am not sure how much data you need to store, so YMMV. But for my setup, that's what I would do. Most of my VMs are 32gb or less because I keep all my data on NFS shares, even docker volumes. Also the striped mirrors will be signifianctly faster. I tried both ways on my latest server, and I wanted to be able to saturate a 10gbe connection. Raid Z1 couldn't do it with SATA SSDs.

If you decide to not use ZFS, you could still use the same approach with other file systems. 4 mirrored VDEVs (or alternatively an ext4 raid 10) will give you better perfomance than a RaidZ1 or Raid 5.

 

[_gabriel]

2x73GB HDD for PVE OS + local storage for ISOs , ZFS mirror.
10 x 800 GB ZFS striped mirrors (PVE call it RAID10) (800GB partition from the 960GB) as data pool
then 8 x 160 GB ZFS striped mirrors ( partition left from 960GB) as os vm pool for example.