OS disk HDD or SSD?

[northflower]

So during the Easter holidays and corona isolation i saw an opportunity to finally move away from Esxi to Proxmox on my home server.
Now i have been able to solve most of my issues during this move by googling and reading. But for one issue i am unable to find reliable information and a generall consensus. I apologize if this information is indeed present on the wiki but i have been unable to find it.

My home server used a Sandisk Extreme 32 GB 3.0 USB drive for the Esxi OS. It is clear that this is a bad choice with proxmox since it will wear out due to proxmox doing many writes.

Therefore i will buy either 2 small consumer SSDs or HDDs for the OS and do a mirror for ZFS rpool.
The VMs will be located on a different pool so these two new drives will be solely for the proxmox OS.

I have read several claims that proxmox will wear out consumer SSDs quickly. Some say it will not.

My question is therefore: Should i get SSDs or HDDs for proxmox OS?

 

[velocity08]

hi @northflower

if your not going to put anything else on the root drive I.e. iso, templates etc small spinning disk could be less expensive If you can find them.

zfs in general does a lot more read/ write work and if you have a power loss situation data loss can occur with ZFS on ssd if they don’t have a capacitor backup.

get the right tool for the job, if your getting ssd for ZFS then make sure they have the capacitor if the power go’s or data will, in advising of this you’ll also need to make sure your HBA card can deliver enough throughput for ZFS other wise you may as well stick with a hardware raid and ext4 On consumer driver will be cheaper.

there are sata ssd with capacitors from Samsung and other brands, just need to do some research, also try to find ones with higher Drive writes per day (DWPD).

ZFS is awesome but you need to make sure every part of the equation is well thought through and correct components selected or it will fail you.

• enough ram for zfs and then for VM’s
• proper hba controller
• correct drive types
• careful planning
• lots of testing

””Cheers
G

 

[northflower]

hi @northflower

if your not going to put anything else on the root drive I.e. iso, templates etc small spinning disk could be less expensive If you can find them.

zfs in general does a lot more read/ write work and if you have a power loss situation data loss can occur with ZFS on ssd if they don’t have a capacitor backup.

get the right tool for the job, if your getting ssd for ZFS then make sure they have the capacitor if the power go’s or data will, in advising of this you’ll also need to make sure your HBA card can deliver enough throughput for ZFS other wise you may as well stick with a hardware raid and ext4 On consumer driver will be cheaper.

there are sata ssd with capacitors from Samsung and other brands, just need to do some research, also try to find ones with higher Drive writes per day (DWPD).

ZFS is awesome but you need to make sure every part of the equation is well thought through and correct components selected or it will fail you.

• enough ram for zfs and then for VM’s
• proper hba controller
• correct drive types
• careful planning
• lots of testing

””Cheers
G

Thanks for the reply. Yes i think i will simply buy some cheap spinners out of ebay for the rpool since they are dirt cheap. Thanks for the tips regarding ZFS. I will be using UPS and intel SSDs with capacitors for the VM storage (not-related to the rpool).

But regarding the question. Like previous threads on this issue they quickly diverge into other details like ZFS sync, zil, powercap etc.
If we try to set aside those issues i.e. lets assume its ok to run a SSD without capacitors for the rpool, will proxmox cause so much writes so that a consumer grade SSD will wear out?

Edit: lets throw in an example. The Crucial BX500 128 GB is a cheap SSD (https://www.crucial.com/ssd/bx500/ct120bx500ssd1) and its has the specification: Endurance120GB drive: 40TB TotalBytes Written (TBW), equal to 21GB per day for 5 years. Lets say our expected lifetime for the server is 10 years that would give us 10,5 GB per day for 10 years.

Can proxmox really be generating such an amount of writes so that a consumer SSD like the above will wear out within the servers expected lifetime? Given these numbers proxmox wearout of consumer SSDs sounds like a non-issue?

 

[velocity08]

Thanks for the reply. Yes i think i will simply buy some cheap spinners out of ebay for the rpool since they are dirt cheap. Thanks for the tips regarding ZFS. I will be using UPS and intel SSDs with capacitors for the VM storage (not-related to the rpool).

But regarding the question. Like previous threads on this issue they quickly diverge into other details like ZFS sync, zil, powercap etc.
If we try to set aside those issues i.e. lets assume its ok to run a SSD without capacitors for the rpool, will proxmox cause so much writes so that a consumer grade SSD will wear out?

Edit: lets throw in an example. The Crucial BX500 128 GB is a cheap SSD (https://www.crucial.com/ssd/bx500/ct120bx500ssd1) and its has the specification: Endurance120GB drive: 40TB TotalBytes Written (TBW), equal to 21GB per day for 5 years. Lets say our expected lifetime for the server is 10 years that would give us 10,5 GB per day for 10 years.

Can proxmox really be generating such an amount of writes so that a consumer SSD like the above will wear out within the servers expected lifetime? Given these numbers proxmox wearout of consumer SSDs sounds like a non-issue?

Hi @northflower

we don't run home labs, we only have production servers and storage so i couldn't begin to tell you about your work loads, obviously if your data is sitting dormant most of the time you'll get a longer shelf life.

for home use sure go consumer grade as long as you have a capacitor on the SSD for power failure it shouldn't be an issue.

With production work loads you never know what clients are doing or are going to do so reads and writes are unpredictable due to mixed random work loads.

as long as you have backups separate from your ZFS pool you should be fine as an outage only effects you and no clients.

I'm sure others will chime in with their own experiences based on their use case scenarios which will differ greatly as workloads will differ.

best of luck with the project

""Cheers
G

 

[northflower]

Hi @northflower

we don't run home labs, we only have production servers and storage so i couldn't begin to tell you about your work loads, obviously if your data is sitting dormant most of the time you'll get a longer shelf life.

for home use sure go consumer grade as long as you have a capacitor on the SSD for power failure it shouldn't be an issue.

With production work loads you never know what clients are doing or are going to do so reads and writes are unpredictable due to mixed random work loads.

as long as you have backups separate from your ZFS pool you should be fine as an outage only effects you and no clients.

I'm sure others will chime in with their own experiences based on their use case scenarios which will differ greatly as workloads will differ.

best of luck with the project

""Cheers
G

Makes sense. My workload is way different than the workload used by the target audience of this product, thus my question is probably moot.

Info for other home-labbs like mine:
I did find two old consumer SSDs (120GB). I have installed proxmox on these and will report back in this thread if i see that their wearout starts to rise. The wearout is currently at 0%.

I did also disable some services that are known to cause some increased writes that i did not need.

systemctl stop pve-ha-lrm
systemctl stop pve-ha-crm
systemctl disable pve-ha-crm
systemctl disable pve-ha-lrm

systemctl edit --full pvesr.timer
change to OnCalendar=monthly
systemctl daemon-reload

 

[lixaotec]

Does any staff member can confirm that is safe to disable HA services? I also do not use them , and have a consumer SSD (kingston a400 120gb) as proxmox main drive.

Thanks

 

[dolan]

Makes sense. My workload is way different than the workload used by the target audience of this product, thus my question is probably moot.

Info for other home-labbs like mine:
I did find two old consumer SSDs (120GB). I have installed proxmox on these and will report back in this thread if i see that their wearout starts to rise. The wearout is currently at 0%.

I did also disable some services that are known to cause some increased writes that i did not need.

systemctl stop pve-ha-lrm
systemctl stop pve-ha-crm
systemctl disable pve-ha-crm
systemctl disable pve-ha-lrm

systemctl edit --full pvesr.timer
change to OnCalendar=monthly
systemctl daemon-reload

So how did it work out for you? Are you still using the two consumer SSD's, and how much have they worn out?

 

[lixaotec]

I´m also using consumer SSDs Kingston. I have disabled ha, and after 3 months I notice just a little increase in wearout, 1% I think. And it is up 24hours a day.

However my VMs are located on other ssds, that are running for heavy duties. even that just increased 1% during this time.

Unless proxmox smart counters for wearout are messed, it seems pretty suitable to use consumer ssds