Determining Backup Server Specs

[ejmerkel]

I have been wanting to build PBS server but have a few questions. I have about 80 TB of VM's to backup.

I know SSD drives are the recommended drives to use in a PBS but SSD drives are extremely expensive at the moment. Does anyone have suggestions on good SSD drives for this type of use that aren't extremely expensive?

I was considering using a Dell 740xd with 12 x 12TB spinning drives in a ZFS configuration with a special device mirror. Would these mirror drives need to be NVMe or would I get by with a pair of SSD's? Would Optane drives be useful in this scenario?

I understand there is a speed vs storage trade off but are spinning drives just a bad idea?

Best regards,
Eric

 

[tom]

I understand there is a speed vs storage trade off but are spinning drives just a bad idea?

In my opinion, yes. Datastore maintenance tasks are optimized for flash storage.

Compared to the costs of a long downtime due to a slow backup/restore system, high end NVMe storage is "cheap".

 

[Dunuin]

Keep in mind that PBS will split everything in chunks with a maximum size of 4MB. So if you got 80TB of VMs that are ATLEAST 20.000.000x 4 MB files. Your daily/weekly garbage collection will need to go through all the 20+ mio chunks file by file and check its metadata. And in case of a complete reverify task it also needs to read and hash the complete 20 mio files again. You know how bad HDDs can handle millions of small files.
A "special device" might help with the GC but still wont help with the even longer taking verify tasks. And because chunks are deduplicated they should fragment over time so you won't get great sequential reads/writes.

 

[tjk]

In my opinion, yes. Datastore maintenance tasks are optimized for flash storage.

Compared to the costs of a long downtime due to a slow backup/restore system, high end NVMe storage is "cheap".

Tom, I love Proxmox, but think this is very bad design on the software side to require flash for backups.

Spinning HDD's are still the primary source for most backup appliances/servers, especially when you are talking 10's to 100's of TB of backup appliances.

Sure, if you have 3 or 5 TB of backup data, no big deal. But go ahead and price out a backup appliance to store 50 to 100TB+ of data, and then x2 for redundancy.

Large flash drives aren't comparable in price to HDD's of the same capacity. There is a reason Seagate and WD keep making HDD's bigger and bigger and haven't moved away from manufacturing larger drives each year. Flash just can't compare per TB to HDD's.

 

[ness1602]

PBS doesnt require flash for backups, i have it on 3 locations and none of them have SSDs. Tom is just saying ,if recovery time is the most important parameter, go with SSDs. If not ,use sata regular drives(I'm using ironwolf, sas drives, whatever i can get)

 

[t.lamprecht]

Tom, I love Proxmox, but think this is very bad design on the software side to require flash for backups.

It's no hard requirement, but it just makes sense to have for the most important server in a company if things go bad:

• much higher reliability (no moving parts), you want the backup data pool last for more than a few years.
• fast enough so that one can even verify the whole backup dataset (which is part of the "only tested backups are valid backups") in a reasonable time frame
• 15.36 TB SSD drives are available now for ~2300 to 2500 € (excl. VAT), and you probably do not need to scale up to 100 TB immediately, starting with 15 or 30 TB capacity now and adding mirror pairs in the future is a valid option too, not a bad investment for increased reliability and reduced time spent on administration and more importantly on recovery if things go bad - cheap 100 TB spinner only sound good as long as you do not need to wait days until, e.g., your main domain controller or mail server is recovered, which can cost a company easily more than the few thousands one saved on cheaper disks.

I made some other posts regarding HW and also advantages of SSDs in the past if you're interested:
https://forum.proxmox.com/threads/garbage-collector-too-slow.86726/#post-381027
https://forum.proxmox.com/threads/hardware-recommendations-for-larger-setup.100221/#post-432707

Also, if you think it's very bad design nobody stops you from using any other backup solution that fits your design needs better, you won't ever outrun basic limits like seek time and bandwidth though.

Large flash drives aren't comparable in price to HDD's of the same capacity. There is a reason Seagate and WD keep making HDD's bigger and bigger and haven't moved away from manufacturing larger drives each year. Flash just can't compare per TB to HDD's.

Same as (enterprise) spinners can't compare at all in terms of seek times, provided bandwidth and also reliability compared to (enterprise) flash. Remember that PBS uses a Content Addressable Storage to reduce the space usage of subsequent, or similar, backups - that nice advantage comes with the cost of having more files due to chunking and the requirement to check them, so flash fares better, but often you also do not need as much of it.

Note that, as said, nobody stops you from buying cheap(er) and slow(er), with things like ZFS special-devices pair for metadata you can even improve the setup speed quite a bit, for some people its fast enough then for their use case and PBS will work just fine, albeit, don't expect miracles as you'll get the bandwidth and seek times you paid for.

 

[rigel.local]

if you think it's very bad design nobody stops you from using any other backup solution that fits your design needs better

nobody stops you from buying cheap(er) and slow(er), with things like ZFS special-devices pair for metadata

What an arrogant and stupid gatekeeper attitude... why are you so mad?
Google prices on Enterprise DWPD more than 1 SATA SSD 7.68TB or 15.36TB. Of course nobody stops us to spend $10K on PBS drives.

 

[tjk]

What an arrogant and stupid gatekeeper attitude... why are you so mad?

I didn't think his reply was arrogant at all. He put the facts out there, you can still run PBS with spinners, I have a dozen PBS servers with 100T HW raid working just fine.

Large enterprise flash is still many times more expensive than enterprise spinners, and will be for awhile. I use NVME for all our PVE nodes for storage and bare metal nodes. The only spinners we run now are on backup nodes and large cheap and deep ceph clusters (150+ PB).

 

[ness1602]

I have more than few PBSes with hdd and sometimes hdd-with-special-device. And it works, maybe the performance isn't all that great but it works.
I have implemented in the last few weeks PBS with 16-disk raidz2. It works great(by my standards). If your standards aren't ultra-enterprise, you will be satisfied with PBS any way.

 

[t.lamprecht]

What an arrogant and stupid gatekeeper attitude... why are you so mad?

Even as you are trying to put my post out of context and highlight it's still the truth. You can use slower storage, especially at random IO loads just fine, in contrary to what you suggest we certainly do NOT gatekeep anybody to do so (besides that even if our projects are quite popular, we really don't have the market dominance for that sentence to make any sense...), but just highlight why one cannot expect wonders from cheaper and slower drives. Also, if there's a solution that does magically all that you want as fast as you want, we certainly do NOT gatekeep you from staying with Proxmox projects.

And you might want to rethink the way you express yourself in our community forum, i.e. actually provide the full context and not just try to insult developers who are trying to provide context without you contributing anything to the discussion yourself; after all you agreed to our forum's terms and rules when signing up.

 

[rigel.local]

Even as you are trying to put my post out of context and highlight it's still the truth. You can use slower storage, especially at random IO loads just fine, in contrary to what you suggest we certainly do NOT gatekeep anybody to do so (besides that even if our projects are quite popular, we really don't have the market dominance for that sentence to make any sense...), but just highlight why one cannot expect wonders from cheaper and slower drives. Also, if there's a solution that does magically all that you want as fast as you want, we certainly do NOT gatekeep you from staying with Proxmox projects.

And you might want to rethink the way you express yourself in our community forum, i.e. actually provide the full context and not just try to insult developers who are trying to provide context without you contributing anything to the discussion yourself; after all you agreed to our forum's terms and rules when signing up.

I apologize for my emotions in my previous post. Insulting developers was not my intention. I just could not fully understand the storage hardware requirements for PBS. It is really complicated for newbies. Documentation says "yes, you can use HDDs, but SSDs are recommended." but then again SSDs these days are so different in terms of durability (QLC,TLC,MLC,IOPs performance etc.). It was not clear what kind of random write OPs PBS is heavily relies on. And the in forum I read from developers that using HDDs is a really really bad idea. While at the same time other users report that they are using HDDs just fine.

Thank you members of this community like you and Dunuin, I now understand approximately what hardware I need for PBS. I would probably suggest to explain in a little bit more details what kind of SSDs are recommended here in documentation in terms of random writes durability:
https://pbs.proxmox.com/docs/system-requirements.html

I sincerely apologize for my words again.