I tired PBS. Not convinced.

[venquessa]

I probably don't really fit in it's market. I think it tried to do too much to complexly.

I had crashed it and corrupted it's disks requiring a full 4 Second button hold and a nonboot within the first hour.

It took me another hour to clear that up ... it literally took an hour to "rm -rf /mnt/pve/dump/.chunks". How many chunks exactly? And just how many IO threads do you launch with them?

If you run this on a 2020 era Seagate HDD on a USB 3.0 mount, it will (practically) NEVER finish. It will completely and utterly trash the IO Layer of anything short of a full 4U blade stack on a dual SAS array. My consumer hardware did not cope and after listening to the HDDs clatter and grind for an hour on the first backup, I tried to stop it. That is how I ended up corrupting the disks, backups and having to hard reset the machine.

After this I asked myself... do you want to have to do this at 3am when you actually NEED a backup and if the PBS host is miss behaving or going to take 4 hours to extract 3 million chunk files into a perfectly good file to start with....

Nope.

As I said, if I had a bigger setup with a half dozen PVE nodes and full SAS/NAS level shared storage with enough memory to cache all those inodes, I could see the advantages outweighing the complexity hardware demands and ultimate fragility through proprietary complexity.

 

[danielb]

I run PBS on a tiny fanless qotom box, with two external disks on an USB3 enclosure (using zfs mirror). Nothing but consumer grade hardware. It's rock solid

 

[Dunuin]

Also note that PBS isn't made for HDDs. It was designed with local SSDs in mind. It's super low demanding when it comes to CPU (all computation is done clientside except for verify jobs) and RAM (especially when you compare ZFS deduplication vs PBS software-level deduplication). All you need is a proper storage, so some ZFS pool with redundnacy consisting of enterprise SSDs.

So yes, if you aren`t willing to buy proper enteprise SSD the results will be not great down to unusable. The IOPS performance is really needed, especially for running the GC. But this all is also mentioned in the system requirements:

• Use only SSDs, for best results
• If HDDs are used: Using a metadata cache is highly recommended, for example, add a ZFS special device mirror.

So best SSD-only and if that isn't possible at least a mix of HDDs for data + SSDs for metadata. While HDD-only or NFS/SMB works, it won't deliver a usable performance outside of a very small homelab.

 

[venquessa]

Also note that PBS isn't made for HDDs. It was designed with local SSDs in mind.

It would seem to be the case. However, writing a high inode count to any filesystem is a sure way to burn it out prematurely.

Given that an HDD is about 10 times slower in sequencial reads and about 100 to 1000 times slower in random access than an SSD, it's likely that a backup by PBS to M.2 that takes 15 minutes, could literally take a week on an HDD. In fact. I found when moving 1 Tb of data between 2 ZFS volumes on the same HDD pool it was over 10 times faster to read to an SSD first, then write it back, so you are not "oscillatig" the heads between read there, write here constantly. The head seek time is 3 or more orders of magnitude slower than an SSD.

Although, again, you mention "Enterprise grade SSDs" that will stand up to constant daily thrashing for their expected service life of 3-5 years with less than a 1% failure rate? No. Just no. If you really mean enterprise SAS arrays and do it properly, my entire network could be purchase for the cost of a single SAS array. So I am definitely out of the market.

Putting this on a standard consumer SSD will see it die early. Unless I miss understand what it's doing.

In almost every system I have used, mass divisioning into hierarchical caches and large abstract datastructures and then putting them on a disk is ALWAYS a mistake in smaller systems. It has been learnt and learnt again. It will be learnt and learnt again. Examples off the top of my head include:
Windows registry.
Gentoo portage database.
Mail queues.
Proxy caches.
Page files.

This is one of the problems with linux (and with users like me!), we install the bells and whistles. The same bells and whistles which are quite often sourced into the linux kernel and utilities from enterprise users in the likes of IBM, Microsoft, Amazon, HP, Dell and so on to meet demands on their high enterprise cloud consumers. We then turn them on and ignore all the warnings about "you need PROPER" hardware to use this and you will not take any advantage of it on a small system. Use only to learn. (Aka, kubernetes cluster for a home dev server).

For systems that NEED to use mass division of storage for distribution, caching, verification and redundancy, typical need to have enough memory to cache the whole filesystem folder for same in RAM. On a "commodity" Dell Power Edge these days you are looking at 64 cores and 256Gb of RAM and it will have no trouble with 200Gb PBS data.

So, again, specific to my needs for backups. I don't see an advantage over the cost. I am not spending money on good SSDs or M.2 drives for a server which runs for about 45 minutes a day.

 

[Dunuin]

Putting this on a standard consumer SSD will see it die early. Unless I miss understand what it's doing.

You can use consumer SSD but this wouldn't be great for ZFS. And without the selfhealing filesystem feature of ZFS I wouldn't run a PBS unless you got multiple copies of your backup snapshots spread across multiple datastores or even better multiple PBS.
Keep in mind that because of deduplication a single corrupted 4MB chunk file might prevent you from restoring all backup snapshots of that VM or even of multiple VMs. So you want a storage that can detect and repair those corruptions and those usually prefer proper enterprise hardware.

 

[venquessa]

The current backup array is 3x2Tb Seagate Barracuda 7500rpm "green" compute HDDs. In a RAIDz1 config. It's only powered on for backups and then shutdown, so zero disk wear. Other than those 3 I have to 2x6Tb HDDs (same model) for bulk media storage of which (due to the powers of ZFS volumes and how easy they are to move around, is "offlined" as all it's stuff moved to a 4Tb SSD. (Samsung evo). No redudunency on these and nothing ireplacable.

On "killing SSDs".

I stand guilty as charged. A cheap "Amazon ValueTechBasics 250Gb" drive which came "Free" with something I bought. Stupid for using it in the first place, but I put my influxdb on it. I then enabled the influx internal monitor database and forgot about it.

If you have never done this, don't. The monitor database on a busy influx server is MASSIVE, many times the size of the data itself in my case. Recieving around 3000 metrics a minute, 50/s results in about 100 writes into the monitor database alone every second. This caused the docker stack to "out of memory" and spill to disk on almost all reads and writes. Resulting in the HD light being on 90% of the time.

Result:

6 months and it died in flight badly. My backups saved me and I only lost 24 hours. But it took me 6 hours to restore a new setup as the main "service configs, docker stacks" where on that drive.

This is exactly what started me on what I now call, "Network 2024". It's time to get a little more serious. But not "Enterprise SSD" serious

 

[venquessa]

The self healing has already saved me.

I tried to use True NAS as a backup server. It also trashed the disk and took ages to clean up. Infesting it with all manor of "legacy" ZFS volumes and then corrupting it when exported. 2 out of 3 disks lost data. ZFS appologised and my heart sank. Then it said it had to lose 5 seconds of data. My heart rose.

On a single disk that was "Game over". A whole disk initialised as "INVALID" and corrupt. Yes ZFS fixed it and corrected the array with minimal loss.

 

[Neobin]

Thanks for letting the readers know, that PBS does not fit you, or better said, your hardware does not fit PBS. Or is there an actual question/problem, I missed?

It is btw also no wonder, since you stack up all the worst things that are possible:

• Not only HDDs, but even SMR-HDDs
• connected through USB
• with not only ZFS on top, but even ZFS-raidZ
• and using this with millions of small files

 

[Dunuin]

It is btw also no wonder, since you stack up all the worst things that are possible:

• Not only HDDs, but even SMR-HDDs
• connected through USB
• with not only ZFS on top, but even ZFS-raidZ
• and using this with millions of small files

Could be worse. PBS could run as a VM using ZFS on top of ZFS

But yes...thats nearly the worst setup you could imagine for IOPS performance.

 

[venquessa]

• and using this with millions of small files

Well, that part was PBS.

If you want to be snarky about it. At least it serves as a warning to others. Only actual enterprise folks may apply. That or stupid domestic users with too much money. If you have SOHO/Consumer hardware, look else where.

And that's fine. Maybe I should have looked at the documentation more closely.

To be honest, TrueNAS was far, far worse. It's basically "let's set the highest possible standards and make the user feel bad when they don't make them." I would also highly doubt that any enterprise that could make use of the complexity it offers would not touch it with a barge pool, seeking instead for commercial alternatives from Cisco, HP, Dell et. al or plane off load to cloud that install TNAS on hardware capable of using it fully. Before anyone mentions LTT ... you realise they only use that "pro-sumer" crap because they want to make videos about it and if their server room was anything more than a prop it would be running far more enteprise level SAS/NAS/SAN in a rack.

In the consumer space this feels familar to... what is it... can't put my finger on it yet.... oh... ah... audiophiles. We now have nas-phile who insist on you must have 10 matching batch number, enterprise SSDs to create a reliable performan home media centre NAS.

/rant.

It's Friday, I need to get it out of the system so I can replace it with some beer and cheer up

 

[venquessa]

Back on point.

HDDs on USB in general and PVE/PBS.

I have noticed and followed many, many articles arguing about PVE's use of ... not quite deprecated "DO NOT USE" kernel O_DIRECT write functions and the like.

In the case of an HDD that is going to be extremely expensive. That is basically cutting through the entire filesystem cache layer and the C function which calls it does not return until it's written. If you consider that happens for every other IO operation it would explain why copying to or from a USB disk on the PVE node causing a load factor of 12. (100% CPU) of which 98% is IOWait.

 

[Neobin]

At least it serves as a warning to others. Only actual enterprise folks may apply. That or stupid domestic users with too much money.

Not that I feel addressed and I can understand your annoyance that the software (whether it is PBS or TrueNAS or ZFS in general), that you would like to use, does not play well or at all with your given hardware, but that is not a reason to become insulting.

But it is, what I first thought: Just another rant thread...
Have a nice weekend.