How to improve NFS performance for PBS backups?

[thimplicity]

Hi everyone,
I am running PBS as a VM on Proxmox and I am also running several other VMs on that Proxmox server. All VMs run on an NVMe drive. I also run Truenas Scale as another VM on the same server. Truenas has a few NFS shares and I use one of the NFS shares (HDD Raid1) to create backups of all my VMs across different servers. The NFS performance is pretty terrible though, about 20-40 MB/s.

I know that NFS is not the preferred way to run PBS backups, but are there any tips or tricks to improve performance?

Thanks in advance for some thoughts!

 

[UdoB]

about 20-40 MB/s.

That's usually irrelevant. In real situations the operations-per-second possible decide how how fast a backup progresses. IOPS are king! The reason has been discussed several times: PBS works with relatively small chunks and each and every one needs one (or several!) disk operation.

You didn't tell us what kind of disks TrueNAS uses.

but are there any tips or tricks to improve performance?

Use 100 GBit/s networking instead of the slow 10 GBit/s. Just kidding...

Proxmox recommends local SSD/NVMe for PBS - https://pbs.proxmox.com/docs/system-requirements.html#recommended-server-system-requirements. The main reason is the mentioned IOPS of local storage. Remote NFS/CIFS/whatever storage is always much slower.

Try to run "fio" on the PBS to test the behavior of the mounted NFS store. Does it give two, three or four digit IOPS for read and for write?

You may try to enable "write caches" on TrueNAS and on the NFS connection. For enhanced data security they are disabled. Personally I prefer data integrity over speed. Note that this does not speed up reading data.

Good luck!


Disclaimer: actually I do also use NFS on some PBS. But I know what to expect and I live with the poor result...

 

[thimplicity]

Thanks for the feedback - I use HDDs on the TrueNAS server. Maybe will also need to live with the speed, until I am able to replace those HDDs with SDDs (at least for the PBS portion). Will try fio next!

 

[VictorSTS]

I am running PBS as a VM on Proxmox and I am also running several other VMs on that Proxmox server. All VMs run on an NVMe drive. I also run Truenas Scale as another VM on the same server. Truenas has a few NFS shares and I use one of the NFS shares (HDD Raid1) to create backups of all my VMs across different servers. The NFS performance is pretty terrible though, about 20-40 MB/s.

This is a very bad idea on it's own: if your PVE fails you will have to recover the Truenas VM first to be able to access the disks were your PBS backups are... from where? Where would you create your Truenas backups? Would them include all the drives, even those used for the PBS datastore?

Just buy an external USB drive, pass it through to your PBS VM and store your backups there. You can import that datastore to a completely different PBS server if need arises, you'll get a much higher change of recovery.

 

[thimplicity]

This is a very bad idea on it's own: if your PVE fails you will have to recover the Truenas VM first to be able to access the disks were your PBS backups are... from where? Where would you create your Truenas backups? Would them include all the drives, even those used for the PBS datastore?

Just buy an external USB drive, pass it through to your PBS VM and store your backups there. You can import that datastore to a completely different PBS server if need arises, you'll get a much higher change of recovery.

Thanks for the feedback - Is it possible to just change the folder of a datastore in PBS (and switch it to an SSD), so that I can keep all my storage and backup jobs as-is? Would like to avoid to set up everything again from scratch. Is it possible to move the data as well or do I have to create all backups again?

 

[VictorSTS]

so that I can keep all my storage and backup jobs as-is?

You can keep the backup job in PVE, but you will have to edit it to use the new PBS datastore that you will have to create on the USB drive.

Is it possible to move the data as well or do I have to create all backups again?

Absolutely: use a sync job to copy all your backups to the new drive.

 

[thimplicity]

You can keep the backup job in PVE, but you will have to edit it to use the new PBS datastore that you will have to create on the USB drive.

I will just give PBS a portion of an NVMe on my server instead of USB.

Absolutely: use a sync job to copy all your backups to the new drive.

Just to make sure I get the steps right

1. Delete the current datastore, but do not delete the data (kind of obvious)
2. Create folder on the new drive (see above)
3. Create new datastore linked to the folder in 2.
4. Use an internal sync job to copy within PBS

Open questions:
- Are the steps above correct?
- If I allocate a size to the NVMe and that fills up, will PBS automatically delete older backups to make space for new ones or will it not be able to add new ones?
- I used encryption for the backups as I plan to rclone them to OneDrive - how can I transition the key to the new datastores? Just upload them as part of the creation process?
- How do I need to edit the existing jobs to linke them to the new datastores?

 

[VictorSTS]

I will just give PBS a portion of an NVMe on my server instead of USB.

What would you do if the NVMe drive that hosts both the VMs and their backups fails? What if the whole host fails? Backup must be outside PVE host to be useful.

1. Delete the current datastore, but do not delete the data (kind of obvious)
2. Create folder on the new drive (see above)
3. Create new datastore linked to the folder in 2.
4. Use an internal sync job to copy within PBS
Open questions:
- Are the steps above correct?

Step 1 must be moved to 4, so the order would be 2, 3, 4, 1. You can't sync from a deleted datastore

- If I allocate a size to the NVMe and that fills up, will PBS automatically delete older backups to make space for new ones or will it not be able to add new ones?

No, PBS does not auto expire backups if datastore gets full. You must keep an eye on space and adapt your retention policy to the available storage in the datastore. If datastore gets completely full you may not even be able to (easily) run GC to expire old backups.

- I used encryption for the backups as I plan to rclone them to OneDrive - how can I transition the key to the new datastores? Just upload them as part of the creation process?

Yes [1]

- How do I need to edit the existing jobs to linke them to the new datastores?

Add a new storage in PVE of type PBS with the new datastore. Then edit PVE backup jobs to use this new storage. Then remove the old PVE storage of type PBS that uses the old datastore.



[1] https://forum.proxmox.com/threads/solved-share-encryption-key-between-two-pbs.86893/post-381053