PBS - new server

[Tage]

Hi

We have the following setup:

1 PVE Cluster with 4 nodes (each with 2 EPYC CPUs) and 256 GB RAM, and 8 x 2TB Enterprise NVMe u.2 disks) - storage CEPH on a separate 10GB network
1 PBS server integrated into PVE with 2 x 1TB (z1) SAS Enterprise disks and an HP LTO Tapedrive 15TB
1 TrueNAS Scale with 32 GB RAM and 12 x 2TB SATA Enterprise SSDs

All connected with a 10GB ethernet - all system are running the latest version of PVE and PBS.

The PBS server uses the TrueNAS server to store backups using a NFS share. It is slow and crashes frequently - and the whole PVE cluster "hangs".

We have purchased a new server for the PBS solution with 6 x 18TB spinning disk (storage) and 2 x 1TB NVMe drives for OS.

I need to remove the OLD PBS server from the PVE and would like some input on how to do this without causing problems for the PVE cluster.

 

[sb-jw]

We have purchased a new server for the PBS solution with 6 x 18TB spinning disk (storage) and 2 x 1TB NVMe drives for OS.

You won't have any fun with that either. I also had my PBS equipped with 8x 2 TB HDDs, after half a day I threw everything away and made SSD Only. I can therefore only recommend that you use SSDs and not HDDs. Thanks to deduplication etc. you don't need that much storage space. We have a dedup factor of 32, with over 2500 backups and 1.4 TB of storage used.

I need to remove the OLD PBS server from the PVE and would like some input on how to do this without causing problems for the PVE cluster.

Do you need any input regarding migration etc. or just how best to remove it from the hypervisor?

 

[ness1602]

Special device would help a bit, but all is depending on the restore speed you expect, eg if you're okay with an hour restore, then work with what you have.

 

[showiproute]

Keep in mind that live restore is also a possibility: While restoring you can already use the VM.

 

[Tage]

You won't have any fun with that either. I also had my PBS equipped with 8x 2 TB HDDs, after half a day I threw everything away and made SSD Only. I can therefore only recommend that you use SSDs and not HDDs. Thanks to deduplication etc. you don't need that much storage space. We have a dedup factor of 32, with over 2500 backups and 1.4 TB of storage used.


Do you need any input regarding migration etc. or just how best to remove it from the hypervisor?

Ok - will change to SSD's. But would like to have an idea on how to remove the "old" (but still in use) PBS server from our PVE - without causing problems with our datacenter/cluster.

 

[Dunuin]

Couldn`t you keep it running while migrating? So keep the old PBS running, set up the new PBS, add new PBS storages to your PVE cluster and edit your clusters backup jobs to backup to the new PBS instead of the old PBS. Set up the old PBS as a remote for your new PBS and then use a one-time sync job to pull all the existing backup snapshots from the old PBS to the new PBS? Then disable all old PBS storages in the PVe cluster and shutdown the old PBS.

 

[Tage]

Thanks, that is a good suggestion. Will "disabling" the old PBS storages be sufficient to "get rid" of the old PBS? I would prefer to have the old PBS completely removed for our PVE.

 

[Dunuin]

Thanks, that is a good suggestion. Will "disabling" the old PBS storages be sufficient to "get rid" of the old PBS? I would prefer to have the old PBS completely removed for our PVE.

Once the new PBS is running and new backup jobs are working you can simply delete the old backup jobs and old PBS storages.
In case the backups are encrypted make sure you got a backup of the encryption key.

 

[cfgmgr]

Couldn`t you keep it running while migrating? So keep the old PBS running, set up the new PBS, add new PBS storages to your PVE cluster and edit your clusters backup jobs to backup to the new PBS instead of the old PBS. Set up the old PBS as a remote for your new PBS and then use a one-time sync job to pull all the existing backup snapshots from the old PBS to the new PBS? Then disable all old PBS storages in the PVe cluster and shutdown the old PBS.

Did this several times recently and worked without issues. Would recommend as if there is issues you can just flip that job and safely fall back.

 

[cfgmgr]

You won't have any fun with that either. I also had my PBS equipped with 8x 2 TB HDDs, after half a day I threw everything away and made SSD Only. I can therefore only recommend that you use SSDs and not HDDs. Thanks to deduplication etc. you don't need that much storage space. We have a dedup factor of 32, with over 2500 backups and 1.4 TB of storage used.


Do you need any input regarding migration etc. or just how best to remove it from the hypervisor?

100% agree here - anything outside of SSD's means a very sad time for backups. We do have our lab cluster backing up to spinners which I'm not fond of but its tolerable (its lab). Prod is all NVMe specifically because some VMs had issues where they would slow down/hang up.

We have had zero problems since moving to NVMe/100G.

 

[Tage]

Ok - I think we will configure 6 x 7.86 TB NVMe disks as the "primary" backup storage and then copy the backups to our 6 x 18TB "spinners" for safe keeping for a month or so. That would solve the performance issue. We will also configure a LTO tape backup to make off-site backups.

The server will be delivered beginning next year (Supermicro server). We normally use HP Proliant servers, but due to their "vendor lock-in" hardware we have changed to Supermicro - and they are not using proprietary hardware.

Thanks very much for the suggestions.

 

[sb-jw]

We normally use HP Proliant servers, but due to their "vendor lock-in" hardware we have changed to Supermicro - and they are not using proprietary hardware.

Supermicro is currently making the transition to Dell and HP. There have already been some negative changes, e.g. the introduction of "complete systems only" which forces you to buy a system with a certain basic equipment, where the components cost correspondingly more. Then with newer systems you have to pay for certain functions in the IPMI. With the AOC, supermicro is also taking the step towards a proprietary connection.

Alternatively, AIC, AsRock or Gigabyte would be available, but here too it is certainly only a matter of time before these also become more expensive.