[SOLVED] Proxmox Cluster / TrueNAS & Proxmox Backup Server configuration

[gildra]

Hi

I have a Proxmox cluster of three servers, a TrueNAS (Storage) as VM and other network storage and a TrueNAS(Backup) server that is used to backup the TrueNAS storage server via ZFS Replication.

So currently my Proxmox VM are not really backuped besides their storage, so I wanted to look into PBS to finalize my backup strategy, however I am facing a few questions, that I can't find the answer to. There have been mayn question similar, but in the end their setup was usually different than mine and i wasn't sure if the answer translates to my situation as well.

I know that running Proxmox Backup server on bare metal with it's own storage is reccomended, however it is currently not possible to have it run on it's own hardware. Also, I am not sure it would actually be beneficial in my case as the backup storage is on it's own server with TrueNAS already.

So I am currently not sure how to go about solving my problem.

To be more precise, my questions are:

1. Should I virtualize PBS?
   1. If yes, should I put it on the Cluster or on the TrueNAS server? Jail? VM?
2. Does PBS need it's own storage attached directly to it?
3. Does it even make sense to have PBS on it's own hardware while the target backup storage is being provided by TrueNAS?
4. Can I configure PBS to Backup the VMs without storage?
   The VM storage itself is already being backup, meaning I would have a double backup if I can't do so. Should I deactivate ZFS replication in that case?
5. Which would be the best way to go about this? Has anyone a similar setup?
   

Greetings

 

[Dunuin]

Should I virtualize PBS?

1. If yes, should I put it on the Cluster or on the TrueNAS server? Jail? VM?

I've got a PBS VM running on my bare metal TrueNAS server. The PBS ISO wasn't working but you can install Debian 11 and install PBS using apt ontop.

• Does PBS need it's own storage attached directly to it?
  • Does it even make sense to have PBS on it's own hardware while the target backup storage is being provided by TrueNAS?

PBS was designed with local SSDs in mind. HDDs or network shares will work but are slow so maintaince tasks like GC/verify/prune might take hours or days depending on how much stuff you backup (here 700GB of deduplicated storage takes around 1-2 hours for a GC and I disabled verify tasks because it would take way to long and ZFS is already preventing bit rot)

• Can I configure PBS to Backup the VMs without storage?

You can exclude disks from backup. But a restore will wipe the complete VM with all it disks and then create a new VM based on the backup (where your excluded disks will be missing) so in case you exclude a virtual disk you will loose it on a restore.

• The VM storage itself is already being backup, meaning I would have a double backup if I can't do so. Should I deactivate ZFS replication in that case?

I personally replicate my PBS datastore to my second bare metal TrueNAS server so I don't loose my backups when the primary TrueNAS server dies.

 

[gildra]

I've got a PBS VM running on my bare metal TrueNAS server. The PBS ISO wasn't working but you can install Debian 11 and install PBS using apt ontop.

PBS was designed with local SSDs in mind. HDDs or network shares will work but are slow so maintaince tasks like GC/verify/prune might take hours or days depending on how much stuff you backup (here 700GB of deduplicated storage takes around 1-2 hours for a GC and I disabled verify tasks because it would take way to long and ZFS is already preventing bit rot)

I see, thanks for the feedback. I wasn't if network storage would cause issues but I assumed PBS was not designed for that way . So, I guess a VM on my TrueNAS backup server would be my best option in my case?

Did you just setup a dedicated Dataset on TrueNAS that you use as PBS VM Storage? And then I guess you also setup a Dataset for the actual PBS backups?

You can exclude disks from backup. But a restore will wipe the complete VM with all it disks and then create a new VM based on the backup (where your excluded disks will be missing) so in case you exclude a virtual disk you will loose it on a restore.

I'll probably disable ZFS replication of the VM storage as a whole in that case, since it will be backed up by PBS on a per VM basis.

Is PBS actually able to backup and restore a VM if its storage is not stored on the Proxmox PVE server itself?
Like the restore would have to restore the configuration to PVE and the storage to TrueNAS, is that actually supported?

I personally replicate my PBS datastore to my second bare metal TrueNAS server so I don't loose my backups when the primary TrueNAS server dies.

Is your VM storage located on the PVE or do you also use a network share for VM storage?
So your PBS backs up on the TrueNAS storage and then also backups to aTrueNAS. Do you use ZFS replication for the backup to TrueNAS from PBS?

Greetings and thanks in advance for the feedback.

 

[Dunuin]

I see, thanks for the feedback. I wasn't if network storage would cause issues but I assumed PBS was not designed for that way . So, I guess a VM on my TrueNAS backup server would be my best option in my case?

Yes, I guess.

Did you just setup a dedicated Dataset on TrueNAS that you use as PBS VM Storage? And then I guess you also setup a Dataset for the actual PBS backups?

I created a zvol (used with virtio SCSI by the VM) as the virtual root disk I installed PVE to.
Then I created a dataset for the backups (PBS datastore), shared it using NFS and mounted that NFS share inside my VM.
But if I would set it up again, I wouldn't use NFS and just create a big zvol formated with xfs instead, to store the backups on. That way it should be faster because of the missing NFS/network stack overhead.

Is PBS actually able to backup and restore a VM if its storage is not stored on the Proxmox PVE server itself?
Like the restore would have to restore the configuration to PVE and the storage to TrueNAS, is that actually supported?

Not sure about that. I think that depends on how you actually store the virtual disks. If you for example store qcow2 images on a SMB/NFS share or you use zvols using ZFS-over-iSCSI I would guess that PBS will delete your excluded virtual disks from your TrueNAS server on a restore.
If you want to exclude data from your TrueNAS from backups I would mount that data using NFS/SMB/iSCSI inside your guests.

Is your VM storage located on the PVE or do you also use a network share for VM storage?

PVE VM storage is local on the PVE node.

So your PBS backs up on the TrueNAS storage and then also backups to aTrueNAS. Do you use ZFS replication for the backup to TrueNAS from PBS?

PBS VM and PBS datastore is both on my main TrueNAS server. Then that main TrueNAS server replicates the PBS VMs virtual disks and the PBS datastore to my second backup TrueNAS server using the Replication build into TrueNAS.
But if you want better availability it would also make sense to disable replication and then run a PBS VM with a datastore each on both TrueNAS servers and then use PBSs sync job to keep both PBS in sync. Then you got a working PBS with all backups on both TrueNAS servers so your backups are always available even if one of the TrueNAS servers is down.

 

[gildra]

I created a zvol (used with virtio SCSI by the VM) as the virtual root disk I installed PVE to.
Then I created a dataset for the backups (PBS datastore), shared it using NFS and mounted that NFS share inside my VM.
But if I would set it up again, I wouldn't use NFS and just create a big zvol formated with xfs instead, to store the backups on. That way it should be faster because of the missing NFS/network stack overhead.

You are right a zvol would be the better option actually. I think I will do the same with the NFS share and back up the dataset via regular ZFS replication. I don't really have any experience with xfs, so I think i'll go the NFS route. Thanks for you input.

Not sure about that. I think that depends on how you actually store the virtual disks. If you for example store qcow2 images on a SMB/NFS share or you use zvols using ZFS-over-iSCSI I would guess that PBS will delete your excluded virtual disks from your TrueNAS server on a restore.
If you want to exclude data from your TrueNAS from backups I would mount that data using NFS/SMB/iSCSI inside your guests.

Now that I thnk about it I am confident that PBS will be able to restore the drives too, as they are mounted as storage in Proxmox. I use the qcow2 image for storing the drives. I have to research further on this topic to be sure.

PBS VM and PBS datastore is both on my main TrueNAS server. Then that main TrueNAS server replicates the PBS VMs virtual disks and the PBS datastore to my second backup TrueNAS server using the Replication build into TrueNAS.
But if you want better availability it would also make sense to disable replication and then run a PBS VM with a datastore each on both TrueNAS servers and then use PBSs sync job to keep both PBS in sync. Then you got a working PBS with all backups on both TrueNAS servers so your backups are always available even if one of the TrueNAS servers is down.

I think I will go with the first option. Running PBS on Storage TrueNAS and then replicate everything via TrueNAS built in replication to the Backup TrueNAS. Still, somehow it doesn't feel very clean as solution doe.

Too bad PVE doesn't have to possibility to schedule backups on a per VM basis.

Thanks for you help!

PS:
If anybody has an other setup or solution, feel free to post it anyway. I am always interested to see what other possibilities there are.

 

[UdoB]

Too bad PVE doesn't have to possibility to schedule backups on a per VM basis.

When you create a backup task via "Server View" --> "Datacenter" --> "Backup" --> "Create: Backup Job" you can select which VMs shall be included. Of course you need to do this for each and every single VM if you want independent schedules or options.

Personally I do backup of pools. All VMs in my "linux-important" pool are backuped continously one after another, usually daily. The pool named "linux-test" will be backuped only once a week instead. Pools can have virtually any number of VMs.

Best regards

 

[gildra]

When you create a backup task via "Server View" --> "Datacenter" --> "Backup" --> "Create: Backup Job" you can select which VMs shall be included. Of course you need to do this for each and every single VM if you want independent schedules or options.

Personally I do backup of pools. All VMs in my "linux-important" pool are backuped continously one after another, usually daily. The pool named "linux-test" will be backuped only once a week instead. Pools can have virtually any number of VMs.

Best regards

Thanks for the information, totally missed that function, lol. That's actually what I was looking for, however I will go down the PBS route, as it's definitely the more complete and fleshed out solution.

Greetings

 

[UdoB]

Thanks for the information, totally missed that function, lol.

I'm happy to help.
I do know this effect by myself very well: I am using Linux since 1997 and every now and then I realize that I did not know a commonly known and useful tool

 

[gildra]

I'm happy to help.
I do know this effect by myself very well: I am using Linux since 1997 and every now and then I realize that I did not know a commonly known and useful tool

Haha true, I have a cheatsheet for basic Linux commands and it help me out regularly

 

[chipbreak]

I created a zvol (used with virtio SCSI by the VM) as the virtual root disk I installed PVE to.
Then I created a dataset for the backups (PBS datastore), shared it using NFS and mounted that NFS share inside my VM.
But if I would set it up again, I wouldn't use NFS and just create a big zvol formated with xfs instead, to store the backups on. That way it should be faster because of the missing NFS/network stack overhead.

im planning to setup pbs a vm on my backup truenas server. how many gigs of ram did you allocate to the vm?

gildra interest to here your feed back too​

 

[Dunuin]

how many gigs of ram did you allocate to the vm?

3 vCPUs and 2GB RAM for the PBS VM with 1TB of backups.

 

[IvanF]

Hello Dunuin.

Sorry my English please.
Long time solved, but i have 1 question. You say:

You can exclude disks from backup. But a restore will wipe the complete VM with all it disks and then create a new VM based on the backup (where your excluded disks will be missing) so in case you exclude a virtual disk you will loose it on a restore.

That wipe VM configuration, or disks? When i have excluded disk, is posible after restore VM, simply reconect excluded disks and continue without data lost?

Thank you.
Ivan

 

[Dunuin]

Depends on how you use the disks. With disk passthrough it should only remove the disks from the configuration. With virtual disks it will wipe those virtual disks. See here: https://forum.proxmox.com/threads/feature-request-advanced-restore-options-in-gui.109707/

 

[IvanF]

Depends on how you use the disks. With disk passthrough it should only remove the disks from the configuration. With virtual disks it will wipe those virtual disks. See here: https://forum.proxmox.com/threads/feature-request-advanced-restore-options-in-gui.109707/

Thank you very much. Beautiful and informative reading.
I'm going to start reorganizing some VMs before disaster strikes.