Is PBS killing windows or is windows killing PBS

[approximater]

We have a proxmox server running one VM of windows server 22 and one vm running PBS. The proxmox server occasionally filled up its 32GB disk with log files so I set up a datastore on the proxmox host and shared it out to PBS via NFS. At the same time I set up another NFS share on the proxmox host for backup storage as backing up to an external synology NAS (also via NFS) was glacial.
About 10 days ago the hard drive filled up on the proxmox host and not only did the backup fail the windows server became corrupted and wouldn't boot saying
C:\Windows\System32\config\SYSTEM was corrupted. Obviously we couldn't locate the files because it was hidden and who would ever need to restore a system file right? If windows marks it as hidden I guess the smart thing to do is not make it findable in the backups. So we did a full restore from the NAS and 30 hours later had our windows server running again. I changed the backup to only store 1 copy on the local drive and sync that to the NAS which stores the last 4 backups and then last night it happened again.
Only this time the single backup was only taking ~600GB space and apparently successfully completed at 10pm but the PBS logs share had a 300GB+ directory in it that filled the drive sometime between midnight and 7am this morning and once again the windows server is refusing to boot with a missing/corrupted C:\Windows\System32\config\SYSTEM
Restoring from the hard drive only took 49 minutes this time but this is about the 4th time an errant PBS process has completely filled the hard drive with GB's worth of logs and the second time it has destroyed a VM. At this point it's getting embarrassing as I was the one that recommended PBS as suitable for production and it's clearly not. Unfortunately I haven't been able to study the logs when they fill up as getting the server working has been the priority and I've had to delete them. This time I've set up a quota of 20GB on the logs dataset and will able to give it a fresh dataset without deleting next time it happens but wondering if anybody knows why PBS would generate 300GB of logs in one night and/or why running out of disk space ends up corrupting files on a VM?

Additional information that may or may not be pertinent is the windows server completed several windows updates when it restored from last nights backup and the server is blocked from sending SMTP so both Prpxmox and PBS have been configured to send notifications via webhooks which has been successfully sending notifications about backups, syncs and garbage collection with no indication of impending disaster.

 

[Chris]

Hi,

We have a proxmox server running one VM of windows server 22 and one vm running PBS. The proxmox server occasionally filled up its 32GB disk with log files so I set up a datastore on the proxmox host and shared it out to PBS via NFS. At the same time I set up another NFS share on the proxmox host for backup storage as backing up to an external synology NAS (also via NFS) was glacial.

what version of PBS are you running, did you check which log files fill up the disk? Further, please note that running PBS with a datastore located on a NFS is problematic for performance. Further, I assume your NFS share is backed by spinning disks? Both will not cope well with PBS, as IO will be your main bottleneck.

About 10 days ago the hard drive filled up on the proxmox host and not only did the backup fail the windows server became corrupted and wouldn't boot saying
C:\Windows\System32\config\SYSTEM was corrupted.

Are the windows VM and the PBS datastore located on the same disk/storage? Could it be that when you are filling up the disk, the Windows VM can no longer write to disk?
Do note that is not recommended at all to use the same storage you use for VM disks also for backups, if that disk fails, your VMs AND backups are gone!

Obviously we couldn't locate the files because it was hidden and who would ever need to restore a system file right? If windows marks it as hidden I guess the smart thing to do is not make it findable in the backups. So we did a full restore from the NAS and 30 hours later had our windows server running again. I changed the backup to only store 1 copy on the local drive and sync that to the NAS which stores the last 4 backups and then last night it happened again.
Only this time the single backup was only taking ~600GB space and apparently successfully completed at 10pm but the PBS logs share had a 300GB+ directory in it that filled the drive sometime between midnight and 7am this morning and once again the windows server is refusing to boot with a missing/corrupted C:\Windows\System32\config\SYSTEM
Restoring from the hard drive only took 49 minutes this time but this is about the 4th time an errant PBS process has completely filled the hard drive with GB's worth of logs and the second time it has destroyed a VM.

Again, do not use the same drive for backups. And identify what logs are filling up your system. Older versions of PBS were rather verbose in the backup task log, not the case anymore for an up to date one.

At this point it's getting embarrassing as I was the one that recommended PBS as suitable for production and it's clearly not.

Your current setup is however very much not recommended, see https://pbs.proxmox.com/docs/installation.html#recommended-server-system-requirements

Unfortunately I haven't been able to study the logs when they fill up as getting the server working has been the priority and I've had to delete them. This time I've set up a quota of 20GB on the logs dataset and will able to give it a fresh dataset without deleting next time it happens but wondering if anybody knows why PBS would generate 300GB of logs in one night and/or why running out of disk space ends up corrupting files on a VM?

Additional information that may or may not be pertinent is the windows server completed several windows updates when it restored from last nights backup and the server is blocked from sending SMTP so both Prpxmox and PBS have been configured to send notifications via webhooks which has been successfully sending notifications about backups, syncs and garbage collection with no indication of impending disaster.

 

[approximater]

Hi,

what version of PBS are you running, did you check which log files fill up the disk? Further, please note that running PBS with a datastore located on a NFS is problematic for performance. Further, I assume your NFS share is backed by spinning disks? Both will not cope well with PBS, as IO will be your main bottleneck.

Running PBS 9.0. I'm aware the NFS and spinning disks are not ideal but it's a case of using what the hardware I have, not the hardware I wish I had. Basically 1 pc with 2 2TB drives, a synology NAS and 2 usb drives for offsite backups. I'd like to pull the drives out of the synology and put them in the server rather than using them over NFS but that's not possible unfortunately.

Are the windows VM and the PBS datastore located on the same disk/storage? Could it be that when you are filling up the disk, the Windows VM can no longer write to disk?

Yes that's no doubt the cause of the windows corruption.

Do note that is not recommended at all to use the same storage you use for VM disks also for backups, if that disk fails, your VMs AND backups are gone!

Duly noted, that's why after a backup to the local disk at 10pm it is then synced to the NAS in the early am and synced again to an external USB in the morning when plugged in. Initially backups were going straight to the NAS over NFS but that put the VM in a degraded state for several hours.

Again, do not use the same drive for backups. And identify what logs are filling up your system. Older versions of PBS were rather verbose in the backup task log, not the case anymore for an up to date one.

Your current setup is however very much not recommended, see https://pbs.proxmox.com/docs/installation.html#pbsrecommended-server-system-requirements

Next time it happens I'll be able to preserve the logs and be back with more information but given PBS has filled it's available storage to capacity 3 times now and the latest effort was generating 300GB worth of of logs between 12am and 7am after a successful backup I have to wonder what kind of setup it takes to reliably run PBS. 300TB?

 

[Chris]

Running PBS 9.0

There is no PBS version 9.0, I assume you are referring to Proxmox VE running on the host here. Please post the output of proxmox-backup-manager version --verbose run on the PBS host.

Next time it happens I'll be able to preserve the logs and be back with more information but given PBS has filled it's available storage to capacity 3 times now and the latest effort was generating 300GB worth of of logs between 12am and 7am after a successful backup I have to wonder what kind of setup it takes to reliably run PBS. 300TB?

No, something seems wrong, as already stated you will have to check what logs are generated... Without more details it will not be possible to help. Did you already try to identify what is fulling up the space by tools like e.g. ncdu? Do you see errors in the systemd journal?

I'm aware the NFS and spinning disks are not ideal but it's a case of using what the hardware I have, not the hardware I wish I had. Basically 1 pc with 2 2TB drives, a synology NAS and 2 usb drives for offsite backups. I'd like to pull the drives out of the synology and put them in the server rather than using them over NFS but that's not possible unfortunately.

But you should not complain about the poor performance either then