Unable to display backups to PBS in CT/VM Backup section

[gmpreussner]

The "Backup" sections for my virtual machines and containers are not populating.
After the spinner is shown for a while, I get communication failure (0) or Connection Timeout (596).

Kernel Version: Linux 5.15.108-1-pve #1 SMP PVE 5.15.108-1 (2023-06-17T09:41Z)
PVE Manager Version: pve-manager/7.4-16/0f39f621

Things that work:
* My nightly incremental CT and VM backups work without problems
* Proxmox GUI correctly shows active backup jobs and backup log outputs
* I am able to log into the PBS Web GUI, and all of its features work
* The backup task is correctly shown in the Datacenter > Backup section
* The backup storage is correctly shown in the Datacenter > Storage section
* I have the correct backup storage selected in the "Storage" drop-down at the top right of the "Backup" section

Things that don't work:
* `Container > Backup` section never populates (connection timeout)
* `VM > Backup` section never populates (connection timeout)

I suspect that it may have to do with my firewall settings. I have the PVE firewall enabled for the datacenter and all nodes, VMs, and CTs. All inbound/outbound is dropped by default, and I use security groups to enable specific inbound/outbound traffic. Corosync, PVE Management, and Backup are running on separate networks. I currently allow inbound connections on TCP/8007 to PBS from all networks in the PBS container's firewall settings.

I have been unable to determine how the Proxmox GUI on the Backup section connects to PBS in order to query the collection of available backups per CT/VM. Since the PBS GUI works without problems from my management workstation's web browser, I presume that it is not the web browser querying PBS, but the PVE web service on the node.

Do I need specific firewall rules on the PVE nodes themselves (I tried to enable outbound TCP/8007 on the node's firewall settings)?
Are there any other ports I need to open for the Backup section to populate successfully?

 

[fabian]

most likely your PBS is just too slow - that content view does a synchronous API call, and that has a timeout of 30s.. does "pvesm list STORAGE" where STORAGE is your PBS storage name work on the CLI? if so, how long does it take (you can measure by adding "time" to the front of the command)?

 

[gmpreussner]

most likely your PBS is just too slow - that content view does a synchronous API call, and that has a timeout of 30s.. does "pvesm list STORAGE" where STORAGE is your PBS storage name work on the CLI? if so, how long does it take (you can measure by adding "time" to the front of the command)?

Thanks for your quick response, Fabian! I got some interesting results...

My system was performing backups, so it was a good opportunity to test the behavior under load. The initial query took 45 seconds. All subsequent queries took between 15 and 18 seconds. It seems that, after the initial query, the results were cached, so for the very first time since installing Proxmox (I've been running it for over a year) I am now able to see the Backup section being populated for CTs and VMs, and the results show up near instantaneously within one second.

I then waited for the backup tasks to complete, and now the pvesm queries take between 0.9 and 1.3 seconds each. I can still see the Backup sections being populated within one second.

Previously, the Backup section never populated and always timed out, even when the container was idle at 0% CPU.

I currently have 4 x 2GHz cores assigned to the container, and it's at 0% CPU at idle, and 35% when running backing tasks. There is currently a total of 86 containers/VMs being backed up, and I did not previously have a retention policy configured, so it kept a very large number of nightly incremental backups. I just configured the retention policy to dramatically reduce the total number of backups.

Do you think that running pvesm list from the console once put Proxmox into a state where it can now successfully query CT/VM backups?
Is there anything else I can do to ensure that backups are listed consistently in the future?

 

[gmpreussner]

It looks like my excitement was premature. It turns out that the containers/VMs for which the Backup section populates (and does so quickly) are only those that I created recently, and which have very few backups so far. The older containers/VMs with lots of backups still time out on the GUI.

I deleted all backups but the last one, and now all CT/VM populate instantaneously.

It seems to me that there is a general performance issue with retrieving backup lists from PBS. How can it be so slow?

 

[fabian]

It looks like my excitement was premature. It turns out that the containers/VMs for which the Backup section populates (and does so quickly) are only those that I created recently, and which have very few backups so far. The older containers/VMs with lots of backups still time out on the GUI.

I deleted all backups but the last one, and now all CT/VM populate instantaneously.

It seems to me that there is a general performance issue with retrieving backup lists from PBS. How can it be so slow?

it is so slow if your random I/O is slow - accessing the backup information requires accessing a lot of metadata files on the PBS end. the solution for such slow systems would be to convert the calls on the PVE side to be async and caching the results.