PBS backups failing and bringing down the PBS UI

[inxsible]

It's stuck at this:

[pbs: ~]── - curl -kv https://localhost:8007
*   Trying 127.0.0.1:8007...
* Connected to localhost (127.0.0.1) port 8007 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):

I am able to get the html when a backup is NOT running.

 

[dcsapak]

It's stuck at this:

[pbs: ~]── - curl -kv https://localhost:8007
*   Trying 127.0.0.1:8007...
* Connected to localhost (127.0.0.1) port 8007 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):

I am able to get the html when a backup is NOT running.

how long did you leave it at that?

also just noticed this:

MiB Mem : 512.0 total, 434.8 free, 37.1 used, 40.1 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 474.9 avail Mem

i don't think 512 MiB is enough for PBS to run properly, the min requirements say 2GiB minimum. (see https://pbs.proxmox.com/docs/installation.html)
could you maybe give pbs more memory and test again?

 

[inxsible]

how long did you leave it at that?

2-3 mins or so.

i don't think 512 MiB is enough for PBS to run properly, the min requirements say 2GiB minimum.

Yes, I know. But this was an instance that was running just fine and even during a backup the RAM used would be around 14-15%.
I will try adding more RAM and see but I don't think that's going to help with this particular issue.

 

[dcsapak]

mhmm.. ok thats super weird... how many cores does the pbs have?

 

[inxsible]

Sorry for the delay in my response.

I doubled the RAM to 1GB and increased the cores to 2 and I got the same result. No GUI access immediately after starting a backup. The datastore contents page was already open when I started the backup and I got the below error



curl -kv localhost:8007 hangs obviously and I have to restart promox-backup-proxy

Then as a test, I increased RAM to 4GB and added 2 more cores for a total of 4 cores. Exact same results !!!



Do you think I should remove the PBS setup completely and re setup the whole datastore etc?

EDIT: I updated PBS to 3.0-2 today, but the problem persists.

 

[dcsapak]

do you have some reverse proxy in front? the messgae '503 service unavailable' does not seem to come from us...

 

[inxsible]

@dcsapak, Yes I am using HAProxy plugin on Opnsense as a reverse proxy for all my local services for SSL offloading.

 

[dcsapak]

ok i just wanted to make sure that message is not from us somewhere.

the thing is, the behavior is definitely not normal and the only thing i can think of why it could happen would be that a backup completely saturates some resource (disk/cpu/ram/network) so that a normal api call/etc cannot be done anymore...

 

[inxsible]

ok i just wanted to make sure that message is not from us somewhere.

the thing is, the behavior is definitely not normal and the only thing i can think of why it could happen would be that a backup completely saturates some resource (disk/cpu/ram/network) so that a normal api call/etc cannot be done anymore...

I agree it's confounding.

Also it started happening after I shutdown both my NAS and Proxmox server for about 10 days. I did the update after the fact, so it's not related to the update to 8.0 either. For it to seemingly stop working without any config changes is very perplexing.

At this point, I think I will just remove the pbsDatastore and the PBS LXC and readd them from scratch to see if that helps. However, I plan to first move my NAS to a different server, so this whole thing could take a while. Meanwhile I am left without backups when I need them the most

Oh well, I will just do manual backups before any and all migrations as the manual backups of the CTs & VMs to the NAS work just fine.