PBS crash crashes also pvestatd on PVE nodes

[flames]

Hello,

in my testlab I have 4x PVE 6.2.x with Ceph 3 OSDs each and PBS on a separate physical machine 2x4tb hdd ZFS mirror, everything updated on daily basis from no-subscription repo.

PBS hangs sporadically once or twice a day (no ping, no GUI, no SSH, need to reset machine), but this is not the issue, beta software here, and i could not yet track why PBS crashes. Might be a hardware problem. Old crappy hardware it is, good for testing.

The actual issue is, that on the PVE cluster pvestatd crashes, when PBS not available (question marks appear everywhere and no storages are accessible via gui), until i reboot PBS and restart pvestatd on all PVE nodes.

Is this a know issue? Guess pvestatd should not crash, but mark PBS storage as offline, as it happens when a CIFS/NFS storage is not accessible. It persists for me since day one of PBS release.

Thanks in advance

 

[thexa4]

I had the same issue with the backup server hanging, I've opened a bug report here: https://bugzilla.proxmox.com/show_bug.cgi?id=2979

 

[flames]

Thanks for your work and ticket! I will try to trace back the hanging PBS and report back. I kind of mixed two different issues here, related, but different.
Crashing/hanging PBS is the one. The other and more important is, that pvestatd on the PVE nodes is crashing, when PBS is not available (also when i shutdown PBS for maintenance). But yes, the pb-client is also beta.
Edit: Currently i am reinstalling the PBS machine with PVE, and will install PBS in a VM + passthrough the ZFS disks to it, just to see, what happens.

 

[thexa4]

Yes, I have the same issue with pvestatd, it happens because the proxmox-backup-client keeps waiting indefinitely for a response when pvestatd asks for the amount of free space available. Killing either the client or pvestatd makes the nodes green again (temporarily).

I haven't submitted a bug report about that yet though.

 

[angelkiller]

Same here, when pbs goes down i can' t login to my pve server.

 

[flames]

@thexa4 I searched bugzilla, if the issue is already reported, nothing found and opened a bug report
https://bugzilla.proxmox.com/show_bug.cgi?id=3008

 

[Jarvar]

I've had this issue as well 2-3 times since the Backup Server was launched. Although restarting PVESTATD makes things green again I had complications until I essentially restarted the server. However, I am running most of my backups onto and external USB which may not be the best practice and could be the cause or the result. Not sure yet.

 

[t.lamprecht]

Yes, I have the same issue with pvestatd, it happens because the proxmox-backup-client keeps waiting indefinitely for a response when pvestatd asks for the amount of free space available. Killing either the client or pvestatd makes the nodes green again (temporarily).

This is a more general issue with pvestatd, also triggered by dead/temporarily unavailable NFS, where the kernel places the whole process in D state (uninterruptible IO).
We planned a redesign of it's architecture since a bit, but currently really thinking about going with rust to avoid some limitations of Perl with the main tradeoff being an not small increase in the initial work to do.
Anyway, here I think we should really checkout the client behavior, which my colleague Stoiko already tries to investigate within the bug report. Fixing that would help also other usages of the client, not only pvestatd.