PVE cluster fails to connect PBS

[barlund]

Hello, I have several Proxmox virtual environment clusters (8.4) with VMs and I'd like to backup those to one PBS (4.0.15). First pve-cluster1 can be connected to PBS ok, but no luck with other clusters?

Seems that in PBS there is no firewall blocking this connectivity, any idea why this is not working? tcpdump from PBS (10.22.69.62) shows that pbs is not answering pve (10.22.69.68) requests:



In PBS side, storage disks are logically separated to different disks:
root@pbs:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 446.6G 0 disk
├─sda1 8:1 0 1007K 0 part
├─sda2 8:2 0 1G 0 part
└─sda3 8:3 0 445.6G 0 part
├─pbs-swap 252:4 0 8G 0 lvm [SWAP]
└─pbs-root 252:5 0 421.6G 0 lvm /
sdb 8:16 0 14T 0 disk
├─data-cluster1 252:0 0 1.4T 0 lvm /cluster1
├─data-cluster3 252:1 0 3T 0 lvm /cluster3
├─data-cluster4 252:2 0 3T 0 lvm /cluster4
└─data-cluster5 252:3 0 3T 0 lvm /cluster5



BR
Jonas

 

[Onslow]

From the screenshot you included, it doesn't necessarily mean that "pbs is not answering pve (10.22.69.68) requests".

What is in the logs in the PBS at the moment you are (unsuccessfully) trying to connect to the PBS? E.g. in journalctl, /var/log/proxmox-backup/api/access.log

What are the permissions in the PBS on "Datastore" tree?

What (in detail) are you doing at the cluster side before you're getting this failure?

 

[barlund]

Try to connect and checked logs. No writings to journalctl nor access.log from 10.22.69.68.
Goes like this:


PBS side datastore tree:


If there should be found some specific log writings I supposed to hunt, please reply.

Thanks
Jonas

 

[Onslow]

If in the "unlucky" PVE host you execute in the shell
telnet 10.22.69.62 8007
- what will you get in the screen?

 

[barlund]

Tried telnet from 2 different PVE hosts.

root@proxm-01-cluster6:~# telnet 10.22.69.62 8007
Trying 10.22.69.62...
Connected to 10.22.69.62.
Escape character is '^]'.

Connection closed by foreign host.



PBS journalctl:
root@pbs:~# journalctl -r

Sep 29 17:23:12 pbs proxmox-backup-proxy[1682]: [[::ffff:10.22.69.68]:39034] failed to check for TLS handshake: timed out while waiting for client to initiate TLS handshake

 

[Onslow]

If the message "Connection closed by foreign host" was NOT immediately after you executed telnet, then we at least can assume the network lets the connection through OK.

Another try: from the "unlucky" PVE execute
proxmox-backup-client list --repository "USER@REALM@HOST:DATASTORE"

What gives?

(This suggestion is from a similar thread https://forum.proxmox.com/threads/p...-500-can-not-get-datastores.93854/post-408245
and in that thread the reason were jumbo frames; of course it may be anything other in your situation).

If the result of above proxmox-backup-client command is successful and creating the storage still fails, I would compare very carefully particular "lucky" clusters and nodes with "unlucky" ones - for any differences in settings and configurations (also in attempted storages).