Intermittent Timeouts Between PVE and PBS

A

aloanwolf

New Member
Feb 3, 2026
2
0
1
I recently spun up a PVE server with 4 VMs, and added PBS as a 5th. I created a datastore on PBS using an external drive that I am passing through PVE, created a "PVE" user with Datastore admin rights, and then added that as a Proxmox Backup Server storage location within PVE. I noticed when adding it (I've removed/re-added multiple times to troubleshoot this connectivity error), it can take a few tries before it no longer throws a 500 timeout error and successfully appears in PVE as a backup datastore.

Unfortunately, the timeouts seem to cause the backup jobs to either not start, or fail shortly after starting, showing a 500 timeout error. I believe I have the firewalls on the Datacenter and PBS VM disabled, as well. I've also tried changing the IP of the PBS server, just in case there could be an IP conflict, and the issue remains. (Note: I removed and re-added the datastore after the IP change.)

To rule out the external drive being the issue, I created/attached a new virtual hard disk to the VM and made that a new datastore (following the same procedure above), and I still get the timeout errors.

Any ideas why the connectivity issues appear to be intermittent? Please let me know if I can provide any additional information or logs to assist with troubleshooting. Thank you!

(As a side note, I'm aware having a backup server running on the same hardware is against best practice, this is mainly so I can test functionality for now.)
 
facing the same issue, but i though maybe it's my network! Im getting rapid notifications from monitoring service.
Code: 
PBS_rust_dir: error fetching datastores - 500 Can't connect to x.x.x.1x:8007 (Connection timed out)
tho host on which pbs is deployed(pxmx9.1) is healthy, without any issues.

How it looks like from PVE host perspective:1770103109187.png
 
Any errors in the systemd journal on the PBS host when the connection errors appear?
 
Chris said:
Any errors in the systemd journal on the PBS host when the connection errors appear?
Click to expand...
from PBS syslog i can't see any warning/errors!
PBS
Code: 
Feb 02 01:31:45 pbs systemd[1]: fstrim.service: Deactivated successfully.
Feb 02 01:31:45 pbs systemd[1]: Finished fstrim.service - Discard unused blocks on filesystems from /etc/fstab.
Feb 02 01:50:26 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.008 seconds)
Feb 02 02:17:01 pbs CRON[24977]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Feb 02 02:17:01 pbs CRON[24979]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Feb 02 02:17:01 pbs CRON[24977]: pam_unix(cron:session): session closed for user root
Feb 02 02:20:26 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 02:25:35 pbs systemd[1]: Starting man-db.service - Daily man-db regeneration...
Feb 02 02:25:35 pbs systemd[1]: man-db.service: Deactivated successfully.
Feb 02 02:25:35 pbs systemd[1]: Finished man-db.service - Daily man-db regeneration.
Feb 02 02:50:26 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 03:10:01 pbs CRON[25479]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Feb 02 03:10:01 pbs CRON[25481]: (root) CMD (test -e /run/systemd/system || SERVICE_MODE=1 /sbin/e2scrub_all -A -r)
Feb 02 03:10:01 pbs CRON[25479]: pam_unix(cron:session): session closed for user root
Feb 02 03:17:01 pbs CRON[25510]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Feb 02 03:17:01 pbs CRON[25512]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Feb 02 03:17:01 pbs CRON[25510]: pam_unix(cron:session): session closed for user root
Feb 02 03:20:27 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 03:50:27 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 04:17:01 pbs CRON[26302]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Feb 02 04:17:01 pbs CRON[26304]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Feb 02 04:17:01 pbs CRON[26302]: pam_unix(cron:session): session closed for user root
Feb 02 04:20:27 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.006 seconds)
Feb 02 04:50:27 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 05:17:01 pbs CRON[27154]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Feb 02 05:17:01 pbs CRON[27156]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Feb 02 05:17:01 pbs CRON[27154]: pam_unix(cron:session): session closed for user root
Feb 02 05:20:27 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 05:50:28 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 05:51:35 pbs systemd[1]: Starting proxmox-backup-daily-update.service - Daily Proxmox Backup Server update and maintenance activities...
Feb 02 05:51:46 pbs systemd[1]: proxmox-backup-daily-update.service: Deactivated successfully.
Feb 02 05:51:46 pbs systemd[1]: Finished proxmox-backup-daily-update.service - Daily Proxmox Backup Server update and maintenance activities.
Feb 02 05:51:46 pbs systemd[1]: proxmox-backup-daily-update.service: Consumed 916ms CPU time, 151.2M memory peak.
Feb 02 06:17:01 pbs CRON[28002]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Feb 02 06:17:01 pbs CRON[28004]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Feb 02 06:17:01 pbs CRON[28002]: pam_unix(cron:session): session closed for user root
Feb 02 06:20:28 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.008 seconds)
Feb 02 06:25:01 pbs CRON[28164]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Feb 02 06:25:01 pbs CRON[28166]: (root) CMD (test -x /usr/sbin/anacron || { cd / && run-parts --report /etc/cron.daily; })
Feb 02 06:25:01 pbs CRON[28164]: pam_unix(cron:session): session closed for user root
Feb 02 06:35:35 pbs systemd[1]: Starting apt-daily-upgrade.service - Daily apt upgrade and clean activities...
Feb 02 06:35:36 pbs systemd[1]: apt-daily-upgrade.service: Deactivated successfully.
Feb 02 06:35:36 pbs systemd[1]: Finished apt-daily-upgrade.service - Daily apt upgrade and clean activities.
Feb 02 06:42:35 pbs systemd[1]: Starting apt-daily.service - Daily apt download activities...
Feb 02 06:42:36 pbs systemd[1]: apt-daily.service: Deactivated successfully.
Feb 02 06:42:36 pbs systemd[1]: Finished apt-daily.service - Daily apt download activities.
Feb 02 06:50:28 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 07:17:01 pbs CRON[29122]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Feb 02 07:17:01 pbs CRON[29124]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Feb 02 07:17:01 pbs CRON[29122]: pam_unix(cron:session): session closed for user root
Feb 02 07:20:28 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.008 seconds)
Feb 02 07:49:28 pbs chronyd[540]: Selected source 162.159.200.123 (2.debian.pool.ntp.org)
Feb 02 07:49:28 pbs chronyd[540]: Source 80.233.131.171 replaced with 178.16.23.50 (2.debian.pool.ntp.org)
Feb 02 07:50:29 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 08:17:01 pbs CRON[29799]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Feb 02 08:17:01 pbs CRON[29801]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Feb 02 08:17:01 pbs CRON[29799]: pam_unix(cron:session): session closed for user root
Feb 02 08:20:29 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 08:50:29 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.011 seconds)
Feb 02 09:17:01 pbs CRON[30690]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Feb 02 09:17:01 pbs CRON[30692]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Feb 02 09:17:01 pbs CRON[30690]: pam_unix(cron:session): session closed for user root
Feb 02 09:20:29 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 09:50:29 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 10:17:01 pbs CRON[31520]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Feb 02 10:17:01 pbs CRON[31522]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Feb 02 10:17:01 pbs CRON[31520]: pam_unix(cron:session): session closed for user root
Feb 02 10:20:30 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 10:50:30 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 11:17:01 pbs CRON[32309]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Feb 02 11:17:01 pbs CRON[32311]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Feb 02 11:17:01 pbs CRON[32309]: pam_unix(cron:session): session closed for user root
Feb 02 11:20:30 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 11:50:30 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 12:17:01 pbs CRON[33094]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Feb 02 12:17:01 pbs CRON[33096]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Feb 02 12:17:01 pbs CRON[33094]: pam_unix(cron:session): session closed for user root
Feb 02 12:20:30 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 12:50:31 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.006 seconds)
Feb 02 13:17:01 pbs CRON[33972]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Feb 02 13:17:01 pbs CRON[33974]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Feb 02 13:17:01 pbs CRON[33972]: pam_unix(cron:session): session closed for user root
Feb 02 13:20:31 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 13:50:31 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 14:17:01 pbs CRON[34814]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Feb 02 14:17:01 pbs CRON[34816]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Feb 02 14:17:01 pbs CRON[34814]: pam_unix(cron:session): session closed for user root
Feb 02 14:20:31 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 14:50:32 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 15:17:02 pbs CRON[35565]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Feb 02 15:17:02 pbs CRON[35567]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Feb 02 15:17:02 pbs CRON[35565]: pam_unix(cron:session): session closed for user root
Feb 02 15:20:32 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 15:50:32 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.011 seconds)
Feb 02 15:52:04 pbs chronyd[540]: Source 178.16.23.50 replaced with 89.36.161.20 (2.debian.pool.ntp.org)
Feb 02 16:17:01 pbs CRON[36367]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Feb 02 16:17:01 pbs CRON[36369]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Feb 02 16:17:01 pbs CRON[36367]: pam_unix(cron:session): session closed for user root
Feb 02 16:20:32 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.010 seconds)
Feb 02 16:50:32 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.009 seconds)
Feb 02 17:17:01 pbs CRON[37095]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Feb 02 17:17:01 pbs CRON[37097]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Feb 02 17:17:01 pbs CRON[37095]: pam_unix(cron:session): session closed for user root
Feb 02 17:20:33 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 17:50:33 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 18:17:01 pbs CRON[37683]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Feb 02 18:17:01 pbs CRON[37685]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Feb 02 18:17:01 pbs CRON[37683]: pam_unix(cron:session): session closed for user root
Feb 02 18:20:33 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 18:50:33 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 19:17:01 pbs CRON[38300]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Feb 02 19:17:01 pbs CRON[38302]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Feb 02 19:17:01 pbs CRON[38300]: pam_unix(cron:session): session closed for user root
Feb 02 19:20:33 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.008 seconds)
Feb 02 19:50:34 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 20:17:01 pbs CRON[39087]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Feb 02 20:17:01 pbs CRON[39089]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Feb 02 20:17:01 pbs CRON[39087]: pam_unix(cron:session): session closed for user root
Feb 02 20:20:34 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.006 seconds)
Feb 02 20:50:34 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 21:17:01 pbs CRON[39816]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Feb 02 21:17:01 pbs CRON[39818]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Feb 02 21:17:01 pbs CRON[39816]: pam_unix(cron:session): session closed for user root
Feb 02 21:20:34 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.008 seconds)
Feb 02 21:50:35 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.008 seconds)
Feb 02 22:17:01 pbs CRON[40524]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Feb 02 22:17:01 pbs CRON[40526]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Feb 02 22:17:01 pbs CRON[40524]: pam_unix(cron:session): session closed for user root
Feb 02 22:20:35 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.010 seconds)
Feb 02 22:50:35 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 23:05:35 pbs systemd[1]: Starting apt-daily.service - Daily apt download activities...
Feb 02 23:05:36 pbs systemd[1]: apt-daily.service: Deactivated successfully.
Feb 02 23:05:36 pbs systemd[1]: Finished apt-daily.service - Daily apt download activities.
Feb 02 23:17:01 pbs CRON[41263]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Feb 02 23:17:01 pbs CRON[41265]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Feb 02 23:17:01 pbs CRON[41263]: pam_unix(cron:session): session closed for user root
Feb 02 23:20:35 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.007 seconds)
Feb 02 23:37:44 pbs systemd[1]: Starting systemd-tmpfiles-clean.service - Cleanup of Temporary Directories...
Feb 02 23:37:44 pbs systemd-tmpfiles[41396]: /usr/lib/tmpfiles.d/legacy.conf:14: Duplicate line for path "/run/lock", ignoring.
Feb 02 23:37:44 pbs systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Feb 02 23:37:44 pbs systemd[1]: Finished systemd-tmpfiles-clean.service - Cleanup of Temporary Directories.
Feb 02 23:50:35 pbs proxmox-backup-proxy[604]: rrd journal successfully committed (20 files in 0.008 seconds)
Feb 03 00:00:35 pbs systemd[1]: Starting dpkg-db-backup.service - Daily dpkg database backup service...
Feb 03 00:00:35 pbs systemd[1]: dpkg-db-backup.service: Deactivated successfully.
Feb 03 00:00:35 pbs systemd[1]: Finished dpkg-db-backup.service - Daily dpkg database backup service.
Feb 03 00:08:35 pbs systemd[1]: Starting logrotate.service - Rotate log files...
Feb 03 00:08:35 pbs systemd[1]: logrotate.service: Deactivated successfully.
Feb 03 00:08:35 pbs systemd[1]: Finished logrotate.service - Rotate log files.
HOST:
nothing else just time-out

Code: 
PBS_rust_dir: error fetching datastores - 500 Can't connect to x.x.x.1x:8007 (Connection timed out)
 
Double check if you have duplicate IP addresses (the MAC address in the output of ip neigh might give a clue if both hosts are in the same network) and check if you can ping the PBS host from the PVE host while the issue appears.
 
Chris said:
Double check if you have duplicate IP addresses (the MAC address in the output of ip neigh might give a clue if both hosts are in the same network) and check if you can ping the PBS host from the PVE host while the issue appears.
Click to expand...
I noticed that if I ran a ping from PVE to PBS, I would get major packet loss regardless if the job was running or not. It'll be consistent for 10-20 pings, then it'll drop for 30-40 pings, and resume, then repeat. The ping FROM PBS to PVE did not drop any packets, however. Both pings were running at the same time.

The output of "ip neigh" on PVE showed the MAC of PBS twice, once for IPv4 (which is what I'm using) and another for IPv6 (not utilizing), which was showing as STALE. The output of "ip neigh" on PBS showed an IPv6 address and IPv4 address for my router's MAC address, the IPv4 showing as stale while the IPv6 is REACHABLE.

I went ahead and flushed the cache to clear out the stale entries on both PVE and PBS. Rerunning the pings and backup job resulted in the same as before, unfortunately.

Performing a traceroute from PBS to PVE shows the single hop to PVE, however when running a traceroute from PVE to PBS, it does not return any hops and times out after the default 30 attempts. I tried running a traceroute from PVE to another (Windows) guest AND another device altogether, and it results in the same timeout after 30 attempts. The pings are successful, and under 1ms.

Definitely seems like PVE's network configuration is the culprit. Any ideas where I might look to make adjustments in PVE?

Edit: It looks like the traceroute and pings from PVE to PBS only act up whenever I'm accessing the datastore, either via running the backup job or trying to view it from the left-hand panel. Traceroute accurately shows the single hop from PVE to PBS, and all pings are fast/successful whenever that PBS datastore isn't being accessed.
 
Last edited:
aloanwolf said:
It looks like the traceroute and pings from PVE to PBS only act up whenever I'm accessing the datastore, either via running the backup job or trying to view it from the left-hand panel. Traceroute accurately shows the single hop from PVE to PBS, and all pings are fast/successful whenever that PBS datastore isn't being accessed.
Click to expand...
Do you see a significant increase in load on the PBS host while the ping drops packets? Communication issues at higher protocol level should not affect the ping at all. Double check also via tcpdump that the packets are entering/leaving the respective interfaces and bridges as expected and where the packet loss occurs.
 
You must log in or register to reply here.
Top Bottom
Back