[SOLVED] error fetching datastores - 500 after upgrade to 2.2

Hi,

In 2.24 the syslog still reporting connection errors:


Code:
Jul 18 13:39:10 proxmox11 pvestatd[1454]: status update time (7.889 seconds)
Jul 18 13:40:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 13:40:10 proxmox11 pvestatd[1454]: status update time (7.455 seconds)
Jul 18 13:41:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 13:41:09 proxmox11 pvestatd[1454]: status update time (7.597 seconds)
Jul 18 13:42:10 proxmox11 pvestatd[1454]: status update time (7.956 seconds)
Jul 18 13:43:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 13:43:09 proxmox11 pvestatd[1454]: status update time (7.359 seconds)
Jul 18 13:44:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 13:44:09 proxmox11 pvestatd[1454]: status update time (7.437 seconds)
Jul 18 13:45:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 13:45:10 proxmox11 pvestatd[1454]: status update time (7.456 seconds)
Jul 18 13:46:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 13:46:10 proxmox11 pvestatd[1454]: status update time (7.459 seconds)
Jul 18 13:47:10 proxmox11 pvestatd[1454]: status update time (7.946 seconds)
Jul 18 13:48:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 13:48:10 proxmox11 pvestatd[1454]: status update time (7.466 seconds)
Jul 18 13:49:10 proxmox11 pvestatd[1454]: status update time (7.998 seconds)
Jul 18 13:50:10 proxmox11 pvestatd[1454]: status update time (7.720 seconds)
Jul 18 13:51:10 proxmox11 pvestatd[1454]: status update time (7.839 seconds)
Jul 18 13:52:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 13:52:10 proxmox11 pvestatd[1454]: status update time (7.780 seconds)
Jul 18 13:53:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 13:53:09 proxmox11 pvestatd[1454]: status update time (7.641 seconds)
Jul 18 13:54:10 proxmox11 pvestatd[1454]: status update time (8.076 seconds)
Jul 18 13:55:10 proxmox11 pvestatd[1454]: status update time (7.699 seconds)
Jul 18 13:56:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 13:56:09 proxmox11 pvestatd[1454]: status update time (7.400 seconds)
Jul 18 13:57:10 proxmox11 pvestatd[1454]: status update time (7.704 seconds)
Jul 18 13:58:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 13:58:09 proxmox11 pvestatd[1454]: status update time (7.438 seconds)
Jul 18 13:59:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 13:59:09 proxmox11 pvestatd[1454]: status update time (7.448 seconds)
Jul 18 14:00:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 14:00:10 proxmox11 pvestatd[1454]: status update time (7.652 seconds)
Jul 18 14:01:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 14:01:10 proxmox11 pvestatd[1454]: status update time (7.528 seconds)
Jul 18 14:02:10 proxmox11 pvestatd[1454]: status update time (7.963 seconds)
Jul 18 14:03:10 proxmox11 pvestatd[1454]: status update time (7.787 seconds)
Jul 18 14:04:10 proxmox11 pvestatd[1454]: status update time (7.843 seconds)
Jul 18 14:05:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 14:05:10 proxmox11 pvestatd[1454]: status update time (7.673 seconds)
Jul 18 14:06:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 14:06:09 proxmox11 pvestatd[1454]: status update time (7.468 seconds)
Jul 18 14:07:10 proxmox11 pvestatd[1454]: status update time (7.989 seconds)
Jul 18 14:08:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 14:08:09 proxmox11 pvestatd[1454]: status update time (7.588 seconds)
Jul 18 14:09:10 proxmox11 pvestatd[1454]: status update time (7.867 seconds)
Jul 18 14:10:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 14:10:10 proxmox11 pvestatd[1454]: status update time (7.568 seconds)
Jul 18 14:11:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 14:11:09 proxmox11 pvestatd[1454]: status update time (7.573 seconds)
Jul 18 14:12:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 14:12:09 proxmox11 pvestatd[1454]: status update time (7.620 seconds)
Jul 18 14:13:10 proxmox11 pvestatd[1454]: status update time (7.865 seconds)
Jul 18 14:14:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 14:14:09 proxmox11 pvestatd[1454]: status update time (7.549 seconds)
Jul 18 14:14:54 proxmox11 pmxcfs[1347]: [status] notice: received log
Jul 18 14:15:10 proxmox11 pvestatd[1454]: status update time (7.655 seconds)
Jul 18 14:15:12 proxmox11 pmxcfs[1347]: [status] notice: received log
Jul 18 14:16:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 14:16:09 proxmox11 pvestatd[1454]: status update time (7.464 seconds)

Best Regards,

Rodrigo
 
I just looked at my syslog and starting with Jul 17th I no longer see those issues on any of my boxes. No idea if @tuxis changed something at mid night or if my "jobs" magically no longer run in sync with the scheduler and as such don't hit the times where it blocks.
 
Hi,
I had the same problem, I upgrade to 2.2-3. => Schedule backup failed with "error fetching datastores - 500 Can't connect to " at each schedule launch.
I tried to reboot the PBS, same error
I've just move from Production repos to no-subscription, install 2.2-5 and reboot.

The problem seems to be solved.
 
upgraded to 2.2.5 and got a kernel panic on boot after not finding root to mount, rebooted previous and all good.
 
Hello!

I upgraded to 2.2.5-1 yesterday, syslog are quiet and not error at this time.

Best Regards,

Rodrigo
 
  • Like
Reactions: mira
Since Tuxis upgraded the errors are gone! I have one new error:
Code:
proxmox-backup-client failed: Error: error trying to connect: error connecting to https://pbs003.tuxis.nl:8007/ - tcp connect error: deadline has elapsed
but it only showed up once. Great work so far, thanks!
 
Ok, sorry to bring this up again, but you are describing my exact issue. I'm on 2.2.7 and long story short:

1. After fresh start I can reach both WEB UI and the datastore just fine.
2. The schedule starts which backups my "light" VMs (normally just copies like a few 100 MB per server, rest is incremental), and then continues with the heavier ones, several GB of data.
3. 15-30 minutes pass give or take.
4. 500s are starting, but I can still reach the WEB UI.
5. 5 minutes later (usually) bot SSH and WEB UI dies as well
6. Console works fine, and I can see that it (right now) runs verification and garbage collection.
7. A reboot always solves it, now go back to #1.

I come from Veeam where this never was an issue! Thought I'd give PBS a try, but now I'm hesitating. So far it's backed up around 10 TB, but now recently it started to freak out.

System:
PVE: 7.3-3 (40 cores 256 GB RAM etc)
PBS: 2.2.7 (Running as a VM on PVE above)
Datastore: QNAP 8 bay, over NFS mounted in `fstab` with version 3.0 and strict permissions.
Everything is 10Gbe
If you switch out PVE against VMware, and PBS against Veeam - the above setup has worked for more than 7 years. Since the change, nope.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!