[SOLVED] error fetching datastores - 500 after upgrade to 2.2

rlljorge · Jul 18, 2022

Hi,

In 2.24 the syslog still reporting connection errors:

Code:

Jul 18 13:39:10 proxmox11 pvestatd[1454]: status update time (7.889 seconds)
Jul 18 13:40:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 13:40:10 proxmox11 pvestatd[1454]: status update time (7.455 seconds)
Jul 18 13:41:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 13:41:09 proxmox11 pvestatd[1454]: status update time (7.597 seconds)
Jul 18 13:42:10 proxmox11 pvestatd[1454]: status update time (7.956 seconds)
Jul 18 13:43:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 13:43:09 proxmox11 pvestatd[1454]: status update time (7.359 seconds)
Jul 18 13:44:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 13:44:09 proxmox11 pvestatd[1454]: status update time (7.437 seconds)
Jul 18 13:45:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 13:45:10 proxmox11 pvestatd[1454]: status update time (7.456 seconds)
Jul 18 13:46:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 13:46:10 proxmox11 pvestatd[1454]: status update time (7.459 seconds)
Jul 18 13:47:10 proxmox11 pvestatd[1454]: status update time (7.946 seconds)
Jul 18 13:48:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 13:48:10 proxmox11 pvestatd[1454]: status update time (7.466 seconds)
Jul 18 13:49:10 proxmox11 pvestatd[1454]: status update time (7.998 seconds)
Jul 18 13:50:10 proxmox11 pvestatd[1454]: status update time (7.720 seconds)
Jul 18 13:51:10 proxmox11 pvestatd[1454]: status update time (7.839 seconds)
Jul 18 13:52:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 13:52:10 proxmox11 pvestatd[1454]: status update time (7.780 seconds)
Jul 18 13:53:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 13:53:09 proxmox11 pvestatd[1454]: status update time (7.641 seconds)
Jul 18 13:54:10 proxmox11 pvestatd[1454]: status update time (8.076 seconds)
Jul 18 13:55:10 proxmox11 pvestatd[1454]: status update time (7.699 seconds)
Jul 18 13:56:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 13:56:09 proxmox11 pvestatd[1454]: status update time (7.400 seconds)
Jul 18 13:57:10 proxmox11 pvestatd[1454]: status update time (7.704 seconds)
Jul 18 13:58:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 13:58:09 proxmox11 pvestatd[1454]: status update time (7.438 seconds)
Jul 18 13:59:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 13:59:09 proxmox11 pvestatd[1454]: status update time (7.448 seconds)
Jul 18 14:00:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 14:00:10 proxmox11 pvestatd[1454]: status update time (7.652 seconds)
Jul 18 14:01:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 14:01:10 proxmox11 pvestatd[1454]: status update time (7.528 seconds)
Jul 18 14:02:10 proxmox11 pvestatd[1454]: status update time (7.963 seconds)
Jul 18 14:03:10 proxmox11 pvestatd[1454]: status update time (7.787 seconds)
Jul 18 14:04:10 proxmox11 pvestatd[1454]: status update time (7.843 seconds)
Jul 18 14:05:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 14:05:10 proxmox11 pvestatd[1454]: status update time (7.673 seconds)
Jul 18 14:06:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 14:06:09 proxmox11 pvestatd[1454]: status update time (7.468 seconds)
Jul 18 14:07:10 proxmox11 pvestatd[1454]: status update time (7.989 seconds)
Jul 18 14:08:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 14:08:09 proxmox11 pvestatd[1454]: status update time (7.588 seconds)
Jul 18 14:09:10 proxmox11 pvestatd[1454]: status update time (7.867 seconds)
Jul 18 14:10:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 14:10:10 proxmox11 pvestatd[1454]: status update time (7.568 seconds)
Jul 18 14:11:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 14:11:09 proxmox11 pvestatd[1454]: status update time (7.573 seconds)
Jul 18 14:12:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 14:12:09 proxmox11 pvestatd[1454]: status update time (7.620 seconds)
Jul 18 14:13:10 proxmox11 pvestatd[1454]: status update time (7.865 seconds)
Jul 18 14:14:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 14:14:09 proxmox11 pvestatd[1454]: status update time (7.549 seconds)
Jul 18 14:14:54 proxmox11 pmxcfs[1347]: [status] notice: received log
Jul 18 14:15:10 proxmox11 pvestatd[1454]: status update time (7.655 seconds)
Jul 18 14:15:12 proxmox11 pmxcfs[1347]: [status] notice: received log
Jul 18 14:16:09 proxmox11 pvestatd[1454]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 14:16:09 proxmox11 pvestatd[1454]: status update time (7.464 seconds)

Best Regards,

Rodrigo

apollo13 · Jul 18, 2022

I just looked at my syslog and starting with Jul 17th I no longer see those issues on any of my boxes. No idea if @tuxis changed something at mid night or if my "jobs" magically no longer run in sync with the scheduler and as such don't hit the times where it blocks.

rlljorge · Jul 19, 2022

Hi !

The problem Still happening in 2.2.4.
I go to upgrade to 2.2.5

Regards,

Rodrigo

fde · Jul 20, 2022

Hi,
I had the same problem, I upgrade to 2.2-3. => Schedule backup failed with "error fetching datastores - 500 Can't connect to " at each schedule launch.
I tried to reboot the PBS, same error
I've just move from Production repos to no-subscription, install 2.2-5 and reboot.

The problem seems to be solved.

DJI · Jul 20, 2022

upgraded to 2.2.5 and got a kernel panic on boot after not finding root to mount, rebooted previous and all good.

tuxis · Jul 20, 2022

I just upgraded pbs003 to 2.2.5-1 (from no-sub) and rebooted the box. @apollo13, are you on that one?

DJI · Jul 20, 2022

tuxis said:
I just upgraded pbs003 to 2.2.5-1 (from no-sub) and rebooted the box. @apollo13, are you on that one?

I was on 2.2.5, I will try 2.2.5-1 today if I get a minute

apollo13 · Jul 20, 2022

tuxis said:
I just upgraded pbs003 to 2.2.5-1 (from no-sub) and rebooted the box. @apollo13, are you on that one?

Yes, let's see. I had an error again yesterday. Maybe it is quiet now

rlljorge · Jul 20, 2022

Hello!

I upgraded to 2.2.5-1 yesterday, syslog are quiet and not error at this time.

Best Regards,

Rodrigo

rlljorge · Jul 21, 2022

Hello everyone !

We are two days without error using the 2.2.5-1 version.

karpuzvepeynir · Jul 21, 2022

Hi,
After upgraded two pbs to 2.2.5-1 yesterday, none of jobs were failed. Thanks to all.

apollo13 · Jul 25, 2022

Since Tuxis upgraded the errors are gone! I have one new error:

Code:

proxmox-backup-client failed: Error: error trying to connect: error connecting to https://pbs003.tuxis.nl:8007/ - tcp connect error: deadline has elapsed

but it only showed up once. Great work so far, thanks!

pmbaeum · Jul 27, 2022

Same for me - no more errors. Thanks @devs

RolandK · Sep 17, 2022

yes, the update also fixed it for me

enoch85 · Nov 27, 2022

Ok, sorry to bring this up again, but you are describing my exact issue. I'm on 2.2.7 and long story short:

1. After fresh start I can reach both WEB UI and the datastore just fine.
2. The schedule starts which backups my "light" VMs (normally just copies like a few 100 MB per server, rest is incremental), and then continues with the heavier ones, several GB of data.
3. 15-30 minutes pass give or take.
4. 500s are starting, but I can still reach the WEB UI.
5. 5 minutes later (usually) bot SSH and WEB UI dies as well
6. Console works fine, and I can see that it (right now) runs verification and garbage collection.
7. A reboot always solves it, now go back to #1.

I come from Veeam where this never was an issue! Thought I'd give PBS a try, but now I'm hesitating. So far it's backed up around 10 TB, but now recently it started to freak out.

System:

PVE: 7.3-3 (40 cores 256 GB RAM etc)
PBS: 2.2.7 (Running as a VM on PVE above)
Datastore: QNAP 8 bay, over NFS mounted in `fstab` with version 3.0 and strict permissions.
Everything is 10Gbe

If you switch out PVE against VMware, and PBS against Veeam - the above setup has worked for more than 7 years. Since the change, nope.

enoch85 · Nov 27, 2022

And here are the syslog from rebooting, to running a backup that fails.

https://pastebin.com/Y9uuuiyR

pmbaeum · Nov 28, 2022

Not the same problem as your connection is successful.

Probably an error with your mountpoint.
https://forum.proxmox.com/threads/pve-cant-backup-to-pbs-with-datastore-on-smb-share.78947/

Search

Search

[SOLVED] error fetching datastores - 500 after upgrade to 2.2

rlljorge

Active Member

apollo13

Well-Known Member

rlljorge

Active Member

fde

Member

DJI

Member

tuxis

Famous Member

DJI

Member

apollo13

Well-Known Member

rlljorge

Active Member

rlljorge

Active Member

karpuzvepeynir

Member

apollo13

Well-Known Member

pmbaeum

Member

RolandK

Renowned Member

enoch85

Member

enoch85

Member

pmbaeum

Member