[SOLVED] error fetching datastores - 500 after upgrade to 2.2

And the failure happened :(

That will teach me to be an optimist!

Back to nursing the backups I guess.
 
We are still having the same issue - Proxmox, could you please throw some more resources at this as there a few clients with this issue.
 
mira said:
Some of our dependencies were upgraded (e.g. tokio), so it might be related to that.
But without a reproducer in house where we can freely test and reproduce the issue, it's difficult to find the actual cause.

As my colleague pointed out, we couldn't yet reproduce this here on multiple setups here. This includes private setups by some of us.
Click to expand...
Hi Mira,

I have this error on one installation, too. I think it could be Proxmox 6 (to be upgraded in the next days) in combination with pbs2.2. I have no other installation where this error occurs (and my setup at our datacenter is very similar to the one at the customer side - except: The customer has more homogenous systems and more memory).

Since 2.2, I have very, very rarely observed any failures on our infrastructure so far.

Tobias
 
@mira I will stop all other jobs from running at the same time (sync jobs, GC cleanup) and let you know how it goes. I have jobs running all the time syncing to a qnap nas as it is going too slow.
Edit: I can confirm the scheduled backup worked now that the proxmox backup server is not running other jobs at the same time. I will test this over the next few days and will let you know the results.

note: The jobs I stopped are a datastore on a remote nas : In remotes, the backup server is pointing to itself so I can sync jobs to a nfs share mounted locally. The other job running was GC on the same store (remote NAS).
edit: scheduled backups failed again last night.
 
Last edited:
I have moved the servers to backup at different intervals to each other and I have not had a failure in 48 hours, 2 hours schedule used.
 
My backups worked last night (no other sync jobs were running at the time). Will see how things go after a few days then will let the sync jobs run (basically backing up to a remote nas)
 
Here We are with intermittent failed with cron, always working with manually execution.

A lot of error in syslog for all pvs servers:

Jul 14 14:40:56 proxmox06 systemd[1]: user@0.service: Succeeded.
Jul 14 14:40:56 proxmox06 systemd[1]: Stopped User Manager for UID 0.
Jul 14 14:40:56 proxmox06 systemd[1]: Stopping User Runtime Directory /run/user/0...
Jul 14 14:40:56 proxmox06 systemd[1]: run-user-0.mount: Succeeded.
Jul 14 14:40:56 proxmox06 systemd[1]: user-runtime-dir@0.service: Succeeded.
Jul 14 14:40:56 proxmox06 systemd[1]: Stopped User Runtime Directory /run/user/0.
Jul 14 14:40:56 proxmox06 systemd[1]: Removed slice User Slice of UID 0.
Jul 14 14:41:09 proxmox06 pvestatd[1354]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 14 14:41:09 proxmox06 pvestatd[1354]: status update time (7.350 seconds)
Jul 14 14:42:09 proxmox06 pvestatd[1354]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 14 14:42:10 proxmox06 pvestatd[1354]: status update time (7.397 seconds)
Jul 14 14:43:09 proxmox06 pvestatd[1354]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 14 14:43:09 proxmox06 pvestatd[1354]: status update time (7.430 seconds)
Jul 14 14:44:09 proxmox06 pvestatd[1354]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 14 14:44:09 proxmox06 pvestatd[1354]: status update time (7.438 seconds)
Jul 14 14:45:09 proxmox06 pvestatd[1354]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 14 14:45:09 proxmox06 pvestatd[1354]: status update time (7.358 seconds)
Jul 14 14:46:10 proxmox06 pvestatd[1354]: status update time (7.739 seconds)
Jul 14 14:47:09 proxmox06 pvestatd[1354]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 14 14:47:09 proxmox06 pvestatd[1354]: status update time (7.395 seconds)
Jul 14 14:48:09 proxmox06 pvestatd[1354]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 14 14:48:10 proxmox06 pvestatd[1354]: status update time (7.457 seconds)
Jul 14 14:49:10 proxmox06 pvestatd[1354]: status update time (7.738 seconds)
Jul 14 14:50:10 proxmox06 pvestatd[1354]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 14 14:50:10 proxmox06 pvestatd[1354]: status update time (7.375 seconds)
Jul 14 14:51:10 proxmox06 pvestatd[1354]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 14 14:51:10 proxmox06 pvestatd[1354]: status update time (7.426 seconds)
Jul 14 14:52:09 proxmox06 pvestatd[1354]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 14 14:52:09 proxmox06 pvestatd[1354]: status update time (7.415 seconds)
Jul 14 14:53:10 proxmox06 pvestatd[1354]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 14 14:53:10 proxmox06 pvestatd[1354]: status update time (7.421 seconds)
Jul 14 14:54:09 proxmox06 pvestatd[1354]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 14 14:54:09 proxmox06 pvestatd[1354]: status update time (7.389 seconds)
Jul 14 14:55:09 proxmox06 pvestatd[1354]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 14 14:55:09 proxmox06 pvestatd[1354]: status update time (7.517 seconds)
Jul 14 14:56:09 proxmox06 pvestatd[1354]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 14 14:56:09 proxmox06 pvestatd[1354]: status update time (7.425 seconds)
Jul 14 14:56:50 proxmox06 pmxcfs[1259]: [status] notice: received log
Jul 14 14:57:09 proxmox06 pvestatd[1354]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 14 14:57:09 proxmox06 pvestatd[1354]: status update time (7.377 seconds)
Jul 14 14:57:46 proxmox06 pmxcfs[1259]: [status] notice: received log
Jul 14 14:58:09 proxmox06 pvestatd[1354]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 14 14:58:10 proxmox06 pvestatd[1354]: status update time (7.413 seconds)
Jul 14 14:59:09 proxmox06 pvestatd[1354]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 14 14:59:10 proxmox06 pvestatd[1354]: status update time (7.387 seconds)
Jul 14 15:00:09 proxmox06 pvestatd[1354]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 14 15:00:09 proxmox06 pvestatd[1354]: status update time (7.392 seconds)
Jul 14 15:01:09 proxmox06 pvestatd[1354]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 14 15:01:09 proxmox06 pvestatd[1354]: status update time (7.434 seconds)
Jul 14 15:02:09 proxmox06 pvestatd[1354]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 14 15:02:09 proxmox06 pvestatd[1354]: status update time (7.386 seconds)
Jul 14 15:03:09 proxmox06 pvestatd[1354]: pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 14 15:03:09 proxmox06 pvestatd[1354]: status update time (7.449 seconds)

Best Regards,

Rodrigo
 
There's a new version (2.2.4-1) available on pbstest. It should improve the situation.

We would appreciate some feedback if you could try it.
 
Last week, I deleted the contents of my offsite NFS store on a NAS which proxmox syncs to and re-added the store in proxmox which recreates the .chunks etc . I have been syncing the backups to this location since and so far no scheduled backup errors. Strange indeed. I will let you know if anything changes.
 
mira said:
There's a new version (2.2.4-1) available on pbstest. It should improve the situation.

We would appreciate some feedback if you could try it.
Click to expand...

I am waiting for tuxis to upgrade their server; then I can report my results back. (I doubt I could reproduce the issue with my own server)
 
Hi !
I did the upgrade just now.

root@pbs02:~# proxmox-backup-manager version
proxmox-backup-server 2.2.4-1 running version: 2.2.4
root@pbs02:~#


Best Regards,

Rodrigo
 
rlljorge said:
Hi !
I did the upgrade just now.

root@pbs02:~# proxmox-backup-manager version
proxmox-backup-server 2.2.4-1 running version: 2.2.4
root@pbs02:~#


Best Regards,

Rodrigo
Click to expand...

Hi, that is great -- sadly I am on pbs003 :)

EDIT:// And I just figured you probably ment your own server and not tuxis -- sorry.
 
Last edited:
Both would be great, is possible!
 
Hello !

@mira I received the error in 2.2.4, the cron job started at 12:30

Code: 
TASK ERROR: could not activate storage 'pbs02': pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007

The syslog of pvs:
Code: 
Jul 18 12:19:10 proxmox04 pvestatd[1496]: status update time (5.006 seconds)
Jul 18 12:21:11 proxmox04 pvestatd[1496]: status update time (5.396 seconds)
Jul 18 12:22:10 proxmox04 pvestatd[1496]: status update time (5.302 seconds)
Jul 18 12:22:46 proxmox04 pmxcfs[1402]: [status] notice: received log
Jul 18 12:23:10 proxmox04 pvestatd[1496]: status update time (5.246 seconds)
Jul 18 12:24:10 proxmox04 pvestatd[1496]: status update time (5.441 seconds)
Jul 18 12:26:10 proxmox04 pvestatd[1496]: status update time (5.323 seconds)
Jul 18 12:26:21 proxmox04 pmxcfs[1402]: [dcdb] notice: data verification successful
Jul 18 12:27:10 proxmox04 pvestatd[1496]: status update time (5.119 seconds)
Jul 18 12:28:10 proxmox04 pvestatd[1496]: status update time (5.257 seconds)
Jul 18 12:29:10 proxmox04 pvestatd[1496]: status update time (5.244 seconds)
Jul 18 12:30:01 proxmox04 pvescheduler[199164]: <root@pam> starting task UPID:proxmox04:000309FD:03A6EAE2:62D57C79:vzdump:190:root@pam:
Jul 18 12:30:08 proxmox04 pvescheduler[199165]: could not activate storage 'pbs02': pbs02: error fetching datastores - 500 Can't connect to 10.250.5.84:8007
Jul 18 12:30:08 proxmox04 postfix/pickup[188399]: 99954206AF: uid=0 from=<proxmox@sianet.com.br>
Jul 18 12:30:08 proxmox04 postfix/cleanup[199217]: 99954206AF: message-id=<20220718153008.99954206AF@proxmox04.sianet.com.br>
Jul 18 12:30:08 proxmox04 postfix/qmgr[1464]: 99954206AF: from=<proxmox@sianet.com.br>, size=1589, nrcpt=1 (queue active)
Jul 18 12:30:08 proxmox04 postfix/smtp[199219]: 99954206AF: to=<suporte@sianet.com.br>, relay=mailgw01.sianet.com.br[187.103.149.74]:25, delay=0.06, delays=0.02/0.01/0.02/0.01, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as A1876A0090)
Jul 18 12:30:08 proxmox04 postfix/qmgr[1464]: 99954206AF: removed
Jul 18 12:33:10 proxmox04 pvestatd[1496]: status update time (5.094 seconds)
Jul 18 12:34:10 proxmox04 pvestatd[1496]: status update time (5.601 seconds)

If you prefer, I can upgrade to 2.2.5

Best Regards,

Rodrigo
 
You must log in or register to reply here.
Top Bottom