pvesr.service all exit codes , I cannot find status=17/n/a

[Szymons]

Hello,

Where Can I find all exit codes of pvesr.service ?
I couldn't find 17/n/a.

Linux 5.4.174-2-pve #1 SMP PVE 5.4.174-2 (Thu, 10 Mar 2022 15:58:44 +0100)
pve-manager/6.4-14/15e2bf61

pvesr.service: Main process exited, code=exited, status=17/n/a

Jul 01 06:12:01 hv-panther-01 pmxcfs[1144]: [status] notice: received log
Jul 01 06:12:06 hv-panther-01 pmxcfs[1144]: [status] notice: received log
Jul 01 06:12:55 hv-panther-01 sudo[7703]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 01 06:12:55 hv-panther-01 sudo[7703]: pam_unix(sudo:session): session closed for user root
Jul 01 06:13:00 hv-panther-01 systemd[1]: Starting Proxmox VE replication runner...
Jul 01 06:13:04 hv-panther-01 sudo[7798]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 01 06:13:04 hv-panther-01 sudo[7798]: pam_unix(sudo:session): session closed for user root
Jul 01 06:13:11 hv-panther-01 pmxcfs[1144]: [status] notice: received log
Jul 01 06:13:11 hv-panther-01 pvesr[7770]: cfs-lock 'file-replication_cfg' error: got lock request timeout
Jul 01 06:13:11 hv-panther-01 systemd[1]: pvesr.service: Main process exited, code=exited, status=17/n/a
Jul 01 06:13:11 hv-panther-01 systemd[1]: pvesr.service: Failed with result 'exit-code'.
Jul 01 06:13:11 hv-panther-01 systemd[1]: Failed to start Proxmox VE replication runner.

Jul 01 06:13:27 hv-panther-03 sudo[19017]: pam_unix(sudo:session): session closed for user root
Jul 01 06:13:30 hv-panther-03 corosync[1141]:   [KNET  ] link: host: 17 link: 0 is down
Jul 01 06:13:30 hv-panther-03 corosync[1141]:   [KNET  ] host: host: 17 (passive) best link: 0 (pri: 1)
Jul 01 06:13:30 hv-panther-03 corosync[1141]:   [KNET  ] host: host: 17 has no active links
Jul 01 06:13:35 hv-panther-03 pmxcfs[1129]: [status] notice: received log
Jul 01 06:13:36 hv-panther-03 sudo[20288]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 01 06:13:36 hv-panther-03 sudo[20288]: pam_unix(sudo:session): session closed for user root
Jul 01 06:13:40 hv-panther-03 corosync[1141]:   [KNET  ] rx: host: 17 link: 0 is up
Jul 01 06:13:40 hv-panther-03 corosync[1141]:   [KNET  ] host: host: 17 (passive) best link: 0 (pri: 1)
Jul 01 06:13:45 hv-panther-03 sudo[20352]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 01 06:13:45 hv-panther-03 sudo[20352]: pam_unix(sudo:session): session closed for user root

 

[Szymons]

anyone knows exit codes of pvesr.service and of course their meaning ?

 

[Szymons]

maybe after I added subscription key to my accont , there will be someone who can help ?

 

[Szymons]

I checked links below and it'is nothing about exit codes:
https://pve.proxmox.com/pve-docs-6/chapter-pvesr.html#_error_handling
https://pve.proxmox.com/pve-docs-6/pvesr.1.htmland there is nothig about exit codes

 

[dcsapak]

hi, instead of checking only the exit code (which is not really useful here anyway) i'd check the complete log output of the systemd unit log

journalctl -b -u pvesr

prints the log for the pvesr service since the last boot... with that, we can maybe see where the problem is

 

[Szymons]

Hello,
thanks for reply here is info :

 cfs-lock 'file-replication_cfg' error: got lock request timeout

lip 18 09:31:00 hv-panther-07 systemd[1]: Started Proxmox VE replication runner.
lip 18 09:31:00 hv-panther-07 systemd[1]: pvesr.service: Succeeded.
lip 18 09:31:00 hv-panther-07 systemd[1]: Starting Proxmox VE replication runner...
lip 18 09:30:01 hv-panther-07 systemd[1]: Started Proxmox VE replication runner.
lip 18 09:30:01 hv-panther-07 systemd[1]: pvesr.service: Succeeded.
lip 18 09:30:00 hv-panther-07 pvesr[6799]: trying to acquire cfs lock 'file-replication_cfg' ...
lip 18 09:30:00 hv-panther-07 systemd[1]: Starting Proxmox VE replication runner...
lip 18 09:29:00 hv-panther-07 systemd[1]: Started Proxmox VE replication runner.
lip 18 09:29:00 hv-panther-07 systemd[1]: pvesr.service: Succeeded.
lip 18 09:29:00 hv-panther-07 systemd[1]: Starting Proxmox VE replication runner...
lip 18 09:28:19 hv-panther-07 systemd[1]: Failed to start Proxmox VE replication runner.
lip 18 09:28:19 hv-panther-07 systemd[1]: pvesr.service: Failed with result 'exit-code'.
lip 18 09:28:19 hv-panther-07 systemd[1]: pvesr.service: Main process exited, code=exited, status=17/n/a
lip 18 09:28:19 hv-panther-07 pvesr[4751]: cfs-lock 'file-replication_cfg' error: got lock request timeout
lip 18 09:28:04 hv-panther-07 pvesr[4751]: trying to acquire cfs lock 'file-replication_cfg' ...
lip 18 09:28:00 hv-panther-07 systemd[1]: Starting Proxmox VE replication runner...
lip 18 09:27:00 hv-panther-07 systemd[1]: Started Proxmox VE replication runner.

 

[dcsapak]

do you have a cluster? is it quorate ?
is there anything else in the log that might indicate a problem?

 

[Szymons]

Yes there is cluster.
There is nothing special during this error .
I can see this error at other nodes also at different time.

 

[dcsapak]

the error says that it could not lock the replication config in time which is normally an indicator for an overloaded cluster network
or busy clusterfilesystem
can you post a longer excerpt from the log? e.g. a few minutes before and after such an error?

 

[Szymons]

before :

Jul 18 10:21:03 hostname pvesr[11132]: trying to acquire cfs lock 'file-replication_cfg' ...
Jul 18 10:21:04 hostname systemd[1]: pvesr.service: Succeeded.
Jul 18 10:21:04 hostname systemd[1]: Started Proxmox VE replication runner.
Jul 18 10:21:13 hostname sudo[11258]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 18 10:21:13 hostname sudo[11258]: pam_unix(sudo:session): session closed for user root
Jul 18 10:22:00 hostname systemd[1]: Starting Proxmox VE replication runner...
Jul 18 10:22:00 hostname systemd[1]: pvesr.service: Succeeded.
Jul 18 10:22:00 hostname systemd[1]: Started Proxmox VE replication runner.
Jul 18 10:22:01 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:22:06 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:22:31 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:23:00 hostname systemd[1]: Starting Proxmox VE replication runner...
Jul 18 10:23:00 hostname pvesr[13191]: trying to acquire cfs lock 'file-replication_cfg' ...
Jul 18 10:23:01 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:23:01 hostname corosync[17256]:   [TOTEM ] Retransmit List: 49e524
Jul 18 10:23:01 hostname corosync[17256]:   [TOTEM ] Retransmit List: 49e525
Jul 18 10:23:01 hostname systemd[1]: pvesr.service: Succeeded.
Jul 18 10:23:01 hostname systemd[1]: Started Proxmox VE replication runner.
Jul 18 10:23:07 hostname corosync[17256]:   [TOTEM ] Retransmit List: 49e5fd
Jul 18 10:23:07 hostname corosync[17256]:   [TOTEM ] Retransmit List: 49e600
Jul 18 10:23:07 hostname corosync[17256]:   [TOTEM ] Retransmit List: 49e605
Jul 18 10:23:11 hostname corosync[17256]:   [TOTEM ] Retransmit List: 49e653
Jul 18 10:23:11 hostname corosync[17256]:   [TOTEM ] Retransmit List: 49e654
Jul 18 10:23:11 hostname corosync[17256]:   [TOTEM ] Retransmit List: 49e658
Jul 18 10:23:11 hostname corosync[17256]:   [TOTEM ] Retransmit List: 49e65c
Jul 18 10:23:12 hostname corosync[17256]:   [TOTEM ] Retransmit List: 49e694
Jul 18 10:23:12 hostname corosync[17256]:   [TOTEM ] Retransmit List: 49e697
Jul 18 10:23:12 hostname corosync[17256]:   [TOTEM ] Retransmit List: 49e69c
Jul 18 10:24:00 hostname systemd[1]: Starting Proxmox VE replication runner...
Jul 18 10:24:00 hostname systemd[1]: pvesr.service: Succeeded.
Jul 18 10:24:00 hostname systemd[1]: Started Proxmox VE replication runner.
Jul 18 10:24:02 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:25:00 hostname systemd[1]: Starting Proxmox VE replication runner...
Jul 18 10:25:00 hostname systemd[1]: pvesr.service: Succeeded.
Jul 18 10:25:00 hostname systemd[1]: Started Proxmox VE replication runner.
Jul 18 10:25:01 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:26:00 hostname systemd[1]: Starting Proxmox VE replication runner...
Jul 18 10:26:00 hostname systemd[1]: pvesr.service: Succeeded.
Jul 18 10:26:00 hostname systemd[1]: Started Proxmox VE replication runner.
Jul 18 10:26:01 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:26:13 hostname sudo[15686]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 18 10:26:13 hostname sudo[15686]: pam_unix(sudo:session): session closed for user root
Jul 18 10:27:00 hostname systemd[1]: Starting Proxmox VE replication runner...
Jul 18 10:27:00 hostname systemd[1]: pvesr.service: Succeeded.
Jul 18 10:27:00 hostname systemd[1]: Started Proxmox VE replication runner.
Jul 18 10:27:01 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:27:02 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:28:00 hostname systemd[1]: Starting Proxmox VE replication runner...
Jul 18 10:28:00 hostname systemd[1]: pvesr.service: Succeeded.
Jul 18 10:28:00 hostname systemd[1]: Started Proxmox VE replication runner.
Jul 18 10:28:01 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:29:00 hostname systemd[1]: Starting Proxmox VE replication runner...
Jul 18 10:29:00 hostname systemd[1]: pvesr.service: Succeeded.
Jul 18 10:29:00 hostname systemd[1]: Started Proxmox VE replication runner.
Jul 18 10:29:01 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:29:22 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:29:53 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:30:00 hostname systemd[1]: Starting Proxmox VE replication runner...
Jul 18 10:30:00 hostname systemd[1]: pvesr.service: Succeeded.
Jul 18 10:30:00 hostname systemd[1]: Started Proxmox VE replication runner.
Jul 18 10:30:01 hostname CRON[19610]: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 18 10:30:02 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:30:07 hostname CRON[19610]: pam_unix(cron:session): session closed for user root
Jul 18 10:31:00 hostname systemd[1]: Starting Proxmox VE replication runner...
Jul 18 10:31:00 hostname pvesr[31722]: trying to acquire cfs lock 'file-replication_cfg' ...
Jul 18 10:31:01 hostname systemd[1]: pvesr.service: Succeeded.
Jul 18 10:31:01 hostname systemd[1]: Started Proxmox VE replication runner.
Jul 18 10:31:01 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:31:13 hostname sudo[31848]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 18 10:31:13 hostname sudo[31848]: pam_unix(sudo:session): session closed for user root
Jul 18 10:32:00 hostname systemd[1]: Starting Proxmox VE replication runner...
Jul 18 10:32:00 hostname systemd[1]: pvesr.service: Succeeded.
Jul 18 10:32:00 hostname systemd[1]: Started Proxmox VE replication runner.
Jul 18 10:32:01 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:32:06 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:33:00 hostname systemd[1]: Starting Proxmox VE replication runner...
Jul 18 10:33:00 hostname pvesr[1591]: trying to acquire cfs lock 'file-replication_cfg' ...
Jul 18 10:33:01 hostname systemd[1]: pvesr.service: Succeeded.
Jul 18 10:33:01 hostname systemd[1]: Started Proxmox VE replication runner.
Jul 18 10:33:02 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:34:00 hostname systemd[1]: Starting Proxmox VE replication runner...
Jul 18 10:34:00 hostname systemd[1]: pvesr.service: Succeeded.
Jul 18 10:34:00 hostname systemd[1]: Started Proxmox VE replication runner.
Jul 18 10:34:01 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:34:36 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:35:00 hostname systemd[1]: Starting Proxmox VE replication runner...
Jul 18 10:35:00 hostname systemd[1]: pvesr.service: Succeeded.
Jul 18 10:35:00 hostname systemd[1]: Started Proxmox VE replication runner.
Jul 18 10:35:01 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:35:51 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:36:00 hostname systemd[1]: Starting Proxmox VE replication runner...
Jul 18 10:36:00 hostname systemd[1]: pvesr.service: Succeeded.
Jul 18 10:36:00 hostname systemd[1]: Started Proxmox VE replication runner.
Jul 18 10:36:02 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:36:13 hostname sudo[4103]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 18 10:36:13 hostname sudo[4103]: pam_unix(sudo:session): session closed for user root
Jul 18 10:37:00 hostname systemd[1]: Starting Proxmox VE replication runner...
Jul 18 10:37:00 hostname systemd[1]: pvesr.service: Succeeded.
Jul 18 10:37:00 hostname systemd[1]: Started Proxmox VE replication runner.
Jul 18 10:37:02 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:38:00 hostname systemd[1]: Starting Proxmox VE replication runner...
Jul 18 10:38:04 hostname pvesr[5983]: trying to acquire cfs lock 'file-replication_cfg' ...
Jul 18 10:38:04 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:38:05 hostname pvesr[5983]: trying to acquire cfs lock 'file-replication_cfg' ...
Jul 18 10:38:06 hostname pvesr[5983]: trying to acquire cfs lock 'file-replication_cfg' ...
Jul 18 10:38:07 hostname pvesr[5983]: trying to acquire cfs lock 'file-replication_cfg' ...
Jul 18 10:38:08 hostname pvesr[5983]: trying to acquire cfs lock 'file-replication_cfg' ...
Jul 18 10:38:09 hostname pvesr[5983]: cfs-lock 'file-replication_cfg' error: got lock request timeout
Jul 18 10:38:09 hostname systemd[1]: pvesr.service: Main process exited, code=exited, status=17/n/a

after :

Jul 18 10:38:09 hostname systemd[1]: pvesr.service: Main process exited, code=exited, status=17/n/a
Jul 18 10:38:09 hostname systemd[1]: pvesr.service: Failed with result 'exit-code'.
Jul 18 10:38:09 hostname systemd[1]: Failed to start Proxmox VE replication runner.
Jul 18 10:39:00 hostname systemd[1]: Starting Proxmox VE replication runner...
Jul 18 10:39:00 hostname systemd[1]: pvesr.service: Succeeded.
Jul 18 10:39:00 hostname systemd[1]: Started Proxmox VE replication runner.
Jul 18 10:39:01 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:39:32 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:39:43 hostname corosync[17256]:   [TOTEM ] Retransmit List: 4a577a
Jul 18 10:40:00 hostname systemd[1]: Starting Proxmox VE replication runner...
Jul 18 10:40:00 hostname systemd[1]: pvesr.service: Succeeded.
Jul 18 10:40:00 hostname systemd[1]: Started Proxmox VE replication runner.
Jul 18 10:40:01 hostname CRON[8027]: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 18 10:40:02 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:40:02 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:40:08 hostname CRON[8027]: pam_unix(cron:session): session closed for user root
Jul 18 10:41:00 hostname systemd[1]: Starting Proxmox VE replication runner...
Jul 18 10:41:00 hostname systemd[1]: pvesr.service: Succeeded.
Jul 18 10:41:00 hostname systemd[1]: Started Proxmox VE replication runner.
Jul 18 10:41:01 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:41:13 hostname sudo[20269]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 18 10:41:13 hostname sudo[20269]: pam_unix(sudo:session): session closed for user root
Jul 18 10:42:00 hostname systemd[1]: Starting Proxmox VE replication runner...
Jul 18 10:42:00 hostname systemd[1]: pvesr.service: Succeeded.
Jul 18 10:42:00 hostname systemd[1]: Started Proxmox VE replication runner.
Jul 18 10:42:01 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:42:02 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:42:06 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:42:10 hostname sudo[20634]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 18 10:42:10 hostname dbus-daemon[899]: [system] Activating via systemd: service name='org.freedesktop.timedate1' unit='dbus-org.freedesktop.timedate1.service' requested by ':1.53351' (uid=0 pid=20724 comm="timedatectl status " label="unconfined")
Jul 18 10:42:10 hostname systemd[1]: Starting Time & Date Service...
Jul 18 10:42:10 hostname dbus-daemon[899]: [system] Successfully activated service 'org.freedesktop.timedate1'
Jul 18 10:42:10 hostname systemd[1]: Started Time & Date Service.
Jul 18 10:42:10 hostname sudo[20634]: pam_unix(sudo:session): session closed for user root
Jul 18 10:42:11 hostname sudo[20831]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 18 10:42:11 hostname sudo[20831]: pam_unix(sudo:session): session closed for user root
Jul 18 10:42:12 hostname sudo[20843]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 18 10:42:12 hostname sudo[20843]: pam_unix(sudo:session): session closed for user root
Jul 18 10:42:19 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:42:19 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:42:40 hostname systemd[1]: systemd-timedated.service: Succeeded.
Jul 18 10:43:00 hostname systemd[1]: Starting Proxmox VE replication runner...
Jul 18 10:43:00 hostname systemd[1]: pvesr.service: Succeeded.
Jul 18 10:43:00 hostname systemd[1]: Started Proxmox VE replication runner.
Jul 18 10:43:02 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:43:38 hostname corosync[17256]:   [TOTEM ] Retransmit List: 4a71ed
Jul 18 10:43:38 hostname corosync[17256]:   [TOTEM ] Retransmit List: 4a71ed 4a71ee
Jul 18 10:44:00 hostname systemd[1]: Starting Proxmox VE replication runner...
Jul 18 10:44:00 hostname systemd[1]: pvesr.service: Succeeded.
Jul 18 10:44:00 hostname systemd[1]: Started Proxmox VE replication runner.
Jul 18 10:44:01 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:44:22 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:44:54 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:45:00 hostname systemd[1]: Starting Proxmox VE replication runner...
Jul 18 10:45:00 hostname systemd[1]: pvesr.service: Succeeded.
Jul 18 10:45:00 hostname systemd[1]: Started Proxmox VE replication runner.
Jul 18 10:45:01 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:46:00 hostname systemd[1]: Starting Proxmox VE replication runner...
Jul 18 10:46:00 hostname systemd[1]: pvesr.service: Succeeded.
Jul 18 10:46:00 hostname systemd[1]: Started Proxmox VE replication runner.
Jul 18 10:46:02 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:46:13 hostname sudo[24906]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jul 18 10:46:13 hostname sudo[24906]: pam_unix(sudo:session): session closed for user root
Jul 18 10:47:00 hostname systemd[1]: Starting Proxmox VE replication runner...
Jul 18 10:47:00 hostname systemd[1]: pvesr.service: Succeeded.
Jul 18 10:47:00 hostname systemd[1]: Started Proxmox VE replication runner.
Jul 18 10:47:02 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:47:34 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:48:00 hostname systemd[1]: Starting Proxmox VE replication runner...
Jul 18 10:48:00 hostname systemd[1]: pvesr.service: Succeeded.
Jul 18 10:48:00 hostname systemd[1]: Started Proxmox VE replication runner.
Jul 18 10:48:01 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:48:02 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:49:00 hostname systemd[1]: Starting Proxmox VE replication runner...
Jul 18 10:49:00 hostname systemd[1]: pvesr.service: Succeeded.
Jul 18 10:49:00 hostname systemd[1]: Started Proxmox VE replication runner.
Jul 18 10:49:02 hostname pmxcfs[17264]: [status] notice: received log
Jul 18 10:49:43 hostname pmxcfs[17264]: [status] notice: received log

 

[dcsapak]

ok, so

Jul 18 10:23:07 hostname corosync[17256]:   [TOTEM ] Retransmit List: 49e5fd
Jul 18 10:23:07 hostname corosync[17256]:   [TOTEM ] Retransmit List: 49e600
Jul 18 10:23:07 hostname corosync[17256]:   [TOTEM ] Retransmit List: 49e605
Jul 18 10:23:11 hostname corosync[17256]:   [TOTEM ] Retransmit List: 49e653
Jul 18 10:23:11 hostname corosync[17256]:   [TOTEM ] Retransmit List: 49e654
Jul 18 10:23:11 hostname corosync[17256]:   [TOTEM ] Retransmit List: 49e658
Jul 18 10:23:11 hostname corosync[17256]:   [TOTEM ] Retransmit List: 49e65c
Jul 18 10:23:12 hostname corosync[17256]:   [TOTEM ] Retransmit List: 49e694

indicates further that your cluster network is overloaded. how fast is it, and how many cluster nodes do you have?

 

[Szymons]

20+ nodes , is there any calculator how much I need for this cluster network ?

 

[dcsapak]

no it largely depends on how much traffic it produces, things such as replication increases it. but it's not really bandwidth that's the limiting factor, more latency..
what kind of cluster network do you have?

 

[Szymons]

20G fiber in LACP agregation

 

[dcsapak]

that should be enough, but is there other traffic simultaniously on that link (e.g. vm traffic, ceph traffic, the replication traffic etc.)
if yes, i'd think about making the cluster network redundant via a seperate physical link, see e.g https://pve.proxmox.com/wiki/Cluster_Manager#pvecm_redundancy