pve-replication-state.json too long - aborting

[Jota V.]

Hi, we have a three nodes cluster with ZFS storage.

We are using replication for our LXC servers. On node 1, we have about 67 LXC containers, using replication to another server.

We have scheduled another replication and we got this errors on webui.

file '/var/lib/pve-manager/pve-replication-state.json' too long - aborting (500)

What can we do?

Our environment.

proxmox-ve: 5.4-1 (running kernel: 4.15.18-12-pve)
pve-manager: 5.4-3 (running version: 5.4-3/0a6eaa62)
pve-kernel-4.15: 5.3-3
pve-kernel-4.15.18-12-pve: 4.15.18-35
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-8
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-50
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-13
libpve-storage-perl: 5.0-41
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-3
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-3
proxmox-widget-toolkit: 1.0-25
pve-cluster: 5.0-36
pve-container: 2.0-37
pve-docs: 5.4-2
pve-edk2-firmware: 1.20190312-1
pve-firewall: 3.0-19
pve-firmware: 2.0-6
pve-ha-manager: 2.0-9
pve-i18n: 1.1-4
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 2.12.1-3
pve-xtermjs: 3.12.0-1
qemu-server: 5.0-50
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.13-pve1~bpo2

 

[wolfgang]

Hi,

how large is your /var/lib/pve-manager/pve-replication-state.json?

ls -al /var/lib/pve-manager/pve-replication-state.json

 

[Jota V.]

-rw-r--r-- 1 root root 32850 Apr 16 12:27 /var/lib/pve-manager/pve-replication-state.json

 

[wolfgang]

Do you get an error in your replication?
The problem is Proxmox VE only allows to read files with a max of 32768.
I have sent a patch what will increase the read size.

 

[Jota V.]

Can't see replication stats :-(

Attached webui screenshot and

root@pve:~# pvesr status
file '/var/lib/pve-manager/pve-replication-state.json' too long - aborting

 

[Jota V.]

Do you get an error in your replication?
The problem is Proxmox VE only allows to read files with a max of 32768.
I have sent a patch what will increase the read size.

Can I patch my environment manually?

 

[wolfgang]

Yes,

edit /usr/share/perl5/PVE/ReplicationState.pm

Line 31 replace
- my $raw = PVE::Tools::file_get_contents($state_path);
+ my $raw = PVE::Tools::file_get_contents($state_path, 1048476);

 

[Jota V.]

Late . Yesterday, deleted json and lock file, deleted all replication jobs and scheduled again.

Apparently, everything is working properly. I'll apply the patches manually just in case.

Thank you very much.