After replication Schedule Now GUI is not responding

[Dave Wood]

Fresh install 4 nodes cluster was running fine.
I tested a LCX VM replication and it worked fine.

As soon as I created a replication and clicked on Schedule Now GUI is not responding.

Hosts --> Replication not loading anything getting Connection Timed Out (596)

I'm not sure what to do. Find below few outputs.

pvesr status
JobID Enabled Target LastSync NextSync Duration FailCount State
100-0 Yes local/pve51 2017-11-04_02:52:01 2017-11-05_01:00:00 47.670547 0 OK
104-0 Yes local/pve54 2017-11-04_20:19:01 pending 139.315594 0 OK

pvesr list
JobID Target Schedule Rate Enabled
100-0 local/pve51 sun 01:00 50 yes
104-0 local/pve54 sun 04:00 50 yes

root@pve50:~# zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
nvme-zfs 464G 2.13G 462G - 0% 0% 1.00x ONLINE -
rpool 3.81T 4.76G 3.81T - 0% 0% 1.00x ONLINE -

root@pve54:~# zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
rpool 8.12T 30.7G 8.09T - 0% 0% 1.00x ONLINE -

pveversion -v
proxmox-ve: 5.1-25 (running kernel: 4.13.4-1-pve)
pve-manager: 5.1-35 (running version: 5.1-35/722cc488)
pve-kernel-4.13.4-1-pve: 4.13.4-25
libpve-http-server-perl: 2.0-6
lvm2: 2.02.168-pve6
corosync: 2.4.2-pve3
libqb0: 1.0.1-1
pve-cluster: 5.0-15
qemu-server: 5.0-17
pve-firmware: 2.0-3
libpve-common-perl: 5.0-20
libpve-guest-common-perl: 2.0-13
libpve-access-control: 5.0-7
libpve-storage-perl: 5.0-16
pve-libspice-server1: 0.12.8-3
vncterm: 1.5-2
pve-docs: 5.1-12
pve-qemu-kvm: 2.9.1-2
pve-container: 2.0-17
pve-firewall: 3.0-3
pve-ha-manager: 2.0-3
ksm-control-daemon: 1.2-2
glusterfs-client: 3.8.8-1
lxc-pve: 2.1.0-2
lxcfs: 2.0.7-pve4
criu: 2.11.1-1~bpo90
novnc-pve: 0.6-4
smartmontools: 6.5+svn4324-1
zfsutils-linux: 0.7.2-pve1~bpo90

 

[carlo497]

After update from 5.0 i've got exactly same problem. I also have high CPU usage by pvedaemon


ps -aux | grep "pvedaemon"
root 11717 0.0 0.1 527132 95072 ? Ss Nov04 0:00 pvedaemon
root 11718 95.1 0.1 537104 101592 ? R Nov04 1047:59 pvedaemon worker
root 11719 95.3 0.1 537104 101596 ? R Nov04 1049:54 pvedaemon worker
root 11720 95.4 0.1 537416 102004 ? R Nov04 1051:30 pvedaemon worker

 

[Dave Wood]

Guys,

I have been running 6 nodes Proxmox 4x cluster with PVE CEPH cluster in a production environment for a log time without any issues.

Proxmox 5.0 or 5.1 is no ready for production yet.
Most attractive feature Storage replication not working as described.

Proxmox Storage replication why not ready for production:

• If there is an issue with replication, only replication should fail, but pvesr make whole node unresponsive and all VMs in this node unmanageable. You need to reboot node to fix his issue. Which is not possible in production environment.
• I found an work around. At the time of add replication if you select sun 01:00 and try to Schedule Now node will hang but if you select default 15 Min schedule or */30 and don't click on Schedule Now replication will complete
• Live migration (KVM) is not working. I tried with CLI still not working. Only work if you shutdown VM.

I rebuild this cluster with fresh install nodes 3 times.

Just applied all patches:

root@pve50:~# pveversion -v
proxmox-ve: 5.1-25 (running kernel: 4.13.4-1-pve)
pve-manager: 5.1-36 (running version: 5.1-36/131401db)
pve-kernel-4.13.4-1-pve: 4.13.4-25
libpve-http-server-perl: 2.0-6
lvm2: 2.02.168-pve6
corosync: 2.4.2-pve3
libqb0: 1.0.1-1
pve-cluster: 5.0-15
qemu-server: 5.0-17
pve-firmware: 2.0-3
libpve-common-perl: 5.0-20
libpve-guest-common-perl: 2.0-13
libpve-access-control: 5.0-7
libpve-storage-perl: 5.0-16
pve-libspice-server1: 0.12.8-3
vncterm: 1.5-2
pve-docs: 5.1-12
pve-qemu-kvm: 2.9.1-2
pve-container: 2.0-17
pve-firewall: 3.0-3
pve-ha-manager: 2.0-3
ksm-control-daemon: 1.2-2
glusterfs-client: 3.8.8-1
lxc-pve: 2.1.0-2
lxcfs: 2.0.7-pve4
criu: 2.11.1-1~bpo90
novnc-pve: 0.6-4
smartmontools: 6.5+svn4324-1
zfsutils-linux: 0.7.2-pve1~bpo90

 

[tom]

both works here and on almost all other places.

I mean zfs replication and qemu live migration. If it does not work for you, dig deeper.