[SOLVED] "unexpected" Replication error: malformed JSON string

[marte]

Hi,

I've two nodes, pve1 and pve2, which both have a (single disk) ZFS Volume "zfs1". I've one VM ("VM102") on pve1 which should be replicated to pve2. zfs on both nodes is empty besides VM102 on pve1:

root@pve1:~# zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
zfs1                10.3G   259G    96K  /zfs1
zfs1/vm-102-disk-0  10.3G   268G  1.01G  -

root@pve2:~# zfs list
NAME   USED  AVAIL  REFER  MOUNTPOINT
zfs1   516K   269G    96K  /zfs1

The storage is visible in the WebGUI on both nodes

root@pve1:~# cat /etc/pve/storage.cfg
[...]
zfspool: zfs1
        pool zfs1
        content images,rootdir
        nodes pve2,pve1
        sparse 0

Cluster seems to be ok:

root@pve2:~# pvecm status
Quorum information
------------------
Date:             Fri Jun 28 13:09:36 2019
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000002
Ring ID:          1/7504
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   2
Highest expected: 2
Total votes:      2
Quorum:           2
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 x.x.x.161
0x00000002          1 x.x.x.162 (local)

I created a replication entry in the webgui:

root@pve1:~# cat /etc/pve/replication.cfg
local: 102-0
        comment dhjkhjks
        target pve2
        rate 50
        source pve1

But, when cron hits in, I get the following error:

Jun 28 13:00:02 pve1 pvesr[14593]: 102-0: got unexpected replication job error - command '/usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=pve2' root@10.100.100.162 -- pvesr prepare-local-job 102-0 zfs1:vm-102-disk-0 --last_sync 0' failed: malformed JSON string, neither tag, array, object, number, string or atom, at character offset 6 (before "VMID NAME           ...") at /usr/share/perl5/PVE/Replication.pm line 96.

When I exactly copy this command and use it via command line, I get:

root@pve1:~# /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=pve2' root@10.100.100.162 -- pvesr prepare-local-job 102-0 zfs1:vm-102-disk-0 --last_sync 0

[...]

{}

Can't find any sollution to this. Any help would be really appreciated!

My versions:

corosync/stable,now 2.4.4-pve1 amd64 [installed]
dmeventd/stable,now 2:1.02.137-pve6 amd64 [installed]
dmsetup/stable,now 2:1.02.137-pve6 amd64 [installed]
grub-common/stable,now 2.02-pve6 amd64 [installed]
grub-efi-amd64-bin/stable,now 2.02-pve6 amd64 [installed]
grub-efi-ia32-bin/stable,now 2.02-pve6 amd64 [installed]
grub-pc/stable,now 2.02-pve6 amd64 [installed]
grub-pc-bin/stable,now 2.02-pve6 amd64 [installed]
grub2-common/stable,now 2.02-pve6 amd64 [installed]
libcfg6/stable,now 2.4.4-pve1 amd64 [installed]
libcmap4/stable,now 2.4.4-pve1 amd64 [installed]
libcorosync-common4/stable,now 2.4.4-pve1 amd64 [installed]
libcpg4/stable,now 2.4.4-pve1 amd64 [installed]
libdevmapper-event1.02.1/stable,now 2:1.02.137-pve6 amd64 [installed]
libdevmapper1.02.1/stable,now 2:1.02.137-pve6 amd64 [installed]
liblvm2app2.2/stable,now 2.02.168-pve6 amd64 [installed]
liblvm2cmd2.02/stable,now 2.02.168-pve6 amd64 [installed]
libnvpair1linux/stable,now 0.7.13-pve1~bpo2 amd64 [installed]
libpve-access-control/stable,now 5.1-10 amd64 [installed]
libpve-apiclient-perl/stable,now 2.0-5 all [installed]
libpve-common-perl/stable,now 5.0-52 all [installed]
libpve-guest-common-perl/stable,now 2.0-20 all [installed]
libpve-http-server-perl/stable,now 2.0-13 all [installed]
libpve-storage-perl/stable,now 5.0-43 all [installed]
libpve-u2f-server-perl/stable,now 1.0-2 amd64 [installed]
libquorum5/stable,now 2.4.4-pve1 amd64 [installed]
libtotem-pg5/stable,now 2.4.4-pve1 amd64 [installed]
libuutil1linux/stable,now 0.7.13-pve1~bpo2 amd64 [installed]
libvotequorum8/stable,now 2.4.4-pve1 amd64 [installed]
libzfs2linux/stable,now 0.7.13-pve1~bpo2 amd64 [installed]
libzpool2linux/stable,now 0.7.13-pve1~bpo2 amd64 [installed]
lvm2/stable,now 2.02.168-pve6 amd64 [installed]
lxc-pve/stable,now 3.1.0-3 amd64 [installed]
lxcfs/stable,now 3.0.3-pve1 amd64 [installed]
novnc-pve/stable,now 1.0.0-3 amd64 [installed]
pve-cluster/stable,now 5.0-37 amd64 [installed]
pve-container/stable,now 2.0-39 all [installed]
pve-docs/stable,now 5.4-2 all [installed]
pve-edk2-firmware/stable,now 1.20190312-1 all [installed]
pve-firewall/stable,now 3.0-22 amd64 [installed]
pve-firmware/stable,now 2.0-6 all [installed]
pve-ha-manager/stable,now 2.0-9 amd64 [installed]
pve-i18n/stable,now 1.1-4 all [installed]
pve-kernel-4.15/stable,now 5.4-4 all [installed]
pve-kernel-4.15.18-12-pve/stable,now 4.15.18-36 amd64 [installed]
pve-kernel-4.15.18-15-pve/stable,now 4.15.18-40 amd64 [installed,automatic]
pve-kernel-4.15.18-16-pve/stable,now 4.15.18-41 amd64 [installed,automatic]
pve-libspice-server1/stable,now 0.14.1-2 amd64 [installed]
pve-manager/stable,now 5.4-7 amd64 [installed]
pve-qemu-kvm/stable,now 3.0.1-4 amd64 [installed]
pve-xtermjs/stable,now 3.12.0-1 amd64 [installed]
spl/stable,now 0.7.13-pve1~bpo2 amd64 [installed]
tar/stable,now 1.29b-2+pve.1 amd64 [installed]
zfs-initramfs/stable,now 0.7.13-pve1~bpo2 all [installed]
zfsutils-linux/stable,now 0.7.13-pve1~bpo2 amd64 [installed]

 

[marte]

SOLVED!

Just don't have any scripts with output in .bashrc of root.

Wow.

edit: typo.

 

[nocin]

Thanks for posting your solution! Was searching for hours.
Had neofetch added in .bashrc...

 

[shabbirh]

SOLVED!

Just don't have any scripts with output in .bashrc of root.

Wow.

edit: typo.

Thanks - I also had this problem - and your post helped resolve it.