pct restore from pipe fails

[Andi]

Heyho!

I've setup up two no hosts in a new datacenter and tried to move LXCs from the old hosts via vzdump & ssh-pipe. This failed several times and I could break the problem down to `pct restore` from pipe:

root@host:~# vzdump 99999 --stdout | pct restore --storage ssd --rootfs 1 99991 -
ERROR: file '-' does not exist
trying to acquire lock...
 OK
INFO: starting new backup job: vzdump 99999
INFO: Starting Backup of VM 99999 (lxc)
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: CT Name: test
INFO: sending archive to stdout
ERROR: Backup of VM 99999 failed - command 'set -o pipefail && tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/var/lib/vz/dump/vzdump-lxc-99999-2019_02_21-22_34_57.tmp' ./etc/vzdump/pct.conf '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./' failed: got signal 13
INFO: Backup job finished with errors
job errors

I assume the issue is already in line 2: ERROR: file '-' does not exist (which probably comes from `pct restore`).

vzdump seems to work, because if I do `vzdump 99999 --stdout | cat -` I get all the contents of the LXC on my terminal.

I'm happy about any idea to this issue

My pveversion (should all be the newest stable):

root@host:~# pveversion --verbose
proxmox-ve: 5.3-1 (running kernel: 4.15.18-10-pve)
pve-manager: 5.3-8 (running version: 5.3-8/2929af8e)
pve-kernel-4.15: 5.3-1
pve-kernel-4.15.18-10-pve: 4.15.18-32
pve-kernel-4.15.17-1-pve: 4.15.17-9
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-43
libpve-guest-common-perl: 2.0-19
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-36
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-2
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
proxmox-widget-toolkit: 1.0-22
pve-cluster: 5.0-33
pve-container: 2.0-33
pve-docs: 5.3-1
pve-edk2-firmware: 1.20181023-1
pve-firewall: 3.0-17
pve-firmware: 2.0-6
pve-ha-manager: 2.0-6
pve-i18n: 1.0-9
pve-libspice-server1: 0.14.1-1
pve-qemu-kvm: 2.12.1-1
pve-xtermjs: 3.10.1-1
qemu-server: 5.0-45
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.12-pve1~bpo1

 

[mbosma]

Hi, your variables seem correct though the orde might matter.
Here's the command I used for container migration:
ssh 192.168.100.231 'vzdump 115 --mode stop --stdout' | pv | pct restore --rootfs 5 115 --storage local-zfs -

Hope it helps.

 

[Andi]

Hi mbosma,

thanks for your reply.

I tried it again with you command but again the same error:

root@host:~# ssh old_host 'vzdump 7701 --mode stop --stdout' | pv | pct restore --rootfs 4 7701 --storage ssd -
unable to restore CT 7701 - ERROR: file '-' does not exist
Use of uninitialized value $et in string eq at /usr/share/perl5/PVE/AccessControl.pm line 638.
Use of uninitialized value $et in string eq at /usr/share/perl5/PVE/AccessControl.pm line 677.
Use of uninitialized value $et in string eq at /usr/share/perl5/PVE/AccessControl.pm line 705.
Use of uninitialized value $et in string eq at /usr/share/perl5/PVE/AccessControl.pm line 724.
Use of uninitialized value $et in string eq at /usr/share/perl5/PVE/AccessControl.pm line 757.
user config - ignore config line: :
INFO: starting new backup job: vzdump 7701 --mode stop
INFO: Starting Backup of VM 7701 (lxc)
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: CT Name: ffdnproxy
INFO: sending archive to stdout
   0 B 0:00:00 [   0 B/s] [<=>                                                                                        ]
ERROR: Backup of VM 7701 failed - command 'set -o pipefail && tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/var/lib/vz/dump/vzdump-lxc-7701-2019_02_22-10_11_36.tmp' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./' failed: got signal 13
INFO: Backup job finished with errors
job errors

 

[Andi]

At least I found the source of the `Use of uninitialized value $et in string eq`-isse: in `/etc/pve/user.cfg` there was one line just with a ":" and nothing else. I removed that line and these five error messages are gone, but the pct restore error still exists...

 

[oguz]

I've setup up two no hosts in a new datacenter and tried to move LXCs from the old hosts via vzdump & ssh-pipe. This failed several times and I could break the problem down to `pct restore` from pipe:

Why don't you use scp to transfer the dumps and then use pct restore on the new host instead? Is there a space problem?

 

[Andi]

Why don't you use scp to transfer the dumps and then use pct restore on the new host instead? Is there a space problem?

Yeah there are actually two reasons:
1. The space problem: All disks in the server (except the one with the Proxmox OS) are ZFS formatted, so they can't handle dump files and the backups are done to an external NFS storage which is noch accessible from the new host (done by a different company, it's complicated...)
2. Some of the containers (we're taking about 50-100) are quite critical when it's about uptime and I thought vzdump & restore over SSH pipe (maybe compressed) is the fastest way to get it back online on the new host without having different versions of the filesystems).

 

[mbosma]

It seems to be tied to the latest update.
Even piping it on the host locally spits out an error.
Another host wich is on 5.3-5 works fine, but my prod host on 5.3-8 gave me the same error you have.

 

[Andi]

Mbosma, thanks for trying! My next step would have been a reinstall of the host, so I can skip that.
@ Proxmox-Staff: can you confirm this is a bug?

 

[mbosma]

No problem, I use this command from somewhat regularly so getting this issue fixed also benefits me (and others).

 

[Andi]

Okay, I can confirm:
Downgrading pve-container to 2.0-31 helps (2.0-33 has the same issue and 2.0-32 was somehow unavailable on my system).
This seems to work without any dependency problems.

Should I create an issue on bugzilla.proxmox.com or is this done by anyone else?

 

[oguz]

Okay, I can confirm:
Downgrading pve-container to 2.0-31 helps (2.0-33 has the same issue and 2.0-32 was somehow unavailable on my system).
This seems to work without any dependency problems.

Should I create an issue on bugzilla.proxmox.com or is this done by anyone else?

Please create a report on bugzilla. Try to include as much as info possible (commands you used, errors you got, if the errors were different when using different versions of pve-container etc.)

 

[mbosma]

Hi Andi,

Have you created a report on bugzilla? I forgot to do it myself.

Edit:
I couldn't find a bugreport so I made one:
https://bugzilla.proxmox.com/show_bug.cgi?id=2117