Proxmox 6.0-11 Can't backup CTs, VM backup works fine

[StanTastic]

My setup:
- ZFS pool on rust (spinning harddrive) named rpool (also root filesystem)
- ZFS pool on SSD named rpool-ssd
- NFS share on Synology
- I'm using ayufan's patches for diff backup

I have setup my VMs on rpool-ssd, the CTs were initially on rpool, but I migrated them.

CTs are working fine, VMs are working fine.

However:
- I can backup VMs just fine
- none of CTs can be backed up - and the error message doesn't say much:

2019-11-27 23:39:53 INFO: Starting Backup of VM 150 (lxc)
2019-11-27 23:39:53 INFO: status = stopped
2019-11-27 23:39:53 INFO: backup mode: stop
2019-11-27 23:39:53 INFO: ionice priority: 7
2019-11-27 23:39:53 INFO: CT Name: sql-ct-1804
2019-11-27 23:39:53 INFO: creating archive '/mnt/pve/synology-nfs/dump/vzdump-lxc-150-2019_11_27-23_39_53.tar.gz'
2019-11-27 23:39:53 INFO: tar: /mnt/pve/synology-nfs/dump/vzdump-lxc-150-2019_11_27-23_39_53.tmp: Cannot open: Permission denied
2019-11-27 23:39:53 INFO: tar: Error is not recoverable: exiting now
2019-11-27 23:39:53 ERROR: Backup of VM 150 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/pve/synology-nfs/dump/vzdump-lxc-150-2019_11_27-23_39_53.tmp' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | gzip --rsyncable >/mnt/pve/synology-nfs/dump/vzdump-lxc-150-2019_11_27-23_39_53.tar.dat' failed: exit code 2

Well, technically tar is saying it can't create a file on NFS share, but for VM it's not a problem:

2019-11-27 23:00:02 INFO: Starting Backup of VM 1001 (qemu)
2019-11-27 23:00:02 INFO: status = stopped
2019-11-27 23:00:02 INFO: update VM 1001: -lock backup
2019-11-27 23:00:02 INFO: backup mode: stop
2019-11-27 23:00:02 INFO: ionice priority: 7
2019-11-27 23:00:02 INFO: VM Name: Win10ProActive
2019-11-27 23:00:02 INFO: include disk 'virtio0' 'zfs-ssd:base-1001-disk-0' 32G
2019-11-27 23:00:02 INFO: creating archive '/mnt/pve/synology-nfs/dump/vzdump-qemu-1001-2019_11_21-23_00_02.vma.gz--differential-2019_11_27-23_00_02.vcdiff'
2019-11-27 23:00:02 INFO: starting template backup
2019-11-27 23:00:02 INFO: /usr/bin/vma create -v -c /mnt/pve/synology-nfs/dump/vzdump-qemu-1001-2019_11_21-23_00_02.vma.gz--differential-2019_11_27-23_00_02.tmp/qemu-server.conf exec:gzip --rsyncable > /mnt/pve/synology-nfs/dump/vzdump-qemu-1001-2019_11_21-23_00_02.vma.gz--differential-2019_11_27-23_00_02.dat drive-virtio0=/dev/zvol/rpool-ssd/base-1001-disk-0
2019-11-27 23:00:03 INFO: progress 0% 0/34359738368 0
2019-11-27 23:00:13 INFO: progress 1% 343605248/34359738368 140926976
2019-11-27 23:00:27 INFO: progress 2% 687210496/34359738368 168079360
2019-11-27 23:00:49 INFO: progress 3% 1030815744/34359738368 175579136
<cut>
2019-11-27 23:11:10 INFO: progress 100% 34359738368/34359738368 22307561472
2019-11-27 23:11:10 INFO: image drive-virtio0: size=34359738368 zeros=22307561472 saved=12052176896
2019-11-27 23:11:11 INFO: archive file size: 5.69GB
2019-11-27 23:11:12 INFO: Finished Backup of VM 1001 (00:11:10)

Because of this I don't think the problem is with NFS, and rather with CT.

I tried to do fsck on an unmouted and stopped CT, here's the result:

# pct fsck 150
unable to run fsck for 'zfs-ssd:subvol-150-disk-0' (format == subvol)

Weird thing is, all the CT subvolumes are mounted on boot:

~# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=16399320k,nr_inodes=4099830,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=3284656k,mode=755)
rpool/ROOT/pve-1 on / type zfs (rw,relatime,xattr,noacl)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=43,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=32841)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
rpool-ssd on /rpool-ssd type zfs (rw,xattr,noacl)
rpool-ssd/subvol-152-disk-0 on /rpool-ssd/subvol-152-disk-0 type zfs (rw,xattr,posixacl)
rpool-ssd/subvol-151-disk-0 on /rpool-ssd/subvol-151-disk-0 type zfs (rw,xattr,posixacl)
rpool-ssd/subvol-150-disk-0 on /rpool-ssd/subvol-150-disk-0 type zfs (rw,xattr,posixacl)
rpool-ssd/subvol-153-disk-0 on /rpool-ssd/subvol-153-disk-0 type zfs (rw,xattr,posixacl)
rpool-ssd/subvol-203-disk-0 on /rpool-ssd/subvol-203-disk-0 type zfs (rw,xattr,posixacl)
rpool on /rpool type zfs (rw,noatime,xattr,noacl)
rpool/ROOT on /rpool/ROOT type zfs (rw,noatime,xattr,noacl)
rpool/data on /rpool/data type zfs (rw,noatime,xattr,noacl)
rpool/data/basevol-150-disk-0 on /rpool/data/basevol-150-disk-0 type zfs (rw,noatime,xattr,posixacl)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
<cut>:/volume1/VMBackups on /mnt/pve/synology-nfs type nfs4 (rw,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.88.249,local_lock=none,addr=192.168.88.245)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=3284652k,mode=700)

What can I do to fix this? What's wrong?

 

[fabian]

set a local tmpdir (in vzdump.conf) that is not on NFS. the backup needs to write as unprivileged user if the container is unprivileged.

 

[StanTastic]

Thanks, that worked!
I'm guessing that CT subvols should not be mounted, and I should disable this via ZFS settings?

 

[fabian]

Thanks, that worked!
I'm guessing that CT subvols should not be mounted, and I should disable this via ZFS settings?

they need to be mounted (they get bind-mounted into the container)