[SOLVED] lxc on omios iscsi - vzdump acl issue

R

RobFantini

Famous Member
May 24, 2012
2,042
107
133
Boston,Mass
Hello

I have some test lxc systems on iscsi/omnios

when vzdump runs this error occurs:
Code: 
INFO: starting new backup job: vzdump 3040 --compress lzo --node sys5 --mode snapshot --remove 0 --storage bkup
INFO: Starting Backup of VM 3040 (lxc)
INFO: status = running
INFO: mode failure - some volumes do not support snapshots
INFO: trying 'suspend' mode instead
INFO: backup mode: suspend
INFO: bandwidth limit: 500000 KB/s
INFO: ionice priority: 7
INFO: starting first sync /proc/21870/root// to /bkup/vzdumptmp15938
INFO: rsync: set_acl: sys_acl_set_file(bin, ACL_TYPE_DEFAULT): Operation not supported (95)
INFO: rsync: set_acl: sys_acl_set_file(boot, ACL_TYPE_DEFAULT): Operation not supported (95)
INFO: rsync: set_acl: sys_acl_set_file(etc, ACL_TYPE_DEFAULT): Operation not supported (95)
ERROR: Backup of VM 3040 failed - command 'rsync --stats -X -A --numeric-ids -aH --delete --no-whole-file --inplace --one-file-system --relative '--bwlimit=500000' '--exclude=/var/log/?*' '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/21870/root///./ /bkup/vzdumptmp15938' failed: exit code 23
INFO: Backup job finished with errors
TASK ERROR: job errors
stop mode backup works:
Code: 
NFO: starting new backup job: vzdump 3040 --storage bkup --remove 0 --mode stop --node sys5 --compress lzo
INFO: Starting Backup of VM 3040 (lxc)
INFO: status = running
INFO: backup mode: stop
INFO: bandwidth limit: 500000 KB/s
INFO: ionice priority: 7
  Found duplicate PV 4j27OuDdsJlyaiZkXR1yrcEvsAFY1bN7: using /dev/sdm5 not /dev/sdl5
  Found duplicate PV bCwewzHSzYTHJOSN2pCpfp6IjrOJRo4D: using /dev/sdp not /dev/sdq
  Found duplicate PV 4j27OuDdsJlyaiZkXR1yrcEvsAFY1bN7: using /dev/sdm5 not /dev/sdl5
  Found duplicate PV bCwewzHSzYTHJOSN2pCpfp6IjrOJRo4D: using /dev/sdp not /dev/sdq
INFO: stopping vm
INFO: creating archive '/bkup/dump/vzdump-lxc-3040-2016_04_13-14_41_29.tar.lzo'
INFO: Total bytes written: 2791383040 (2.6GiB, 110MiB/s)
INFO: archive file size: 817MB
INFO: restarting vm
INFO: vm is online again after 33 seconds
INFO: Finished Backup of VM 3040 (00:00:33)
INFO: Backup job finished successfully
TASK OK


Is there a setting to use on zfs to fix the acl issue?
 
can you post the storage.cfg part relating to "bkup"?

is that also zfs? zfs over iscsi?
if yes, please post the properties of that zfs dataset (with "zfs get acltype DATASET", replace DATASET with actual path). the error message indicates that rsync cannot set the acl on the backup tmp dir "/bkup/...".
If not, please post the output of "mount | grep bkup"
 
fabian said:
can you post the storage.cfg part relating to "bkup"?

is that also zfs? zfs over iscsi?
if yes, please post the properties of that zfs dataset (with "zfs get acltype DATASET", replace DATASET with actual path). the error message indicates that rsync cannot set the acl on the backup tmp dir "/bkup/...".
If not, please post the output of "mount | grep bkup"
Click to expand...

the storage is zfs not zfs over iscsi. a lvm is used.

I need help running 'zfs get acltype DATASET' . do I do that on pve or omnios .

and how do I find the 'DATASET' name?

/bkup:
Code: 
sys3  ~ # mount | grep bku
/dev/sdg1 on /bkup type ext4 (rw,noatime,data=ordered)


thank you for assisting.
 
Could you post the container configuration and the storage configuration?
 
the orig lxc 3040 is deleted. 3032 was a clone, has same storage and backup issue:
Code: 
INFO: starting new backup job: vzdump 3032 --compress lzo --node sys5 --mode snapshot --storage dump-save --remove 0
INFO: Starting Backup of VM 3032 (lxc)
INFO: status = running
INFO: mode failure - some volumes do not support snapshots
INFO: trying 'suspend' mode instead
INFO: backup mode: suspend
INFO: bandwidth limit: 500000 KB/s
INFO: ionice priority: 7
INFO: starting first sync /proc/32145/root// to /bkup/vzdumptmp32287
INFO: rsync: set_acl: sys_acl_set_file(bin, ACL_TYPE_DEFAULT): Operation not supported (95)
INFO: rsync: set_acl: sys_acl_set_file(boot, ACL_TYPE_DEFAULT): Operation not supported (95)
INFO: rsync: set_acl: sys_acl_set_file(etc, ACL_TYPE_DEFAULT): Operation not supported (95)

so here is 3032.conf
Code: 
#iscsi lvm
arch: amd64
cpulimit: 1
cpuunits: 1024
hostname: lxc-template
memory: 1024
net0: bridge=vmbr1,hwaddr=66:36:62:30:64:64,ip=dhcp,name=eth0,type=veth
onboot: 1
ostype: debian
protection: 1
rootfs: iscsi--for-lxc:vm-3032-disk-1,size=10G
swap: 512

pve storage
Code: 
dir: local
  path /var/lib/vz
  maxfiles 1
  content vztmpl

zfspool: lxc-zfs
  pool tank/lxc
  nodes dell1,sys5,sys4,sys3
  content rootdir

zfspool: kvm-zfs
  pool tank/kvm
  nodes sys3,sys5,sys4,dell1
  content images

dir: bkup
  path /bkup
  maxfiles 1
  content backup,vztmpl,iso

dir: dump-save
  path /bkup/dump-save
  maxfiles 2
  content backup
  nodes sys3,sys5,sys4,dell1

zfs: iscsi-sys4
  target iqn.2010-09.org.napp-it:1459891666
  pool data
  portal 10.2.2.41
  iscsiprovider comstar
  blocksize 8k
  content images
  nowritecache

iscsi: sys4-lxc-vol
  portal 10.2.2.41
  target iqn.2010-09.org.napp-it:1459891666
  content none

lvm: iscsi-for-lxc
  vgname lxc-iscsi-vg
  shared
  base sys4-lxc-vol:0.0.3.scsi-3600144f0000008080000570c0bb60001
  content rootdir
 
any special umasks or ACLs set on the /bkup filesystem?
what is the content of "/etc/vzdump.conf" ?
 
for now I have moved all lxc off of lvm on remote zfs/iscsi

omnios/nappit is being used here for kvm and backups.

the acl issue also occurs when I rsync dump files from omnios to extra backup storage on some systems. I have another thread going for that.

I think the acl issues can be fixed by zfs settings . when they are fixed I'll try lxc .
 
The error message indicates it cannot set the ACL on files/directories in the temporary directory in "/bkup", which according to your mount output is ext4 with standard mount options, not zfs.. You can try disabling ACLs on the zfs level (property "acltype" set to none), maybe omnios sets some ACL values which rsync on linux cannot interpret correctly and the ACL setting fails because of that - but the error message does not say anything like that ;)
 
Fabian - thank you for the solution.

for some reason zfs was set to this:
Code: 
# mount|grep bkup
tank/bkup on /bkup type zfs (rw,relatime,xattr,noacl)
i checked zpool history and do not see how it got set to noacl..

fixed with:
Code: 
zfs set acltype=posixacl tank/bkup
i noticed that acltype is used on lxc subvols when created, so used same on tank/bkup
Code: 
zfs create -o acltype=posixacl -o xattr=sa -o refquota=6291456k tank/lxc/subvol-5992-disk-1

so backups work . thank you.
 
the strange thing is that another system as the same setting:
Code: 
tank/bkup on /bkup type zfs (rw,noatime,xattr,noacl)

and again zpool history shows no trace of noacl. so it was not caused by a
Code: 
zpool history|grep acl
2016-01-04.19:38:48 zfs create -o acltype=posixacl -o xattr=sa -o refquota=31457280k tank/lxc/subvol-7106-disk-1
2016-01-09.18:09:10 zfs create -o acltype=posixacl -o xattr=sa -o refquota=8388608k tank/lxc/subvol-1543-disk-1
2016-01-09.18:24:24 zfs create -o acltype=posixacl -o xattr=sa -o refquota=8388608k tank/lxc/subvol-1744-disk-1
2016-02-09.10:09:19 zfs create -o acltype=posixacl -o xattr=sa -o refquota=4194304k tank/lxc/subvol-101-disk-1
2016-03-08.13:39:48 zfs create -o acltype=posixacl -o xattr=sa -o refquota=5242880k tank/lxc/subvol-113-disk-1
2016-04-14.22:08:15 zfs create -o acltype=posixacl -o xattr=sa -o refquota=6291456k tank/lxc/subvol-3123-disk-1
2016-04-17.15:03:34 zfs create -o acltype=posixacl -o xattr=sa -o refquota=2097152k tank/lxc/subvol-337-disk-1

now check this:
Code: 
# mount|grep acl
rpool/ROOT/pve-1 on / type zfs (rw,relatime,xattr,noacl)
tank/bkup on /bkup type zfs (rw,noatime,xattr,noacl)
rpool on /rpool type zfs (rw,noatime,xattr,noacl)
rpool/ROOT on /rpool/ROOT type zfs (rw,noatime,xattr,noacl)
tank on /tank type zfs (rw,noatime,xattr,noacl)
tank/kvm on /tank/kvm type zfs (rw,noatime,xattr,noacl)
tank/lxc on /tank/lxc type zfs (rw,noatime,xattr,noacl)
tank/lxc/subvol-100-disk-1 on /tank/lxc/subvol-100-disk-1 type zfs (rw,noatime,xattr,noacl)
tank/lxc/subvol-101-disk-1 on /tank/lxc/subvol-101-disk-1 type zfs (rw,noatime,xattr,posixacl)
tank/lxc/subvol-107-disk-1 on /tank/lxc/subvol-107-disk-1 type zfs (rw,noatime,xattr,noacl)
tank/lxc/subvol-108-disk-1 on /tank/lxc/subvol-108-disk-1 type zfs (rw,noatime,xattr,noacl)
tank/lxc/subvol-109-disk-1 on /tank/lxc/subvol-109-disk-1 type zfs (rw,noatime,xattr,posixacl)
tank/lxc/subvol-110-disk-1 on /tank/lxc/subvol-110-disk-1 type zfs (rw,noatime,xattr,posixacl)
tank/lxc/subvol-113-disk-1 on /tank/lxc/subvol-113-disk-1 type zfs (rw,noatime,xattr,posixacl)
tank/lxc/subvol-3105-disk-1 on /tank/lxc/subvol-3105-disk-1 type zfs (rw,noatime,xattr,noacl)
tank/lxc/subvol-3108-disk-1 on /tank/lxc/subvol-3108-disk-1 type zfs (rw,noatime,xattr,posixacl)
tank/lxc/subvol-3110-disk-1 on /tank/lxc/subvol-3110-disk-1 type zfs (rw,noatime,xattr,posixacl)
tank/lxc/subvol-3123-disk-1 on /tank/lxc/subvol-3123-disk-1 type zfs (rw,noatime,xattr,posixacl)
tank/lxc/subvol-7102-disk-1 on /tank/lxc/subvol-7102-disk-1 type zfs (rw,noatime,xattr,posixacl)
tank/lxc/subvol-7106-disk-1 on /tank/lxc/subvol-7106-disk-1 type zfs (rw,noatime,xattr,posixacl)
tank/lxc/subvol-7123-disk-1 on /tank/lxc/subvol-7123-disk-1 type zfs (rw,noatime,xattr,posixacl)
tank/pve on /tank/pve type zfs (rw,noatime,xattr,noacl)
tank/pve-zsync on /tank/pve-zsync type zfs (rw,noatime,xattr,noacl)
tank/pve-zsync/Weekly-dell1 on /tank/pve-zsync/Weekly-dell1 type zfs (rw,noatime,xattr,noacl)
tank/pve-zsync/Weekly-dell1/subvol-4444-disk-1 on /tank/pve-zsync/Weekly-dell1/subvol-4444-disk-1 type zfs (rw,noatime,xattr,noacl)
tank/temp on /tank/temp type zfs (rw,noatime,xattr,noacl)

Code: 
# mount|grep  rpool
rpool/ROOT/pve-1 on / type zfs (rw,relatime,xattr,noacl)
rpool on /rpool type zfs (rw,noatime,xattr,noacl)
rpool/ROOT on /rpool/ROOT type zfs (rw,noatime,xattr,noacl)
note some have noacl.

some get it inherited from tank. I'll check further.
 
I checked zpool history on 2 pve hosts . rpool and tank did not have a set noacl command.

next time someone creates a pool or does a new pve install , check if noacl is default.
 
You must log in or register to reply here.
Top Bottom