Unable to move LXC container disk(s) to another storage / rsync error:11 / quota exceeded

U

Urbanovits

Active Member
Mar 14, 2021
54
4
28
57
Hello

i'm stucked with this issue
Guessing... it could be a ZFS quota issue, but no quota were set, and inode usage on 1%

LXC container volumes need to be relocated to another (iSCSI) storage. Problem with source?

Log say:

Wiping dos signature on /dev/vg02/vm-100-disk-0.
Logical volume "vm-100-disk-0" created.
Creating filesystem with 2621440 4k blocks and 655360 inodes
Filesystem UUID: dca0f4b6-e4f3-458a-9543-a1b0dd580687
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
rsync: [receiver] write failed on "/var/lib/lxc/100/.copy-volume-1/var/urbackup/backup_server_files.db": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(381) [receiver=3.2.7]
rsync: [sender] write error: Broken pipe (32)
Logical volume "vm-100-disk-0" successfully removed.
TASK ERROR: command 'rsync --stats -X -A --numeric-ids -aH --whole-file --sparse --one-file-system '--bwlimit=0' /var/lib/lxc/100/.copy-volume-2/ /var/lib/lxc/100/.copy-volume-1' failed: exit code 11

Server has 32G ram and no VM/LXC rumning.

df -h

Code: 
Data02/subvol-100-disk-0   10G  4.5G  5.6G  45% /Data02/subvol-100-disk-0
Data02/subvol-100-disk-1  4.9T  2.4T  2.6T  49% /Data02/subvol-100-disk-1

zfs get refquota Data02/subvol-100-disk-0
and
zfs get refquota Data02/subvol-100-disk-1

Code: 
Data02/subvol-100-disk-1  refquota  4.88T     local
root@pve04:~# zfs get refquota Data02/subvol-100-disk-0
NAME                      PROPERTY  VALUE     SOURCE
Data02/subvol-100-disk-0  refquota  10G       local
root@pve04:~# zfs get refquota Data02/subvol-100-disk-1
NAME                      PROPERTY  VALUE     SOURCE
Data02/subvol-100-disk-1  refquota  4.88T     local


Code: 
df -i

Data02/subvol-100-disk-0    11797360    85080    11712280    1% /Data02/subvol-100-disk-0
Data02/subvol-100-disk-1  5422031467 33072667  5388958800    1% /Data02/subvol-100-disk-1

Please advise what to do
 
Last edited:
I don't think so

root@pve04:~# df -h /var/lib/lxc/
Filesystem Size Used Avail Use% Mounted on
rpool/ROOT/pve-1 450G 20G 430G 5% /
 
I see this thread died, but I'm encountering a similar issue. The only difference is moving LXC volume from zfs datastore to lvm-thin datastore, but otherwise identical results, while plenty of free space on /var/lib/lxc/

Please post back if you found a solution, but assume you likely backed up entire guest then restored to the other datastore ?

This seems to have broken recently so wonder if there is an actual regression in PVE ?
 
fiveangle said:
I see this thread died, but I'm encountering a similar issue. The only difference is moving LXC volume from zfs datastore to lvm-thin datastore, but otherwise identical results, while plenty of free space on /var/lib/lxc/

Please post back if you found a solution, but assume you likely backed up entire guest then restored to the other datastore ?

This seems to have broken recently so wonder if there is an actual regression in PVE ?
Click to expand...
I had no idea how to solve. It is a simple copy task on "large" sized volume. My guess it should be some timeout issue on debian hosted rsync services or rsync itself. Workaround would be CT poweroff, then backup and restore on different location OR rsync cmd line with --partial and -n(normal reads) and -v (verbose) to find more info, but it is so time consuming process. Not tested because lack of free time.

FYI: https://serverfault.com/questions/762524/rsync-hangs-on-one-big-file
 
Last edited:
UPDATE: so in my case (and yours) the target vol indeed filled up before the move could complete. Turns out the size=nG spec for LXC rootfs/mpX virtual disks stored on a zfs-backed PVE storage pool references only the literal amount of physical blocks that the the LXC's virtual disk consumes from the parent zfs pool, and does not reference the logicalusedor logicalreferenced values of the virutal disk's underlying zfs dataset at all (!)

This is a glaring functional design flaw since moving a virutal disk from being stored on a zfs dataset (like with LXCs) to a storage pool backed by anything other than zfs, the storage for that target disk will be created at same size as the source zfs vol, yet it must somehow accept the fully-uncompressed data that the virtual disk's zfs dataset represents in the real-world, plus the space needed to store any associated inodes, xattrs, etc that might be needed by the target fs to store thoses files.


Sooooooo... [more gray matter logic that has to be ingrained before moving any LXC vols from zfs storage to any other...]

One has to MANUALLY CHECK THEMSELVES before moving that the size of the zfs logicalreferenced property of the virtual disk's underlying zfs dataset is smaller than its refquota property (plus target storage fs overhead). Otherwise, pvesm will gladly setup the target vol, rsync will gleefully copy each file to it, and zfs will merrily decompress/checksum all of it as it leaves.... until the target volume fills up (typically towards the very end): BOMBS AWAY ! :eyeroll:

So, print this out and post it near every PVE admin's monitor:

Code: 
root@richie:~# zfs get all rpool/data/subvol-122-disk-1|egrep "refquota|referenced|refcompressratio"
rpool/data/subvol-122-disk-1  referenced            2.23G                          -
rpool/data/subvol-122-disk-1  refquota              4G                             local
rpool/data/subvol-122-disk-1  refcompressratio      3.99x                          -
rpool/data/subvol-122-disk-1  logicalreferenced     6.57G                          -
root@richie:~# zfs set refquota=8G rpool/data/subvol-122-disk-1
root@richie:~# zfs get all rpool/data/subvol-122-disk-1|egrep "refquota|referenced|refcompressratio"
rpool/data/subvol-122-disk-1  referenced            2.23G                          -
rpool/data/subvol-122-disk-1  refquota              8G                             local
rpool/data/subvol-122-disk-1  refcompressratio      3.99x                          -
rpool/data/subvol-122-disk-1  logicalreferenced     6.57G                          -
root@richie:~# pct rescan
rescan volumes...
CT 122: updated volume size of 'local-zfs:subvol-122-disk-1' in config.
root@richie:~# grep subvol-122-disk-1 /etc/pve/lxc/*.conf
/etc/pve/lxc/122.conf:rootfs: local-zfs:subvol-122-disk-1,mountoptions=noatime,size=7G


TL;DR: logicalreferenced+target storage fs overhead MUST BE SMALLER THAN refquota in order to move any LXC virtual disk off of zfs.


Ps. At the time the error was logged in your case, the path being referenced is /var/lib/lxc/100/.copy-volume-1 so you would need to 'df /var/lib/lxc/100/.copy-volume-1/' while the move is occurring in order to see the volume filling up. After it fails, PVE rolls back the state machine to before initiating the move, so it naturally removes the target volume, which is why "df"ing /var/lib/lxc/100/ at a later time only gets you the current state of the node's local file system, which naturally isn't useful.
Click to expand...
 
Last edited:
That is what I have

Move failed to relocate these ZFS volumes to an iSCSI LVM (8Tbyte).


1746023771788.png

root@pve04:~# zfs get all /Data02/subvol-100-disk-0|egrep "refquota|referenced|refcompressratio"
Data02/subvol-100-disk-0 referenced 4.50G -
Data02/subvol-100-disk-0 refquota 10G local
Data02/subvol-100-disk-0 refcompressratio 3.93x -
Data02/subvol-100-disk-0 logicalreferenced 17.2G
-
root@pve04:~# zfs get all /Data02/subvol-100-disk-1|egrep "refquota|referenced|refcompressratio"
Data02/subvol-100-disk-1 referenced 2.38T -
Data02/subvol-100-disk-1 refquota 4.88T local
Data02/subvol-100-disk-1 refcompressratio 2.11x -
Data02/subvol-100-disk-1 logicalreferenced 4.85T -

I'm afraid I have no other choice to left this disks alone on ZFS.

Any idea how to increase refquota temporarily?
 
Yeah, why I posted it for you above:
Bash: 
[...]
########the following command "resizes" the virtual disk size##########
root@richie:~# zfs set refquota=8G rpool/data/subvol-122-disk-1 <=======this command
[...]
########the following command instructs pvesm to realize the new size by updating the /etc/pve/lxc/[ID].conf########
root@richie:~# pct rescan 
rescan volumes...
CT 122: updated volume size of 'local-zfs:subvol-122-disk-1' in config.
[...]
[go back up and follow the sequence of what i did...
-got the info about the zfs vol to determine how big to resize it
-resize it
-have pct update pve on the new disk size
-move, move, move


So since your logicalreferenced is ~4.85T, minimally you'd just:

zfs set refquota=5T /Data02/subvol-100-disk-1
pct rescan
move vol to lvm storage pool in gui

If you intend to increase the disk size above 5TB after the move, just do the adjustment on zfs with the zfs-set above, then move it in the gui and pvesm will layout the storage on your lvm optimally, without the need for a subsequent resize (as it will allocate adequate inodes and backup btrees for the final slice size).
 
You must log in or register to reply here.
Top Bottom