move vm/lxc from ZFSHD to local-vm(pve)

[kreutpet]

HI,

since a few day i do get error that indicates that my hard disk is dying.

Dec 18 13:07:59 pve smartd[1925]: Device: /dev/sdg [SAT], 1 Offline uncorrectable sectors


The partition /dev/sdg1 is the disk that is allocated to the ZFSHD

/etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content vztmpl,iso,backup

lvmthin: local-lvm
        thinpool data
        vgname pve
        content images,rootdir

zfspool: ZFSHD
        pool ZFSHD
        content rootdir,images
        mountpoint /ZFSHD
        nodes pve

pbs: backupPBS
        datastore backupPVE
        server 127.0.0.1
        content backup
        fingerprint b5:49:01:f5:e6:e5:ad:8d:21:38:40:fb:77:68:96:46:c9:54:93:a6:7e:0d:46:3b:c7:64:97:33:5c:a6:a6:f7
        nodes pve
        prune-backups keep-all=1
        username root@pam

to move the VM/LXC i tried to clone a stopped LXC on ZFSHD to the local-lvm

Header
Proxmox
Virtual Environment 7.1-8
Storage 'ZFSHD' on node 'pve'
Enabled
 
Yes
Active
 
Yes
Content
 
Disk image, Container
Type
 
ZFS
Usage
 
10.53% (76.23 GB of 724.10 GB)
Logs
()
create full clone of mountpoint rootfs (ZFSHD:subvol-118-disk-0)
  Logical volume "vm-106-disk-0" created.
Creating filesystem with 393216 4k blocks and 98304 inodes
Filesystem UUID: 58dfb352-5e56-47ac-bc06-e5a0bbc14668
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912
rsync: [receiver] write failed on "/var/lib/lxc/106/.copy-volume-1/var/log/journal/db91a8d9813048b3b0d9627361bf92a2/system@ebe07a11427749cdae31bc510bfe0c40-000000000004d62f-0005d07da3d06a3f.journal": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(378) [receiver=3.2.3]
rsync: [sender] write error: Broken pipe (32)
  Logical volume "vm-106-disk-0" successfully removed
TASK ERROR: clone failed: command 'rsync --stats -X -A --numeric-ids -aH --whole-file --sparse --one-file-system '--bwlimit=0' /var/lib/lxc/106/.copy-volume-2/ /var/lib/lxc/106/.copy-volume-1' failed: exit code 11

the cloning fails with a "No space left on device (28)"

df
Dateisystem              1K-Blöcke    Benutzt  Verfügbar Verw% Eingehängt auf
udev                       8154872          0    8154872    0% /dev
tmpfs                      1637532       1524    1636008    1% /run
/dev/mapper/pve-root      98559220   10244252   83265420   11% /

pvs
  PV         VG  Fmt  Attr PSize    PFree
  /dev/sda3  pve lvm2 a--  <931.01g 15.99g

lvs
  LV            VG  Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data          pve twi-aotz-- <794.79g             3.34   0.39                           
  root          pve -wi-ao----   96.00g                                                   
  swap          pve -wi-ao----    8.00g                                                   
  vm-100-disk-0 pve Vwi-a-tz--    4.00g data        58.56                                 
  vm-101-disk-0 pve Vwi-aotz--    6.00g data        99.76                                 
  vm-114-disk-0 pve Vwi-aotz--    2.50g data        99.51                                 
  vm-115-disk-0 pve Vwi-aotz--    3.00g data        99.81                                 
  vm-116-disk-0 pve Vwi-aotz--    2.50g data        99.82                                 
  vm-117-disk-0 pve Vwi-aotz--    3.00g data        95.55                                 
  vm-119-disk-0 pve Vwi-aotz--    3.50g data        99.79                                 
  vm-120-disk-0 pve Vwi-aotz--   <3.91g data        99.69

My understand is that the Thinpool on the pve volumen group has enough disk space be able to move the ZFSHD VM/LXC to it.

could you advice why i do get this error message?
what is the best way to move the VM/LXC from a dying disk to my local-lvm?

thanks

 

[Dunuin]

If I understand it right a clone still requires the original VM/LXC it was cloned from. So you might want to do a direct migration or better use Vzdump/PBS to backup everything on that ZFS pool in case it might die tomorrow and then restore there VMs/LXCs from the backups to the LVM-thin.

 

[kreutpet]

i do have backups in pbs and did try to restore it to local-lvm.
The error is the same the target has not space left.

()
recovering backed-up configuration from 'backupPBS:backup/ct/118/2021-12-14T05:35:45Z'
  Logical volume "vm-118-disk-0" created.
Creating filesystem with 393216 4k blocks and 98304 inodes
Filesystem UUID: dc9e1733-4576-4acf-a077-bc232a21c139
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912
restoring 'backupPBS:backup/ct/118/2021-12-14T05:35:45Z' now..
Error: error extracting archive - error at entry "system@ebe07a11427749cdae31bc510bfe0c40-000000000003f68c-0005d03c9c3d91fb.journal": failed to copy file contents: No space left on device (os error 28)
  Logical volume "vm-118-disk-0" successfully removed
TASK ERROR: unable to restore CT 118 - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client restore '--crypt-mode=none' ct/118/2021-12-14T05:35:45Z root.pxar /var/lib/lxc/118/rootfs --allow-existing-dirs --repository root@pam@127.0.0.1:backupPVE' failed: exit code 255

In the restore i cannot choose LVM-thin but only only local-lvm.
how to i specify the restore the VM/LXC to the LVM-thin?
i was searching for a cli example but could'nt find something

 

[Dunuin]

In the restore i cannot choose LVM-thin but only only local-lvm.
how to i specify the restore the VM/LXC to the LVM-thin?

I don't get what you mean. "local-lvm" is the name of your LVM-thin. So you are already using it.

 

[kreutpet]

well then the original problem seems to be that the original data (post #1) i shared shows plenty of space on the "data" LVM.
So why do i get a "no space left on device" on both operation?

 

[Syrrys]

I'm getting the same error. I'm trying to move a 2GB disk image from local-zfs to my cephrbd storage. I have tried to move the volume in the GUI and tried deleting the CT and restoring from backup; I can restore to local-zfs on any node but trying to restore to the cephrbd also produces a no space available.
I've tried to expand the drive size to 4Gb in case there was need to make a full copy within the container with no change....
I'm quite puzzled

Edit: here's error trying to move the volume

rsync: [receiver] write failed on "/var/lib/lxc/105/.copy-volume-1/var/log/journal/c855a9be5702ba9392e5d44e5fb289f9/system@394746c4f17a4c1f854cebb019d152a8-00000000000137bc-0005cf9a6ba712e1.journal": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(378) [receiver=3.2.3]
rsync: [sender] write error: Broken pipe (32)
Removing image: 100% complete...done.
TASK ERROR: command 'rsync --stats -X -A --numeric-ids -aH --whole-file --sparse --one-file-system '--bwlimit=4096' /var/lib/lxc/105/.copy-volume-2/ /var/lib/lxc/105/.copy-volume-1' failed: exit code 11

Edit2: I might have had an actual issue with my log file sizes. reverting to a prior backup and clearing old replication jobs/disk images has got me to where I want to be.