Environment
Attempt 1 (10:08) failed while creating the target image. Preallocation was still the default (metadata), and creating a 10 TiB qcow2 on the slow NFS server exceeded the storage lock timeout:
After that I set
and started attempt 2 (10:12). It copied for about 9 hours and finished with TASK OK:
Result: The source zvol was destroyed (zpool history: 19:20:34 zfs destroy -r rpool/data/vm-600-disk-1), the target file does not exist, and the VM config contains unused1: ds212:600/vm-600-disk-0.qcow2 pointing to nothing. Attaching it fails with volume 'ds212:600/vm-600-disk-0.qcow2' does not exist (500).
Core issue: Proxmox logged that it could not stat the freshly written target and still deleted the source and reported success. Whatever made the file disappear, a failed post-copy check of the target should be fatal and must never lead to removing the only remaining copy.
What I checked
Possible secondary issue: Can the cleanup of attempt 1 (aborted by the lock timeout while qemu-img create was still running on slow NFS) remove the volume name later, after attempt 2 has allocated the same name? That would explain the unlink without any user action.
Deterministic repro of the core issue (please test on a throwaway VM):
Logs and the full task log available on request.
- PVE 9.2.21, kernel 7.0.14-19-pve
- Source: local-zfs (ZFS mirror, NVMe + SATA SSD)
- Target: NFS storage ds212 on a Synology DS212+ (DSM 6.2.4), NFSv4.0, sec=sys, no root squash, sync export
- Disk: unused1 = local-zfs:vm-600-disk-1, thin zvol, volsize 10 TiB, ~1.1 TiB logically used
- VM stopped → offline move via qemu-img convert, target format qcow2, "Delete source" enabled
Attempt 1 (10:08) failed while creating the target image. Preallocation was still the default (metadata), and creating a 10 TiB qcow2 on the slow NFS server exceeded the storage lock timeout:
Code:
TASK ERROR: storage migration failed: unable to create image: 'storage-ds212'-locked command timed out - abortingAfter that I set
Code:
pvesm set ds212 --preallocation off
Code:
copying volume 'local-zfs:vm-600-disk-1' from current storage 'local-zfs' to target storage 'ds212'
create full clone of drive unused1 (local-zfs:vm-600-disk-1)
Formatting '/mnt/pve/ds212/images/600/vm-600-disk-0.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off preallocation=off compression_type=zlib size=10995116277760 lazy_refcounts=off refcount_bits=16
allocated target volume 'ds212:600/vm-600-disk-0.qcow2'
transferred 0.0 B of 10.0 TiB (0.00%)
transferred 102.4 GiB of 10.0 TiB (1.00%)
transferred 204.8 GiB of 10.0 TiB (2.00%)
transferred 307.2 GiB of 10.0 TiB (3.00%)
transferred 409.6 GiB of 10.0 TiB (4.00%)
transferred 512.0 GiB of 10.0 TiB (5.00%)
transferred 614.4 GiB of 10.0 TiB (6.00%)
transferred 716.8 GiB of 10.0 TiB (7.00%)
transferred 819.2 GiB of 10.0 TiB (8.00%)
transferred 921.6 GiB of 10.0 TiB (9.00%)
transferred 1.0 TiB of 10.0 TiB (10.00%)
transferred 1.1 TiB of 10.0 TiB (11.00%)
transferred 1.2 TiB of 10.0 TiB (12.00%)
transferred 1.3 TiB of 10.0 TiB (13.00%)
transferred 1.4 TiB of 10.0 TiB (14.00%)
transferred 1.5 TiB of 10.0 TiB (15.00%)
transferred 1.6 TiB of 10.0 TiB (16.00%)
transferred 1.7 TiB of 10.0 TiB (17.00%)
transferred 1.8 TiB of 10.0 TiB (18.00%)
transferred 1.9 TiB of 10.0 TiB (19.00%)
transferred 2.0 TiB of 10.0 TiB (20.00%)
transferred 2.1 TiB of 10.0 TiB (21.00%)
transferred 2.2 TiB of 10.0 TiB (22.00%)
transferred 2.3 TiB of 10.0 TiB (23.00%)
transferred 2.4 TiB of 10.0 TiB (24.00%)
transferred 2.5 TiB of 10.0 TiB (25.00%)
transferred 2.6 TiB of 10.0 TiB (26.00%)
transferred 2.7 TiB of 10.0 TiB (27.00%)
transferred 2.8 TiB of 10.0 TiB (28.00%)
transferred 2.9 TiB of 10.0 TiB (29.00%)
transferred 3.0 TiB of 10.0 TiB (30.00%)
transferred 3.1 TiB of 10.0 TiB (31.00%)
transferred 3.2 TiB of 10.0 TiB (32.00%)
transferred 3.3 TiB of 10.0 TiB (33.00%)
transferred 3.4 TiB of 10.0 TiB (34.00%)
transferred 3.5 TiB of 10.0 TiB (35.00%)
transferred 3.6 TiB of 10.0 TiB (36.00%)
transferred 3.7 TiB of 10.0 TiB (37.00%)
transferred 3.8 TiB of 10.0 TiB (38.00%)
transferred 3.9 TiB of 10.0 TiB (39.00%)
transferred 4.0 TiB of 10.0 TiB (40.00%)
transferred 4.1 TiB of 10.0 TiB (41.00%)
transferred 4.2 TiB of 10.0 TiB (42.00%)
transferred 4.3 TiB of 10.0 TiB (43.00%)
transferred 4.4 TiB of 10.0 TiB (44.00%)
transferred 4.5 TiB of 10.0 TiB (45.00%)
transferred 4.6 TiB of 10.0 TiB (46.00%)
transferred 4.7 TiB of 10.0 TiB (47.00%)
transferred 4.8 TiB of 10.0 TiB (48.00%)
transferred 4.9 TiB of 10.0 TiB (49.00%)
transferred 5.0 TiB of 10.0 TiB (50.00%)
transferred 5.1 TiB of 10.0 TiB (51.00%)
transferred 5.2 TiB of 10.0 TiB (52.00%)
transferred 5.3 TiB of 10.0 TiB (53.00%)
transferred 5.4 TiB of 10.0 TiB (54.00%)
transferred 5.5 TiB of 10.0 TiB (55.00%)
transferred 5.6 TiB of 10.0 TiB (56.00%)
transferred 5.7 TiB of 10.0 TiB (57.00%)
transferred 5.8 TiB of 10.0 TiB (58.00%)
transferred 5.9 TiB of 10.0 TiB (59.00%)
transferred 6.0 TiB of 10.0 TiB (60.00%)
transferred 6.1 TiB of 10.0 TiB (61.00%)
transferred 6.2 TiB of 10.0 TiB (62.00%)
transferred 6.3 TiB of 10.0 TiB (63.00%)
transferred 6.4 TiB of 10.0 TiB (64.00%)
transferred 6.5 TiB of 10.0 TiB (65.00%)
transferred 6.6 TiB of 10.0 TiB (66.00%)
transferred 6.7 TiB of 10.0 TiB (67.00%)
transferred 6.8 TiB of 10.0 TiB (68.00%)
transferred 6.9 TiB of 10.0 TiB (69.00%)
transferred 7.0 TiB of 10.0 TiB (70.00%)
transferred 7.1 TiB of 10.0 TiB (71.00%)
transferred 7.2 TiB of 10.0 TiB (72.00%)
transferred 7.3 TiB of 10.0 TiB (73.00%)
transferred 7.4 TiB of 10.0 TiB (74.00%)
transferred 7.5 TiB of 10.0 TiB (75.00%)
transferred 7.6 TiB of 10.0 TiB (76.00%)
transferred 7.7 TiB of 10.0 TiB (77.00%)
transferred 7.8 TiB of 10.0 TiB (78.00%)
transferred 7.9 TiB of 10.0 TiB (79.00%)
transferred 8.0 TiB of 10.0 TiB (80.00%)
transferred 8.1 TiB of 10.0 TiB (81.00%)
transferred 8.2 TiB of 10.0 TiB (82.00%)
transferred 8.3 TiB of 10.0 TiB (83.00%)
transferred 8.4 TiB of 10.0 TiB (84.00%)
transferred 8.5 TiB of 10.0 TiB (85.00%)
transferred 8.6 TiB of 10.0 TiB (86.00%)
transferred 8.7 TiB of 10.0 TiB (87.00%)
transferred 8.8 TiB of 10.0 TiB (88.00%)
transferred 8.9 TiB of 10.0 TiB (89.00%)
transferred 9.0 TiB of 10.0 TiB (90.00%)
transferred 9.1 TiB of 10.0 TiB (91.00%)
transferred 9.2 TiB of 10.0 TiB (92.00%)
transferred 9.3 TiB of 10.0 TiB (93.00%)
transferred 9.4 TiB of 10.0 TiB (94.00%)
transferred 9.5 TiB of 10.0 TiB (95.00%)
transferred 9.6 TiB of 10.0 TiB (96.00%)
transferred 9.7 TiB of 10.0 TiB (97.00%)
transferred 9.8 TiB of 10.0 TiB (98.00%)
transferred 9.9 TiB of 10.0 TiB (99.00%)
transferred 10.0 TiB of 10.0 TiB (100.00%)
transferred 10.0 TiB of 10.0 TiB (100.00%)
failed to stat '/mnt/pve/ds212/images/600/vm-600-disk-0.qcow2'
removing source volume 'local-zfs:vm-600-disk-1' after successful copy
TASK OKResult: The source zvol was destroyed (zpool history: 19:20:34 zfs destroy -r rpool/data/vm-600-disk-1), the target file does not exist, and the VM config contains unused1: ds212:600/vm-600-disk-0.qcow2 pointing to nothing. Attaching it fails with volume 'ds212:600/vm-600-disk-0.qcow2' does not exist (500).
Core issue: Proxmox logged that it could not stat the freshly written target and still deleted the source and reported success. Whatever made the file disappear, a failed post-copy check of the target should be fatal and must never lead to removing the only remaining copy.
What I checked
- Storage usage graph of ds212: grew steadily during the copy (≈1.1 → ≈2.35 TB) and dropped by ≈1.25 TB right when the copy ended (~19:20). The data was written and freed exactly at the end.
- This pattern matches NFS "silly rename": if a file is unlinked while a client still has it open, the client renames it to .nfsXXXX (invisible to ls), keeps writing, and deletes it on close. The stat on the original name then fails with ENOENT.
- No imgdel task around 19:20 in pvenode task list.
- No rm of that file in the saved bash history. After attempt 1 I had planned to remove a possible leftover vm-600-disk-0.qcow2. Note that attempt 2 reused exactly the same volume name. I cannot fully rule out that a removal was executed in another shell while attempt 2 was already running. Either way, the source should not have been deleted.
- NAS side (/var/log/messages, dmesg): no ext4 errors, no OOM, no reboot, nfsd running normally.
- Client journal around 19:10–19:40: only nfsidmap domain mismatch warnings (root@localdomain vs. my NFSv4 domain), plus the failed to stat from pvedaemon.
- find /mnt/pve/ds212 -name '*.qcow2': nothing.
Possible secondary issue: Can the cleanup of attempt 1 (aborted by the lock timeout while qemu-img create was still running on slow NFS) remove the volume name later, after attempt 2 has allocated the same name? That would explain the unlink without any user action.
Deterministic repro of the core issue (please test on a throwaway VM):
- Create a test VM with a disk on local storage, VM stopped.
- Start qm disk move <vmid> <disk> <nfs-storage> --format qcow2 --delete 1.
- While it is copying: rm /mnt/pve/<nfs-storage>/images/<vmid>/<target>.qcow2.
- Wait for completion. Expected: task error, source kept. Observed in my case: failed to stat + source removed + TASK OK.
- Is it intended that a failed stat/size check of the target after copy is only a warning?
- Can anyone reproduce this, also with other target storages?
- Should the target be verified (exists, size, qemu-img check) before --delete removes the source?
Logs and the full task log available on request.