qm disk move (offline, --delete) removes source although target file is missing ("failed to stat") – TASK OK, data loss

K

kcastner_de

New Member
Oct 20, 2026
11
4
3
Environment
  • PVE 9.2.21, kernel 7.0.14-19-pve
  • Source: local-zfs (ZFS mirror, NVMe + SATA SSD)
  • Target: NFS storage ds212 on a Synology DS212+ (DSM 6.2.4), NFSv4.0, sec=sys, no root squash, sync export
  • Disk: unused1 = local-zfs:vm-600-disk-1, thin zvol, volsize 10 TiB, ~1.1 TiB logically used
  • VM stopped → offline move via qemu-img convert, target format qcow2, "Delete source" enabled
What happened

Attempt 1 (10:08) failed while creating the target image. Preallocation was still the default (metadata), and creating a 10 TiB qcow2 on the slow NFS server exceeded the storage lock timeout:
Code: 
TASK ERROR: storage migration failed: unable to create image: 'storage-ds212'-locked command timed out - aborting

After that I set
Code: 
pvesm set ds212 --preallocation off
and started attempt 2 (10:12). It copied for about 9 hours and finished with TASK OK:
Code: 
copying volume 'local-zfs:vm-600-disk-1' from current storage 'local-zfs' to target storage 'ds212'
create full clone of drive unused1 (local-zfs:vm-600-disk-1)
Formatting '/mnt/pve/ds212/images/600/vm-600-disk-0.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off preallocation=off compression_type=zlib size=10995116277760 lazy_refcounts=off refcount_bits=16
allocated target volume 'ds212:600/vm-600-disk-0.qcow2'
transferred 0.0 B of 10.0 TiB (0.00%)
transferred 102.4 GiB of 10.0 TiB (1.00%)
transferred 204.8 GiB of 10.0 TiB (2.00%)
transferred 307.2 GiB of 10.0 TiB (3.00%)
transferred 409.6 GiB of 10.0 TiB (4.00%)
transferred 512.0 GiB of 10.0 TiB (5.00%)
transferred 614.4 GiB of 10.0 TiB (6.00%)
transferred 716.8 GiB of 10.0 TiB (7.00%)
transferred 819.2 GiB of 10.0 TiB (8.00%)
transferred 921.6 GiB of 10.0 TiB (9.00%)
transferred 1.0 TiB of 10.0 TiB (10.00%)
transferred 1.1 TiB of 10.0 TiB (11.00%)
transferred 1.2 TiB of 10.0 TiB (12.00%)
transferred 1.3 TiB of 10.0 TiB (13.00%)
transferred 1.4 TiB of 10.0 TiB (14.00%)
transferred 1.5 TiB of 10.0 TiB (15.00%)
transferred 1.6 TiB of 10.0 TiB (16.00%)
transferred 1.7 TiB of 10.0 TiB (17.00%)
transferred 1.8 TiB of 10.0 TiB (18.00%)
transferred 1.9 TiB of 10.0 TiB (19.00%)
transferred 2.0 TiB of 10.0 TiB (20.00%)
transferred 2.1 TiB of 10.0 TiB (21.00%)
transferred 2.2 TiB of 10.0 TiB (22.00%)
transferred 2.3 TiB of 10.0 TiB (23.00%)
transferred 2.4 TiB of 10.0 TiB (24.00%)
transferred 2.5 TiB of 10.0 TiB (25.00%)
transferred 2.6 TiB of 10.0 TiB (26.00%)
transferred 2.7 TiB of 10.0 TiB (27.00%)
transferred 2.8 TiB of 10.0 TiB (28.00%)
transferred 2.9 TiB of 10.0 TiB (29.00%)
transferred 3.0 TiB of 10.0 TiB (30.00%)
transferred 3.1 TiB of 10.0 TiB (31.00%)
transferred 3.2 TiB of 10.0 TiB (32.00%)
transferred 3.3 TiB of 10.0 TiB (33.00%)
transferred 3.4 TiB of 10.0 TiB (34.00%)
transferred 3.5 TiB of 10.0 TiB (35.00%)
transferred 3.6 TiB of 10.0 TiB (36.00%)
transferred 3.7 TiB of 10.0 TiB (37.00%)
transferred 3.8 TiB of 10.0 TiB (38.00%)
transferred 3.9 TiB of 10.0 TiB (39.00%)
transferred 4.0 TiB of 10.0 TiB (40.00%)
transferred 4.1 TiB of 10.0 TiB (41.00%)
transferred 4.2 TiB of 10.0 TiB (42.00%)
transferred 4.3 TiB of 10.0 TiB (43.00%)
transferred 4.4 TiB of 10.0 TiB (44.00%)
transferred 4.5 TiB of 10.0 TiB (45.00%)
transferred 4.6 TiB of 10.0 TiB (46.00%)
transferred 4.7 TiB of 10.0 TiB (47.00%)
transferred 4.8 TiB of 10.0 TiB (48.00%)
transferred 4.9 TiB of 10.0 TiB (49.00%)
transferred 5.0 TiB of 10.0 TiB (50.00%)
transferred 5.1 TiB of 10.0 TiB (51.00%)
transferred 5.2 TiB of 10.0 TiB (52.00%)
transferred 5.3 TiB of 10.0 TiB (53.00%)
transferred 5.4 TiB of 10.0 TiB (54.00%)
transferred 5.5 TiB of 10.0 TiB (55.00%)
transferred 5.6 TiB of 10.0 TiB (56.00%)
transferred 5.7 TiB of 10.0 TiB (57.00%)
transferred 5.8 TiB of 10.0 TiB (58.00%)
transferred 5.9 TiB of 10.0 TiB (59.00%)
transferred 6.0 TiB of 10.0 TiB (60.00%)
transferred 6.1 TiB of 10.0 TiB (61.00%)
transferred 6.2 TiB of 10.0 TiB (62.00%)
transferred 6.3 TiB of 10.0 TiB (63.00%)
transferred 6.4 TiB of 10.0 TiB (64.00%)
transferred 6.5 TiB of 10.0 TiB (65.00%)
transferred 6.6 TiB of 10.0 TiB (66.00%)
transferred 6.7 TiB of 10.0 TiB (67.00%)
transferred 6.8 TiB of 10.0 TiB (68.00%)
transferred 6.9 TiB of 10.0 TiB (69.00%)
transferred 7.0 TiB of 10.0 TiB (70.00%)
transferred 7.1 TiB of 10.0 TiB (71.00%)
transferred 7.2 TiB of 10.0 TiB (72.00%)
transferred 7.3 TiB of 10.0 TiB (73.00%)
transferred 7.4 TiB of 10.0 TiB (74.00%)
transferred 7.5 TiB of 10.0 TiB (75.00%)
transferred 7.6 TiB of 10.0 TiB (76.00%)
transferred 7.7 TiB of 10.0 TiB (77.00%)
transferred 7.8 TiB of 10.0 TiB (78.00%)
transferred 7.9 TiB of 10.0 TiB (79.00%)
transferred 8.0 TiB of 10.0 TiB (80.00%)
transferred 8.1 TiB of 10.0 TiB (81.00%)
transferred 8.2 TiB of 10.0 TiB (82.00%)
transferred 8.3 TiB of 10.0 TiB (83.00%)
transferred 8.4 TiB of 10.0 TiB (84.00%)
transferred 8.5 TiB of 10.0 TiB (85.00%)
transferred 8.6 TiB of 10.0 TiB (86.00%)
transferred 8.7 TiB of 10.0 TiB (87.00%)
transferred 8.8 TiB of 10.0 TiB (88.00%)
transferred 8.9 TiB of 10.0 TiB (89.00%)
transferred 9.0 TiB of 10.0 TiB (90.00%)
transferred 9.1 TiB of 10.0 TiB (91.00%)
transferred 9.2 TiB of 10.0 TiB (92.00%)
transferred 9.3 TiB of 10.0 TiB (93.00%)
transferred 9.4 TiB of 10.0 TiB (94.00%)
transferred 9.5 TiB of 10.0 TiB (95.00%)
transferred 9.6 TiB of 10.0 TiB (96.00%)
transferred 9.7 TiB of 10.0 TiB (97.00%)
transferred 9.8 TiB of 10.0 TiB (98.00%)
transferred 9.9 TiB of 10.0 TiB (99.00%)
transferred 10.0 TiB of 10.0 TiB (100.00%)
transferred 10.0 TiB of 10.0 TiB (100.00%)
failed to stat '/mnt/pve/ds212/images/600/vm-600-disk-0.qcow2'
removing source volume 'local-zfs:vm-600-disk-1' after successful copy
TASK OK

Result: The source zvol was destroyed (zpool history: 19:20:34 zfs destroy -r rpool/data/vm-600-disk-1), the target file does not exist, and the VM config contains unused1: ds212:600/vm-600-disk-0.qcow2 pointing to nothing. Attaching it fails with volume 'ds212:600/vm-600-disk-0.qcow2' does not exist (500).

Core issue: Proxmox logged that it could not stat the freshly written target and still deleted the source and reported success. Whatever made the file disappear, a failed post-copy check of the target should be fatal and must never lead to removing the only remaining copy.

What I checked
  • Storage usage graph of ds212: grew steadily during the copy (≈1.1 → ≈2.35 TB) and dropped by ≈1.25 TB right when the copy ended (~19:20). The data was written and freed exactly at the end.
  • This pattern matches NFS "silly rename": if a file is unlinked while a client still has it open, the client renames it to .nfsXXXX (invisible to ls), keeps writing, and deletes it on close. The stat on the original name then fails with ENOENT.
  • No imgdel task around 19:20 in pvenode task list.
  • No rm of that file in the saved bash history. After attempt 1 I had planned to remove a possible leftover vm-600-disk-0.qcow2. Note that attempt 2 reused exactly the same volume name. I cannot fully rule out that a removal was executed in another shell while attempt 2 was already running. Either way, the source should not have been deleted.
  • NAS side (/var/log/messages, dmesg): no ext4 errors, no OOM, no reboot, nfsd running normally.
  • Client journal around 19:10–19:40: only nfsidmap domain mismatch warnings (root@localdomain vs. my NFSv4 domain), plus the failed to stat from pvedaemon.
  • find /mnt/pve/ds212 -name '*.qcow2': nothing.
Data was recovered from vzdump backups (two independent copies), so no permanent loss here. On a system without backups this would be total data loss.

Possible secondary issue: Can the cleanup of attempt 1 (aborted by the lock timeout while qemu-img create was still running on slow NFS) remove the volume name later, after attempt 2 has allocated the same name? That would explain the unlink without any user action.

Deterministic repro of the core issue (please test on a throwaway VM):
  1. Create a test VM with a disk on local storage, VM stopped.
  2. Start qm disk move <vmid> <disk> <nfs-storage> --format qcow2 --delete 1.
  3. While it is copying: rm /mnt/pve/<nfs-storage>/images/<vmid>/<target>.qcow2.
  4. Wait for completion. Expected: task error, source kept. Observed in my case: failed to stat + source removed + TASK OK.
Questions:
  1. Is it intended that a failed stat/size check of the target after copy is only a warning?
  2. Can anyone reproduce this, also with other target storages?
  3. Should the target be verified (exists, size, qemu-img check) before --delete removes the source?

Logs and the full task log available on request.
 
Hi,
please share the full system logs/journal from the Proxmox VE node around the time of the issue including both attempts, the storage configuration cat /etc/pve/storage.cfg as well as the output of findmnt for the storage and the output of pveversion -v. If there is any information about the storage operations/the access from the Proxmox VE node visible on the NAS side, please also include the logs from there. I'll try to reproduce the issue and work out patches to improve the checks in any case.
 
  • Like
Reactions: kcastner_de
Hi @fiona,

Thanks for looking into it. Attached bug8209-attachments.tar.gz
(hostnames, PBS details, remote IPs and key fingerprints redacted):

Proxmox VE node:
- journal-1000-1945.txt: full journal 2026-10-14 10:00-19:45 (both attempts)
- journal-attempt1-start2.txt: 10:00-10:30 (attempt 1 + start of attempt 2)
- journal-end-attempt2.txt: 19:00-19:45 (end of attempt 2)
- task-attempt1.txt / task-attempt2.txt: full task logs of both qmmove tasks
- storage.cfg, findmnt.txt (incl. nfsstat -m), pveversion.txt
- zpool-history-vm600.txt

NAS (Synology DS212+, DSM 6.2.4-25556):
- ds212-dmesg.txt (shortened to storage/network relevant lines)
- ds212-rm-log-20261014.txt
- ds212-synoservice-20261014.txt

Observations:
- Attempt 1 failed at 10:09:33 (lock timeout while creating the 10 TiB qcow2
with preallocation=metadata). Attempt 2 started at 10:12:20 and reused the
same volume name vm-600-disk-0.qcow2.
- Between the attempts a GUI shell was open (10:11:06-10:11:54). The shell
history shows no rm of the target, and there was no shell activity on the
target during attempt 2.
- qemu-img convert writes buffered to NFS: repeated hung task traces during
the copy (first at 10:22:55) show
pwrite64 -> nfs_file_write -> nfs_write_begin -> folio_wait_writeback.
- pvestatd reports timeouts / 13-22 s status updates between 19:18 and 19:20,
right before "failed to stat" at 19:20:33. The source zvol was destroyed
at 19:20:34.
- Storage usage of the NFS share grew during the whole copy and dropped by
~1.25 TB exactly at the end, so the data was written and freed only when
the file was closed.
- NAS side: DSM logs every invocation of the rm binary in /var/log/rm.log.
On 2026-10-14 there are only entries from package removals between 00:27
and 00:29 (paths under /usr and /var), none during the copy and none under
the export /volume1/proxmox. This log does not cover unlinks via nfsd or
other syscalls. DSM 6.2 does not log NFS client operations.
- NAS dmesg: no filesystem errors, no OOM, no reboot since the boot on
2026-10-13 ~23:39. The data volume is ext4 with data=writeback and
barriers disabled.

Happy to test patches or run additional checks.

EDIT:
Just to clarify, the PVE and NAS are both running on Layer 2 within the same network. I'm using a /16 subnet here.
The NAS is connected directly to the server via vmbr, which maps VLAN 100 from eno1 to untagged on eno4. The NAS is connected to eno4.
 

Attachments

Last edited:
Hi @fiona,

regarding to you bugzilla post:

Regarding the missing "end task" line: the qmmove task was started at
10:12:20 by pvedaemon worker 529057. That worker was recycled at 10:32:21
(journal: "pvedaemon[529057]: worker exit" / "pvedaemon[3344]: worker
529057 finished"), long before the task ended. The task process itself
(pid 534447) kept running and logged "failed to stat" at 19:20:33, so
presumably no process was left to write the "end task" line to the journal.

The task index records the end:

Code: 
UPID:ugrwffaghevmcauw:000827AF:0998C8C0:6ACF3964:qmmove:600:root@pam: 6ACFB9EC OK

0x6ACFB9EC = 2026-10-14 19:20:44 CEST (started 0x6ACF3964 = 10:12:20).
The task log file was last modified at 19:20:

Code: 
-rw-r----- 1 www-data root 4752 Oct 14 19:20 /var/log/pve/tasks/4/UPID:ugrwffaghevmcauw:000827AF:0998C8C0:6ACF3964:qmmove:600:root@pam:
 
Last edited:
You must log in or register to reply here.
Top Bottom
Back