I/O Write Errors in VM running on NetApp NFS with sparsefile support enabled (DEALLOCATE returns NFS4ERR_IO)

C

CarstenGrohmann

New Member
May 7, 2025
4
1
3
I run a stacked setup with Debian 13 on PVE (hostname pve150) on PVE and use an NFS v4.2 datastore backed by a NetApp ONTAP Simulator.

I see some serious errors in my VM at 14:19 CEST:

Code: 
[ 43.534144] I/O error, dev sda, sector 812584 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 2
[ 43.534248] EXT4-fs (sda1): failed to convert unwritten extents to written extents -- potential data loss! (inode 2391, error -5)

Since I suspected NFS issues, I ran tcpdump, and it also reported no problems with (nfs.nfsstat4 > 0). I also checked the PVE server kernel log to locate the reason for these I/O errors, but the kernel log was "empty."

Where in the stack is the error?
How can I narrow down the error further?

dmesg -T

Code: 
Aug 21 14:17:01 pve150 CRON[812214]: pam_unix(cron:session): session closed for user root
Aug 21 14:19:17 pve150 pvedaemon[812698]: start VM 100: UPID:pve150:000C669A:0185C1AD:6A884245:qmstart:100:root@pam:
Aug 21 14:19:17 pve150 pvedaemon[780617]: <root@pam> starting task UPID:pve150:000C669A:0185C1AD:6A884245:qmstart:100:root@pam:
Aug 21 14:19:17 pve150 nfsidmap[812701]: nss_getpwnam: name 'root@defaultv4iddomain.com' does not map into domain 'example.local'
Aug 21 14:19:17 pve150 nfsidmap[812702]: nss_name_to_gid: name 'root@defaultv4iddomain.com' does not map into domain 'example.local'
Aug 21 14:19:17 pve150 systemd[1]: Started 100.scope.
Aug 21 14:19:18 pve150 kernel: tap100i0: entered promiscuous mode
Aug 21 14:19:18 pve150 kernel: vmbr0: port 2(fwpr100p0) entered blocking state
Aug 21 14:19:18 pve150 kernel: vmbr0: port 2(fwpr100p0) entered disabled state
Aug 21 14:19:18 pve150 kernel: fwpr100p0: entered allmulticast mode
Aug 21 14:19:18 pve150 kernel: fwpr100p0: entered promiscuous mode
Aug 21 14:19:18 pve150 kernel: vmbr0: port 2(fwpr100p0) entered blocking state
Aug 21 14:19:18 pve150 kernel: vmbr0: port 2(fwpr100p0) entered forwarding state
Aug 21 14:19:18 pve150 kernel: fwbr100i0: port 1(fwln100i0) entered blocking state
Aug 21 14:19:18 pve150 kernel: fwbr100i0: port 1(fwln100i0) entered disabled state
Aug 21 14:19:18 pve150 kernel: fwln100i0: entered allmulticast mode
Aug 21 14:19:18 pve150 kernel: fwln100i0: entered promiscuous mode
Aug 21 14:19:18 pve150 kernel: fwbr100i0: port 1(fwln100i0) entered blocking state
Aug 21 14:19:18 pve150 kernel: fwbr100i0: port 1(fwln100i0) entered forwarding state
Aug 21 14:19:18 pve150 kernel: fwbr100i0: port 2(tap100i0) entered blocking state
Aug 21 14:19:18 pve150 kernel: fwbr100i0: port 2(tap100i0) entered disabled state
Aug 21 14:19:18 pve150 kernel: tap100i0: entered allmulticast mode
Aug 21 14:19:18 pve150 kernel: fwbr100i0: port 2(tap100i0) entered blocking state
Aug 21 14:19:18 pve150 kernel: fwbr100i0: port 2(tap100i0) entered forwarding state
Aug 21 14:19:19 pve150 pvedaemon[812698]: VM 100 started with PID 812713 using pve-qemu-kvm_11.0.3-2.
Aug 21 14:19:19 pve150 pvedaemon[780617]: <root@pam> end task UPID:pve150:000C669A:0185C1AD:6A884245:qmstart:100:root@pam: OK
Aug 21 14:19:19 pve150 pvedaemon[780617]: <root@pam> starting task UPID:pve150:000C66E7:0185C25F:6A884247:vncproxy:100:root@pam:
Aug 21 14:19:19 pve150 pvedaemon[812775]: starting vnc proxy UPID:pve150:000C66E7:0185C25F:6A884247:vncproxy:100:root@pam:
Aug 21 14:19:42 pve150 pveproxy[806325]: Clearing outdated entries from certificate cache
Aug 21 14:21:41 pve150 pvedaemon[780617]: <root@pam> successful auth for user 'root@pam'
Aug 21 14:22:16 pve150 pmxcfs[1300]: [dcdb] notice: data verification successful


NFS mount details
Code: 
root@pve150:/mnt/pve/nfs_ontap/images/100# fgrep nfs /proc/mounts
10.12.140.160:/nfs /mnt/pve/nfs_ontap nfs4 rw,relatime,vers=4.2,rsize=65536,wsize=65536,namlen=255,hard,fatal_neterrors=none,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.12.140.150,local_lock=none,addr=10.12.140.160 0 0
 

Attachments

Last edited:
I had the same error messages after updating to the latest patch on my 3-node cluster. It damaged GRUB and my partitions on multiple VMs. I had to do a restore because some VMs were production-critical. Thinking of reverting back to the old patch before something else breaks.

What could cause this?
 
TLDR;

The error messages below are triggered in an environment with VMs on a NetApp NFS share. Sparsefile support enabled ( nfs modify -v4.2-sparsefile-ops enabled) on that NFS share. It's specific to the incomplete implementation of sparsefile handling done by NetApp.

A simple way to check if you are affected: run strace and check that fallocate reports an EIO (Input/output error).

Partial deallocate is not implemented in NetApp ONTAP and returns an NFS4ERR_IO error. This error is passed from QEMU to the operating system, where it triggers the following error messages. The error occurs only when v4.2-sparsefile-ops is enabled in the NFS SVM.

If enabled, disable v4.2-sparsefile-ops again.

Error message
Code: 
[ 43.534144] I/O error, dev sda, sector 812584 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 2
[ 43.534248] EXT4-fs (sda1): failed to convert unwritten extents to written extents -- potential data loss! (inode 2391, error -5)

Details

I traced my system to find the root cause of the earlier reported error (I/O error, dev sda, sector 812584 op 0x1: (WRITE) / EXT4-fs (sda1): failed to convert unwritten extents to written extents):

Level 0A: Setup
  • PVE:
    • 9.2.10, up-to-date patched
    • Hostname: pve150
    • Qemu: 11.0.3-2
  • Storage:
    • NFS provided by NetApp ONTAP Simulator 9.18.1
    • NFS v4.2 with v4.2-sparsefile-ops enabled
  • Client:
    • Debian
    • Hostname: debian
    • installed from debian-13.3.0-amd64-netinst.iso
    • Disk: scsi0: nfs_ontap:100/vm-100-disk-0.qcow2,discard=on,iothread=1,size=4G,ssd=1

Level 0B: Datastore configuration on PVE
Code: 
root@pve150:~/nfstest# fgrep nfs /proc/mounts
10.12.140.160:/nfs /mnt/pve/nfs_ontap nfs4 rw,relatime,vers=4.2,rsize=65536,wsize=65536,namlen=255,hard,fatal_neterrors=none,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.12.140.150,local_lock=none,addr=10.12.140.160 0 0
root@pve150:~/nfstest# cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content images,iso,import,backup,vztmpl
        shared 0

zfspool: local-zfs
        pool rpool/data
        content rootdir,images
        sparse 1

nfs: nfs_ontap
        export /nfs
        path /mnt/pve/nfs_ontap
        server 10.12.140.160
        content images
        prune-backups keep-all=1

Level 0C: SVM configuration
Code: 
nacl::*> nfs show -fields v3,v4.0,v4.1,v4.2-sparsefile-ops,vstorage
vserver v3      v4.0    v4.1    vstorage v4.2-sparsefile-ops
------- ------- ------- ------- -------- -------------------
nfs     enabled enabled enabled disabled enabled

Level 1: dmesg -T
Code: 
[Mo Aug 24 09:26:11 2026] I/O error, dev sda, sector 985208 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2
[Mo Aug 24 09:26:11 2026] EXT4-fs warning (device sda1): ext4_end_bio:353: I/O error 10 writing to inode 2405 starting block 123151)
[Mo Aug 24 09:26:11 2026] sd 0:0:0:0: [sda] tag#53 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
[Mo Aug 24 09:26:11 2026] sd 0:0:0:0: [sda] tag#53 Sense Key : Aborted Command [current]
[Mo Aug 24 09:26:11 2026] sd 0:0:0:0: [sda] tag#53 Add. Sense: I/O process terminated
[Mo Aug 24 09:26:11 2026] sd 0:0:0:0: [sda] tag#53 CDB: Write(10) 2a 00 00 0f 09 f0 00 00 08 00
[Mo Aug 24 09:26:11 2026] I/O error, dev sda, sector 985584 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2
[Mo Aug 24 09:26:11 2026] EXT4-fs warning (device sda1): ext4_end_bio:353: I/O error 10 writing to inode 2405 starting block 123198)
[Mo Aug 24 09:26:11 2026] sd 0:0:0:0: [sda] tag#50 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
[Mo Aug 24 09:26:11 2026] sd 0:0:0:0: [sda] tag#50 Sense Key : Aborted Command [current]
[Mo Aug 24 09:26:11 2026] sd 0:0:0:0: [sda] tag#50 Add. Sense: I/O process terminated
[Mo Aug 24 09:26:11 2026] sd 0:0:0:0: [sda] tag#50 CDB: Write(10) 2a 00 00 0f 08 88 00 00 08 00
[Mo Aug 24 09:26:11 2026] I/O error, dev sda, sector 985224 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2
[Mo Aug 24 09:26:11 2026] EXT4-fs warning (device sda1): ext4_end_bio:353: I/O error 10 writing to inode 2405 starting block 123153)
[Mo Aug 24 09:26:11 2026] EXT4-fs (sda1): failed to convert unwritten extents to written extents -- potential data loss!  (inode 2405, error -5)
[Mo Aug 24 09:26:12 2026] EXT4-fs (sda1): failed to convert unwritten extents to written extents -- potential data loss!  (inode 2405, error -5)
[Mo Aug 24 09:26:12 2026] EXT4-fs (sda1): failed to convert unwritten extents to written extents -- potential data loss!  (inode 2405, error -5)

Level 2A: strace -ttf -p $(cat /run/qemu-server/100.pid) -e trace=fallocate
Code: 
[pid 1649064] 09:26:11.975921 fallocate(19, FALLOC_FL_ZERO_RANGE, 9043968, 65536) = -1 EIO (Input/output error)
[pid 1649076] 09:26:11.978035 fallocate(19, FALLOC_FL_ZERO_RANGE, 9043968, 65536) = -1 EIO (Input/output error)

Level 2B: socat -,ignoreeof UNIX-CONNECT:/run/qemu-server/100.qmp + {"execute":"qmp_capabilities"}
Code: 
{"timestamp": {"seconds": 1787556371, "microseconds": 975426}, "event": "BLOCK_IO_ERROR", "data": {"device": "", "nospace": false, "node-name": "drive-scsi0", "qom-path": "/machine/peripheral/scsi0", "reason": "Input/output error", "operation": "write", "action": "report"}}

Code: 
1787556371.975426 == 09:26:11.975

Level 3: tcpdump -i vmbr0 -n -B 196608 -s 0 -w nfstest_alloc_20260824_0910.pcap host 10.12.140.160
Code: 
No.        Time                Source            Destination        Protocol    Length    Info
45710    1787556371.976447    10.12.140.150    10.12.140.160    NFS            318        V4 Call (Reply In 45711) DEALLOCATE StateID: 0xe0c9 Offset: 9043968 Len: 65536 | ALLOCATE StateID: 0xe0c9 Offset: 9043968 Len: 65536
45711    1787556371.976547    10.12.140.160    10.12.140.150    NFS            166        V4 Reply (Call In 45710) DEALLOCATE Status: NFS4ERR_IO

Level 4: Support for DEALLOCATE on NetApp ONTAP

NetApp states in Does this v4.2-sparsefile-ops perform ALLOCATE/DEALLOCATE? : "ONTAP support full file deallocate, not partial deallocates"
Why a DEALLOCATE error surfaces as a guest write error

This is the part that makes the failure so confusing, and it comes down to a single error code.

QEMU's file backend maps fallocate() failures through translate_err() in block/file-posix.c. Only ENODEV, ENOSYS, EOPNOTSUPP and ENOTTY are converted to -ENOTSUP; EIO is passed through unchanged. This matches the observed behaviour: with EOPNOTSUPP the request is handled gracefully, with EIO the error reaches the guest.

Further tests

NFS v4.2 datastores and VMs w/o discard produce the same error.

NFS v4.2 datastores, v4.2-sparsefile-ops disabled and VMs with or without the discard option set, solves the issue. The NFS client reports now EOPNOTSUPP (Operation not supported) for fallocate (DEALLOCATE) Qemu handles the request well and the VM continue without any error message.

Code: 
[pid 1659810] 10:15:04.565374 fallocate(17, FALLOC_FL_ZERO_RANGE, 10682368, 65536) = -1 EOPNOTSUPP (Operation not supported)
[pid 1659810] 10:15:04.565557 fallocate(17, FALLOC_FL_KEEP_SIZE|FALLOC_FL_PUNCH_HOLE, 10682368, 65536) = -1 EOPNOTSUPP (Operation not supported)

Solution

Disable sparse-file operations on the SVM:

Code: 
vserver nfs modify -vserver <svm> -v4.2-sparsefile-ops disabled

or try to convert the qcow2 disk image into a raw format disk image.

Trade-off

The downside is that you lose hole punching. fstrim in the guest and discard=on no longer do anything to the size of the qcow2 file, so the images only ever grow. To get space back you have to shut the VM down and either run qemu-img convert or move the disk off to another storage and back again. Thin provisioning itself still works and the images are still sparse, you just can't reclaim anything while the VM is running.

Check your guests

The "potential data loss" warning is real: on affected extents you get either zeroes or stale data back, depending on your kernel. Run fsck -f, and check important data.

Open questions
  1. Should a failed DEALLOCATE be reported to the guest as a write error at all? QEMU's fallback path exists precisely for backends that cannot punch holes — it is only bypassed because the error arrives as EIO.
  2. Is NFS4ERR_IO the correct answer for an operation the server does not implement in this form? NFS4ERR_NOTSUPP would let every client handle this gracefully.
I would consider this primarily an ONTAP issue, but I would be interested in whether QEMU should be more defensive here as well.
 
Last edited:
  • Like
Reactions: waltar
JoMe said:
I had the same error messages after updating to the latest patch on my 3-node cluster. It damaged GRUB and my partitions on multiple VMs. I had to do a restore because some VMs were production-critical. Thinking of reverting back to the old patch before something else breaks.

What could cause this?
Click to expand...
Do you use a NetApp NFS share with sparsefiles enabled (like described in my TLDR)?
 
You must log in or register to reply here.
Top Bottom
Back