kvm: virtio: zero sized buffers are not allowed

davemcl

Member
Sep 24, 2022
109
12
18
I get this error before the Windows 2019 guest drive in question starts throwing these errors and the VM has to be reset.
On the guest the drive goes read only with:
Reset to device, \Device\RaidPort2, was issued.

The host either shows
QEMU[1093377]: kvm: virtio: zero sized buffers are not allowed
or
QEMU[2198]: kvm: Desc next is 4

PVE8 with all updates applied.

Code:
root@joi:~# strace -c -p $(cat /var/run/qemu-server/110.pid)
strace: Process 2136222 attached
^Cstrace: Process 2136222 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 95.24    0.083687          74      1127           ppoll
  2.93    0.002571           0      4080           write
  0.89    0.000786           0       998           recvmsg
  0.85    0.000751           0      1086           read
  0.04    0.000035           1        20           sendmsg
  0.02    0.000014           0        24           ioctl
  0.01    0.000010           2         4           close

Code:
gdb --batch --ex 't a a bt' -p $(cat /var/run/qemu-server/110.pid)
[New LWP 2136223]
[New LWP 2136224]
[New LWP 2136225]
[New LWP 2136265]
[New LWP 2136266]
[New LWP 2136267]
[New LWP 2136268]
[New LWP 2136269]
[New LWP 2136270]
[New LWP 2136271]
[New LWP 2136272]
[New LWP 2136283]
[New LWP 2136284]
[New LWP 893472]
[New LWP 939640]
[New LWP 946158]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f1e9819c136 in __ppoll (fds=0x561574101400, nfds=75, timeout=<optimized out>, timeout@entry=0x7fff295443c0, sigmask=sigmask@entry=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:42
42      ../sysdeps/unix/sysv/linux/ppoll.c: No such file or directory.

Thread 17 (Thread 0x7f1e87fff6c0 (LWP 946158) "iou-wrk-2136224"):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

Thread 16 (Thread 0x7f1e87fff6c0 (LWP 939640) "iou-wrk-2136224"):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

Thread 15 (Thread 0x7f1e877fe6c0 (LWP 893472) "iou-wrk-2136225"):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

Thread 14 (Thread 0x7f1b551bf6c0 (LWP 2136284) "vnc_worker"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x56157319a5a8) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x56157319a5a8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x00007f1e98125e0b in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x56157319a5a8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f1e98128468 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x56157319a5b8, cond=0x56157319a580) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x56157319a580, mutex=mutex@entry=0x56157319a5b8) at ./nptl/pthread_cond_wait.c:618
#5  0x0000561571b0c6fb in qemu_cond_wait_impl (cond=0x56157319a580, mutex=0x56157319a5b8, file=0x561571b99bb4 "../ui/vnc-jobs.c", line=248) at ../util/qemu-thread-posix.c:225
#6  0x0000561571572fdd in vnc_worker_thread_loop (queue=queue@entry=0x56157319a580) at ../ui/vnc-jobs.c:248
#7  0x0000561571573ce8 in vnc_worker_thread (arg=arg@entry=0x56157319a580) at ../ui/vnc-jobs.c:361
#8  0x0000561571b0bbe8 in qemu_thread_start (args=0x56157319a610) at ../util/qemu-thread-posix.c:541
#9  0x00007f1e98129044 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007f1e981a95fc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 13 (Thread 0x7f1e8d2282c0 (LWP 2136283) "iou-wrk-2136222"):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

Thread 12 (Thread 0x7f1b751ff6c0 (LWP 2136272) "CPU 7/KVM"):
#0  __GI___ioctl (fd=40, request=request@entry=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1  0x000056157197855f in kvm_vcpu_ioctl (cpu=cpu@entry=0x561572d51cc0, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3127
#2  0x00005615719786b5 in kvm_cpu_exec (cpu=cpu@entry=0x561572d51cc0) at ../accel/kvm/kvm-all.c:2939
#3  0x0000561571979cfd in kvm_vcpu_thread_fn (arg=arg@entry=0x561572d51cc0) at ../accel/kvm/kvm-accel-ops.c:51
#4  0x0000561571b0bbe8 in qemu_thread_start (args=0x561572d5a9d0) at ../util/qemu-thread-posix.c:541
#5  0x00007f1e98129044 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007f1e981a95fc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 11 (Thread 0x7f1b75dff6c0 (LWP 2136271) "CPU 6/KVM"):
#0  __GI___ioctl (fd=39, request=request@entry=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1  0x000056157197855f in kvm_vcpu_ioctl (cpu=cpu@entry=0x561572d48610, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3127
#2  0x00005615719786b5 in kvm_cpu_exec (cpu=cpu@entry=0x561572d48610) at ../accel/kvm/kvm-all.c:2939
#3  0x0000561571979cfd in kvm_vcpu_thread_fn (arg=arg@entry=0x561572d48610) at ../accel/kvm/kvm-accel-ops.c:51
#4  0x0000561571b0bbe8 in qemu_thread_start (args=0x561572d51320) at ../util/qemu-thread-posix.c:541
#5  0x00007f1e98129044 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007f1e981a95fc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 10 (Thread 0x7f1b769ff6c0 (LWP 2136270) "CPU 5/KVM"):
#0  __GI___ioctl (fd=38, request=request@entry=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1  0x000056157197855f in kvm_vcpu_ioctl (cpu=cpu@entry=0x561572d3ef60, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3127
#2  0x00005615719786b5 in kvm_cpu_exec (cpu=cpu@entry=0x561572d3ef60) at ../accel/kvm/kvm-all.c:2939
#3  0x0000561571979cfd in kvm_vcpu_thread_fn (arg=arg@entry=0x561572d3ef60) at ../accel/kvm/kvm-accel-ops.c:51
#4  0x0000561571b0bbe8 in qemu_thread_start (args=0x561572d47c70) at ../util/qemu-thread-posix.c:541
#5  0x00007f1e98129044 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007f1e981a95fc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 9 (Thread 0x7f1b775fe6c0 (LWP 2136269) "CPU 4/KVM"):
#0  __GI___ioctl (fd=37, request=request@entry=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1  0x000056157197855f in kvm_vcpu_ioctl (cpu=cpu@entry=0x561572d35540, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3127
#2  0x00005615719786b5 in kvm_cpu_exec (cpu=cpu@entry=0x561572d35540) at ../accel/kvm/kvm-all.c:2939
#3  0x0000561571979cfd in kvm_vcpu_thread_fn (arg=arg@entry=0x561572d35540) at ../accel/kvm/kvm-accel-ops.c:51
#4  0x0000561571b0bbe8 in qemu_thread_start (args=0x561572d3e5c0) at ../util/qemu-thread-posix.c:541
#5  0x00007f1e98129044 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007f1e981a95fc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 8 (Thread 0x7f1b77dff6c0 (LWP 2136268) "CPU 3/KVM"):
#0  __GI___ioctl (fd=36, request=request@entry=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1  0x000056157197855f in kvm_vcpu_ioctl (cpu=cpu@entry=0x561572d2c060, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3127
#2  0x00005615719786b5 in kvm_cpu_exec (cpu=cpu@entry=0x561572d2c060) at ../accel/kvm/kvm-all.c:2939
#3  0x0000561571979cfd in kvm_vcpu_thread_fn (arg=arg@entry=0x561572d2c060) at ../accel/kvm/kvm-accel-ops.c:51
#4  0x0000561571b0bbe8 in qemu_thread_start (args=0x561572d34bf0) at ../util/qemu-thread-posix.c:541
#5  0x00007f1e98129044 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007f1e981a95fc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 7 (Thread 0x7f1e851ff6c0 (LWP 2136267) "CPU 2/KVM"):
#0  __GI___ioctl (fd=35, request=request@entry=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1  0x000056157197855f in kvm_vcpu_ioctl (cpu=cpu@entry=0x561572d22c70, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3127
#2  0x00005615719786b5 in kvm_cpu_exec (cpu=cpu@entry=0x561572d22c70) at ../accel/kvm/kvm-all.c:2939
#3  0x0000561571979cfd in kvm_vcpu_thread_fn (arg=arg@entry=0x561572d22c70) at ../accel/kvm/kvm-accel-ops.c:51
#4  0x0000561571b0bbe8 in qemu_thread_start (args=0x561572d2b6c0) at ../util/qemu-thread-posix.c:541
#5  0x00007f1e98129044 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007f1e981a95fc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 6 (Thread 0x7f1e85dff6c0 (LWP 2136266) "CPU 1/KVM"):
#0  __GI___ioctl (fd=34, request=request@entry=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1  0x000056157197855f in kvm_vcpu_ioctl (cpu=cpu@entry=0x561572d19760, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3127
#2  0x00005615719786b5 in kvm_cpu_exec (cpu=cpu@entry=0x561572d19760) at ../accel/kvm/kvm-all.c:2939
#3  0x0000561571979cfd in kvm_vcpu_thread_fn (arg=arg@entry=0x561572d19760) at ../accel/kvm/kvm-accel-ops.c:51
#4  0x0000561571b0bbe8 in qemu_thread_start (args=0x561572d222d0) at ../util/qemu-thread-posix.c:541
#5  0x00007f1e98129044 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007f1e981a95fc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 5 (Thread 0x7f1e86ffd6c0 (LWP 2136265) "CPU 0/KVM"):
#0  __GI___ioctl (fd=33, request=request@entry=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1  0x000056157197855f in kvm_vcpu_ioctl (cpu=cpu@entry=0x561572ce98d0, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3127
#2  0x00005615719786b5 in kvm_cpu_exec (cpu=cpu@entry=0x561572ce98d0) at ../accel/kvm/kvm-all.c:2939
#3  0x0000561571979cfd in kvm_vcpu_thread_fn (arg=arg@entry=0x561572ce98d0) at ../accel/kvm/kvm-accel-ops.c:51
#4  0x0000561571b0bbe8 in qemu_thread_start (args=0x5615728ab7c0) at ../util/qemu-thread-posix.c:541
#5  0x00007f1e98129044 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007f1e981a95fc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 4 (Thread 0x7f1e877fe6c0 (LWP 2136225) "kvm"):
#0  0x00007f1e9819c136 in __ppoll (fds=0x7f1e78003590, nfds=12, timeout=<optimized out>, timeout@entry=0x0, sigmask=sigmask@entry=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:42
#1  0x0000561571b21c45 in ppoll (__ss=0x0, __timeout=0x0, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/poll2.h:64
#2  0x0000561571b09889 in fdmon_poll_wait (ctx=0x561572b15390, ready_list=0x7f1e877f9248, timeout=-1) at ../util/fdmon-poll.c:80
#3  0x0000561571b08d2d in aio_poll (ctx=0x561572b15390, blocking=blocking@entry=true) at ../util/aio-posix.c:680
#4  0x00005615719bb176 in iothread_run (opaque=opaque@entry=0x561572b14740) at ../iothread.c:63
#5  0x0000561571b0bbe8 in qemu_thread_start (args=0x561572b16fa0) at ../util/qemu-thread-posix.c:541
#6  0x00007f1e98129044 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#7  0x00007f1e981a95fc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 3 (Thread 0x7f1e87fff6c0 (LWP 2136224) "kvm"):
#0  0x00007f1e9819c136 in __ppoll (fds=0x7f1e80003160, nfds=12, timeout=<optimized out>, timeout@entry=0x0, sigmask=sigmask@entry=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:42
#1  0x0000561571b21c45 in ppoll (__ss=0x0, __timeout=0x0, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/poll2.h:64
#2  0x0000561571b09889 in fdmon_poll_wait (ctx=0x561572b149f0, ready_list=0x7f1e87ffa248, timeout=-1) at ../util/fdmon-poll.c:80
#3  0x0000561571b08d2d in aio_poll (ctx=0x561572b149f0, blocking=blocking@entry=true) at ../util/aio-posix.c:680
#4  0x00005615719bb176 in iothread_run (opaque=opaque@entry=0x56157295cf00) at ../iothread.c:63
#5  0x0000561571b0bbe8 in qemu_thread_start (args=0x561572b15040) at ../util/qemu-thread-posix.c:541
#6  0x00007f1e98129044 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#7  0x00007f1e981a95fc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7f1e8d0ba6c0 (LWP 2136223) "call_rcu"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000561571b0cd6a in qemu_futex_wait (val=<optimized out>, f=<optimized out>) at ./include/qemu/futex.h:29
#2  qemu_event_wait (ev=ev@entry=0x56157241cce8 <rcu_call_ready_event>) at ../util/qemu-thread-posix.c:464
#3  0x0000561571b165c2 in call_rcu_thread (opaque=opaque@entry=0x0) at ../util/rcu.c:261
#4  0x0000561571b0bbe8 in qemu_thread_start (args=0x5615728b34a0) at ../util/qemu-thread-posix.c:541
#5  0x00007f1e98129044 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007f1e981a95fc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7f1e8d2282c0 (LWP 2136222) "kvm"):
#0  0x00007f1e9819c136 in __ppoll (fds=0x561574101400, nfds=75, timeout=<optimized out>, timeout@entry=0x7fff295443c0, sigmask=sigmask@entry=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:42
#1  0x0000561571b21bee in ppoll (__ss=0x0, __timeout=0x7fff295443c0, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/poll2.h:64
#2  qemu_poll_ns (fds=<optimized out>, nfds=<optimized out>, timeout=timeout@entry=1188659764) at ../util/qemu-timer.c:351
#3  0x0000561571b1f4ee in os_host_main_loop_wait (timeout=1188659764) at ../util/main-loop.c:308
#4  main_loop_wait (nonblocking=nonblocking@entry=0) at ../util/main-loop.c:592
#5  0x000056157173baf7 in qemu_main_loop () at ../softmmu/runstate.c:731
#6  0x0000561571982a46 in qemu_default_main () at ../softmmu/main.c:37
#7  0x00007f1e980c71ca in __libc_start_call_main (main=main@entry=0x561571548390 <main>, argc=argc@entry=82, argv=argv@entry=0x7fff295445d8) at ../sysdeps/nptl/libc_start_call_main.h:58
#8  0x00007f1e980c7285 in __libc_start_main_impl (main=0x561571548390 <main>, argc=82, argv=0x7fff295445d8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff295445c8) at ../csu/libc-start.c:360
#9  0x0000561571549e71 in _start ()
[Inferior 1 (process 2136222) detached]
 
could you please post

- pveversion -v
- VM config file contents
- /etc/pve/storage.cfg

thanks!
 
Code:
root@joi:~# pveversion -v
proxmox-ve: 8.0.1 (running kernel: 6.2.16-4-pve)
pve-manager: 8.0.3 (running version: 8.0.3/bbf3993334bfa916)
pve-kernel-6.2: 8.0.3
pve-kernel-6.2.16-4-pve: 6.2.16-5
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-3
libknet1: 1.25-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.0
libpve-access-control: 8.0.3
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.6
libpve-guest-common-perl: 5.0.3
libpve-http-server-perl: 5.0.4
libpve-rs-perl: 0.8.4
libpve-storage-perl: 8.0.2
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
openvswitch-switch: 3.1.0-2
proxmox-backup-client: 3.0.1-1
proxmox-backup-file-restore: 3.0.1-1
proxmox-kernel-helper: 8.0.2
proxmox-mail-forward: 0.2.0
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.2
proxmox-widget-toolkit: 4.0.6
pve-cluster: 8.0.2
pve-container: 5.0.4
pve-docs: 8.0.4
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.3
pve-firmware: 3.7-1
pve-ha-manager: 4.0.2
pve-i18n: 3.0.5
pve-qemu-kvm: 8.0.2-3
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.6
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.1.12-pve1

Code:
root@joi:~# qm config 110
agent: 1
bios: ovmf
boot: order=scsi0;net0;ide0
cores: 8
cpu: host
efidisk0: tank:vm-110-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
ide0: none,media=cdrom
machine: pc-q35-8.0
memory: 12288
meta: creation-qemu=8.0.2,ctime=1688721840
name: Win-Srv-19
net0: virtio=A2:B2:0F:D9:1D:6C,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
parent: sql_installed
scsi0: tank:vm-110-disk-2,discard=on,iothread=1,size=48G
scsi1: tank:vm-110-disk-1,discard=on,iothread=1,size=112G
scsihw: virtio-scsi-single
smbios1: uuid=dd6e90d0-88c2-4c84-bb67-ec121d808825
sockets: 1
tags: windows
vmgenid: 44a59351-b0ed-492f-b465-7bcddd41911a

Code:
cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content iso,vztmpl,backup

zfspool: local-zfs
        pool rpool/data
        content rootdir,images
        sparse 1

cifs: juno
        path /mnt/pve/juno
        server 172.16.16.6
        share backup
        content snippets,backup
        prune-backups keep-all=1
        username xcp-ng

dir: backup-nas
        path /mnt/pve/juno/proxmox
        content snippets,backup
        prune-backups keep-all=1
        shared 0

zfspool: tank
        pool tank
        content images,rootdir
        mountpoint /tank
        nodes joi

Thanks Fabian.
 
I get this on multiple servers running MS SQL workloads - above is from a Ryzen setup here at home. It also occurs on Dell R640 & R740 all with enterprise SSD storage (local).
The errors are usually one of these

Code:
QEMU[xxxx]: kvm: virtio: zero sized buffers are not allowed
QEMU[xxxx]: kvm: Desc next is 3
I also have seen one time
QEMU[2576028]: kvm: Incorrect order for descriptors
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!