[SOLVED] unable to create an Linux VM with VirtIO drive since update

FYI, we had some people in the German sub-forum pile up, and they had issues with using SATA as VM disk bus under the 5.13 kernel (mostly for Windows VMs, but that could be correlation only). After a bit of investigation it turned out that it could be worked around by setting the Async IO to mode to threads (for cache = write back/through) or native (for cache = off, none or direct sync), respectively.

So I know you do not use SATA, but still, could you try out changing the Async IO mode (visible at the bottom right of your screenshot).
https://forum.proxmox.com/threads/proxmox-ve-7-1-released.99847/page-3#post-431438

JOduMonT said:
also I would like to stress again the fact that when I'm changing virtio to scsi it is at the hard drive bus/device level not at the SCSI Controller level.
Click to expand...
Never doubted that, just wanted to tell you that SCSI may be the better choice anyway, so if it works there wouldn't be any harm done ;-)
 
  • Like
Reactions: cyonjan
Hello, the same issues at my side .... With virtio disks Ubuntu/Mint distros are failing at fresh install (one way or another) ... With IDE devices no issues at all ....
No matter Enterprise or No-Subscription .... Prox 7.1 latest update ....
 
tonci said:
With virtio disks Ubuntu/Mint distros are failing at fresh install (one way or another)
Click to expand...
Did you tried:
t.lamprecht said:
FYI, we had some people in the German sub-forum pile up, and they had issues with using SATA as VM disk bus under the 5.13 kernel (mostly for Windows VMs, but that could be correlation only). After a bit of investigation it turned out that it could be worked around by setting the Async IO to mode to threads (for cache = write back/through) or native (for cache = off, none or direct sync), respectively.
Click to expand...
 
Hi,
I have the same problems since upgrading to Proxmox with qemu 6.1. I have tried changing the Async IO settings as advised above but with no luck. (I'm running with cache off - the default, the VirtIO SCSI controller and a mix of VirtIO and SCSI block devices) The problem was still the same without any changes. (I have rebooted VMs and the host to be sure) I ended up downgrading the qemu package to 6.0. Downgrading has resolved this issue for me.

Bash: 
$ apt install pve-qemu-kvm=6.0.0-4

I am running Proxmox VE 7.1-5 (bullseye) with pve-no-subscription. I have everything up to date except for the now downgraded pve-qemu-kvm package.

Bash: 
$ uname -a
Linux xxxxxx 5.13.19-1-pve #1 SMP PVE 5.13.19-2 (Tue, 09 Nov 2021 12:59:38 +0100) x86_64 GNU/Linux
 
  • Like
Reactions: Rastta
These new linux VMs are not critical at all and IDE diska are acceptable too, but I'm just wandering whether we have to worry about actual running linux/win VMs ? ... they seem to be stable so far (?!) ... Is any data-corruption possible while using virtio disks at the moment (and in this situation) ?
 
@tonci

Yes, I was experiencing data corruption while using the new 6.1 qemu. For example downloaded files were randomly broken, installing packages randomly resulted in segfaulting apps or not running at all. After downgrading all seems to be working fine. This affected both long running VMs and freshly created ones alike.

dmesg from latest Debian 11 on top of latest Proxmox VE 7.1-5, VirtIO SCSI controller with VirtIO block device, no cache, Async IO default (later I've tried native, same results)

Code: 
[ 1458.036434] blk_update_request: I/O error, dev vda, sector 10060520 op 0x1:(WRITE) flags 0x800 phys_seg 82 prio class 0
[ 1458.037672] EXT4-fs warning: 5 callbacks suppressed
[ 1458.037674] EXT4-fs warning (device vda1): ext4_end_bio:345: I/O error 10 writing to inode 403754 starting block 1257728)
[ 1458.038869] blk_update_request: I/O error, dev vda, sector 10061824 op 0x1:(WRITE) flags 0x4800 phys_seg 160 prio class 0
[ 1458.039440] blk_update_request: I/O error, dev vda, sector 10064384 op 0x1:(WRITE) flags 0x800 phys_seg 97 prio class 0
[ 1458.040031] EXT4-fs warning (device vda1): ext4_end_bio:345: I/O error 10 writing to inode 403754 starting block 1258241)
[ 1458.040602] blk_update_request: I/O error, dev vda, sector 10065928 op 0x1:(WRITE) flags 0x4800 phys_seg 156 prio class 0
[ 1458.041191] blk_update_request: I/O error, dev vda, sector 10068488 op 0x1:(WRITE) flags 0x800 phys_seg 100 prio class 0
[ 1458.041774] EXT4-fs warning (device vda1): ext4_end_bio:345: I/O error 10 writing to inode 403754 starting block 1258761)
[ 1458.042353] blk_update_request: I/O error, dev vda, sector 10070088 op 0x1:(WRITE) flags 0x4800 phys_seg 160 prio class 0
[ 1458.042938] blk_update_request: I/O error, dev vda, sector 10072648 op 0x1:(WRITE) flags 0x800 phys_seg 96 prio class 0
[ 1458.043537] EXT4-fs warning (device vda1): ext4_end_bio:345: I/O error 10 writing to inode 403754 starting block 1259273)
[ 1458.044125] blk_update_request: I/O error, dev vda, sector 10074184 op 0x1:(WRITE) flags 0x4800 phys_seg 158 prio class 0
[ 1458.044747] blk_update_request: I/O error, dev vda, sector 10076744 op 0x1:(WRITE) flags 0x800 phys_seg 92 prio class 0
[ 1458.045312] EXT4-fs warning (device vda1): ext4_end_bio:345: I/O error 10 writing to inode 403754 starting block 1259776)
[ 1458.045939] blk_update_request: I/O error, dev vda, sector 10078208 op 0x1:(WRITE) flags 0x4800 phys_seg 175 prio class 0
[ 1458.046529] EXT4-fs warning (device vda1): ext4_end_bio:345: I/O error 10 writing to inode 403754 starting block 1257591)
[ 1458.047497] buffer_io_error: 13869 callbacks suppressed
[ 1458.047500] Buffer I/O error on device vda1, logical block 1259017
[ 1458.048914] Buffer I/O error on device vda1, logical block 1259018
[ 1458.049587] Buffer I/O error on device vda1, logical block 1259019
[ 1458.049602] EXT4-fs warning (device vda1): ext4_end_bio:345: I/O error 10 writing to inode 403754 starting block 1260385)
[ 1458.050159] Buffer I/O error on device vda1, logical block 1259020
[ 1458.051522] Buffer I/O error on device vda1, logical block 1259021
[ 1458.052133] Buffer I/O error on device vda1, logical block 1259022
[ 1458.052723] Buffer I/O error on device vda1, logical block 1259023
[ 1458.053301] Buffer I/O error on device vda1, logical block 1259024
[ 1458.053934] Buffer I/O error on device vda1, logical block 1259025
[ 1458.054510] Buffer I/O error on device vda1, logical block 1259026
[ 1458.064017] EXT4-fs warning (device vda1): ext4_end_bio:345: I/O error 10 writing to inode 403754 starting block 1265920)
[ 1458.064875] EXT4-fs warning (device vda1): ext4_end_bio:345: I/O error 10 writing to inode 403754 starting block 1266328)
[ 1458.065456] EXT4-fs warning (device vda1): ext4_end_bio:345: I/O error 10 writing to inode 403754 starting block 1266842)
 
Last edited:
FWIW - I'm also noticing random data corruption since upgrading to 7.1 with qemu 6.2, with Virtio disks. I've tried changing the disks to (VirtIO) SCSI and the problem also seems to persist - random EXT4 IO errors inside the machine, leading to data corruption.
Guests are Arch/NixOS Stable with recent-ish kernels, if it helps.

I have rolled back qemu to 6.0 as mentioned previously and it seems to be back fine.
Also learned the hard lesson about backups with one of those VMS, and also a PSA: don't run fsck inside a VM in such state as IO access is still corrupt which will destroy the filesystems even more.
 
Ok, it seems we've got stuck here :1637480004146.png
Now (with 6.0.0-4 version) error messages such as above ones do not appear any more ...

Waiting for some official response about this issue ...

Thanks in advance

BR

Tonci
 
There's a new kernel package available on the pvetest repository, namely pve-kernel-5.13.19-1-pve with package version 5.13.19-3

It includes some fixes that addresses issues with similar sounding symptoms as described in this thread here, correlating to using io_uring.

We had two reproducers that triggered quite effectively, one was using SATA as disk bus for a Windows VM (froze completely), and another reproducer was an ansible playbook that setup a nested PVE cluster for some internal testing, both work now Ok with that kernel. If you run into IO errors in kernel log or frozen Windows VMs I'd recommend trying that kernel version.
 
updated to 7.1.6 ... but pve-qemu-kvm is still 6.1.0-2 and with this version still have error like this (ubuntu fresh install virtio)
1638273054945.png

downgrading to 6.0.0-4 solved this out ....

Now I'm afraid upgrading from 6 to 7 ... Did that only once at one client and had some wierd problem with zfs-replication ... I will test little bit more and send you some logs

BR
Tonci
 
@t.lamprecht

today i installed i new proxmox from iso 7-1, I had to downgrade my kernel to 5.11.22-6 otherwize my pci-passthrough didnt work
yes my english is bad, no I dont have log, but im very annoyed since with 5.13 proxmox dont even reboot by itself.
 
Last edited:
again; a new update which dont fix the situation
With kernels 5.13.19-1-pve and 5.13.19-2-pve the situation remains

I'm still stuck with the kernel 5.11.22-6-pve

otherwise my any VM with PCI Passthrough won't boot, such as a VM Ubuntu 20.04LTS which exhibit the message: unable to read tail (got 0 bytes) and a non rebootable server (proxmox system) where I have to do a hard reset to reboot.

since the beginning of that issue, I reinstall from scratch proxmox with the latest version, to eliminate possible issue with the upgrade and miss configuration made with time; I also changed the drives where proxmox is installed and have a new NVM for hosting the pool where the VM are.

what proxmox team propose ?
 
JOduMonT said:
since the beginning of that issue, I reinstall from scratch proxmox with the latest version, to eliminate possible issue with the upgrade and miss configuration made with time; I also changed the drives where proxmox is installed and have a new NVM for hosting the pool where the VM are.
Click to expand...
How can you say that you eliminated a bad configuration under 5.13 if you never got it to work there?
Did you try the 5.15 based kernel we provide?
https://forum.proxmox.com/threads/opt-in-linux-kernel-5-15-for-proxmox-ve-7-x-available.100936/

JOduMonT said:
I means how I could reinstall an old kernel without being auto remove by your script ???
Click to expand...
The auto-removal script does not remove the following kernels automatically
- the currently booted version, if still installed
- the kernel version we've been called for
- the latest kernel version (as determined by Debian version number)
- the second-latest kernel version
- the latest kernel version of each series (e.g. 4.13, 4.15, 5.0) by marking the meta-packages

So all sensible cases are covered, if those wouldn't get auto-removed the boot partition would run out of space sooner or later..

If you want to keep a specific kernel I'd suggest to start actually reading the docs first before complaining here...
https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysboot_proxmox_boot_setup

If you cannot do that and just cannot help yourself there's always enterprise support:
https://www.proxmox.com/proxmox-ve/pricing
 
Last edited:
t.lamprecht said:
How can you say that you eliminated a bad configuration under 5.13 if you never got it to work there?
Click to expand...
what I means, is I reinstall proxmox, to be sure the situation was not cause by a modification (bad configuration) from me such as removing and/or stopping a services and/or editing a file in /etc, ...

t.lamprecht said:
Did you try the 5.15 based kernel we provide?
Click to expand...
yes, when i reinstall proxmox 7.1, I retrograde one by one from the original kernel (if I remember well 5.13.17 to 5.11.22-60 which is the only one working; the issue started at 5.11.22-7-pve, so if you means if I tried 5.11.15, yes I tried this kernel.
 
Last edited:
t.lamprecht said:
Did you try the 5.15 based kernel we provide?
Click to expand...
yes I tried, but the kernel 5.15 is not available in the boot partition

Code: 
root@pve:~# ls /boot/
config-5.11.22-6-pve      initrd.img-5.11.22-7-pve  System.map-5.13.19-2-pve
config-5.11.22-7-pve      initrd.img-5.13.19-1-pve  vmlinuz-5.11.22-6-pve
config-5.13.19-1-pve      initrd.img-5.13.19-2-pve  vmlinuz-5.11.22-7-pve
config-5.13.19-2-pve      pve/                      vmlinuz-5.13.19-1-pve
efi/                      System.map-5.11.22-6-pve  vmlinuz-5.13.19-2-pve
grub/                     System.map-5.11.22-7-pve 
initrd.img-5.11.22-6-pve  System.map-5.13.19-1-pve 
root@pve:~# ls /boot/^C
root@pve:~# kernel list^C
root@pve:~# proxmox-boot-tool kernel add 5.11.22-7^Cve
root@pve:~# apt install pve-headers-5.15.5-1-pve pve-headers-5.15.5-1-pve
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
pve-headers-5.15.5-1-pve is already the newest version (5.15.5-1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
root@pve:~# proxmox-boot-tool refresh^C
root@pve:~# proxmox-boot-tool kernel add 5.15.5-1-pve
E: no kernel image found in /boot for '5.15.5-1-pve', not adding.
root@pve:~#
 
ok I got it;

everything works as expected with 5.15
if I install it like this: apt install pve-headers-5.15 pve-kernel-5.15

sorry to being I pain in your b***
but over time I lost hope and while I taught apt upgrade would automatically upgrade to the last kernel
 
JOduMonT said:
but over time I lost hope and while I taught apt upgrade would automatically upgrade to the last kernel
Click to expand...
the 5.15 kernel is at the moment optional, normally when you apt update && apt dist-upgrade (never run apt upgrade!), you should get our newest non-testing kernel
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom