[SOLVED] unable to create an Linux VM with VirtIO drive since update

[t.lamprecht]

FYI, we had some people in the German sub-forum pile up, and they had issues with using SATA as VM disk bus under the 5.13 kernel (mostly for Windows VMs, but that could be correlation only). After a bit of investigation it turned out that it could be worked around by setting the Async IO to mode to threads (for cache = write back/through) or native (for cache = off, none or direct sync), respectively.

So I know you do not use SATA, but still, could you try out changing the Async IO mode (visible at the bottom right of your screenshot).
https://forum.proxmox.com/threads/proxmox-ve-7-1-released.99847/page-3#post-431438

also I would like to stress again the fact that when I'm changing virtio to scsi it is at the hard drive bus/device level not at the SCSI Controller level.

Never doubted that, just wanted to tell you that SCSI may be the better choice anyway, so if it works there wouldn't be any harm done ;-)

 

[tonci]

Hello, the same issues at my side .... With virtio disks Ubuntu/Mint distros are failing at fresh install (one way or another) ... With IDE devices no issues at all ....
No matter Enterprise or No-Subscription .... Prox 7.1 latest update ....

 

[t.lamprecht]

With virtio disks Ubuntu/Mint distros are failing at fresh install (one way or another)

Did you tried:

FYI, we had some people in the German sub-forum pile up, and they had issues with using SATA as VM disk bus under the 5.13 kernel (mostly for Windows VMs, but that could be correlation only). After a bit of investigation it turned out that it could be worked around by setting the Async IO to mode to threads (for cache = write back/through) or native (for cache = off, none or direct sync), respectively.

 

[dgc8lat0k4mq]

Hi,
I have the same problems since upgrading to Proxmox with qemu 6.1. I have tried changing the Async IO settings as advised above but with no luck. (I'm running with cache off - the default, the VirtIO SCSI controller and a mix of VirtIO and SCSI block devices) The problem was still the same without any changes. (I have rebooted VMs and the host to be sure) I ended up downgrading the qemu package to 6.0. Downgrading has resolved this issue for me.

$ apt install pve-qemu-kvm=6.0.0-4

I am running Proxmox VE 7.1-5 (bullseye) with pve-no-subscription. I have everything up to date except for the now downgraded pve-qemu-kvm package.

$ uname -a
Linux xxxxxx 5.13.19-1-pve #1 SMP PVE 5.13.19-2 (Tue, 09 Nov 2021 12:59:38 +0100) x86_64 GNU/Linux

 

[tonci]

These new linux VMs are not critical at all and IDE diska are acceptable too, but I'm just wandering whether we have to worry about actual running linux/win VMs ? ... they seem to be stable so far (?!) ... Is any data-corruption possible while using virtio disks at the moment (and in this situation) ?

 

[dgc8lat0k4mq]

@tonci

Yes, I was experiencing data corruption while using the new 6.1 qemu. For example downloaded files were randomly broken, installing packages randomly resulted in segfaulting apps or not running at all. After downgrading all seems to be working fine. This affected both long running VMs and freshly created ones alike.

dmesg from latest Debian 11 on top of latest Proxmox VE 7.1-5, VirtIO SCSI controller with VirtIO block device, no cache, Async IO default (later I've tried native, same results)

[ 1458.036434] blk_update_request: I/O error, dev vda, sector 10060520 op 0x1:(WRITE) flags 0x800 phys_seg 82 prio class 0
[ 1458.037672] EXT4-fs warning: 5 callbacks suppressed
[ 1458.037674] EXT4-fs warning (device vda1): ext4_end_bio:345: I/O error 10 writing to inode 403754 starting block 1257728)
[ 1458.038869] blk_update_request: I/O error, dev vda, sector 10061824 op 0x1:(WRITE) flags 0x4800 phys_seg 160 prio class 0
[ 1458.039440] blk_update_request: I/O error, dev vda, sector 10064384 op 0x1:(WRITE) flags 0x800 phys_seg 97 prio class 0
[ 1458.040031] EXT4-fs warning (device vda1): ext4_end_bio:345: I/O error 10 writing to inode 403754 starting block 1258241)
[ 1458.040602] blk_update_request: I/O error, dev vda, sector 10065928 op 0x1:(WRITE) flags 0x4800 phys_seg 156 prio class 0
[ 1458.041191] blk_update_request: I/O error, dev vda, sector 10068488 op 0x1:(WRITE) flags 0x800 phys_seg 100 prio class 0
[ 1458.041774] EXT4-fs warning (device vda1): ext4_end_bio:345: I/O error 10 writing to inode 403754 starting block 1258761)
[ 1458.042353] blk_update_request: I/O error, dev vda, sector 10070088 op 0x1:(WRITE) flags 0x4800 phys_seg 160 prio class 0
[ 1458.042938] blk_update_request: I/O error, dev vda, sector 10072648 op 0x1:(WRITE) flags 0x800 phys_seg 96 prio class 0
[ 1458.043537] EXT4-fs warning (device vda1): ext4_end_bio:345: I/O error 10 writing to inode 403754 starting block 1259273)
[ 1458.044125] blk_update_request: I/O error, dev vda, sector 10074184 op 0x1:(WRITE) flags 0x4800 phys_seg 158 prio class 0
[ 1458.044747] blk_update_request: I/O error, dev vda, sector 10076744 op 0x1:(WRITE) flags 0x800 phys_seg 92 prio class 0
[ 1458.045312] EXT4-fs warning (device vda1): ext4_end_bio:345: I/O error 10 writing to inode 403754 starting block 1259776)
[ 1458.045939] blk_update_request: I/O error, dev vda, sector 10078208 op 0x1:(WRITE) flags 0x4800 phys_seg 175 prio class 0
[ 1458.046529] EXT4-fs warning (device vda1): ext4_end_bio:345: I/O error 10 writing to inode 403754 starting block 1257591)
[ 1458.047497] buffer_io_error: 13869 callbacks suppressed
[ 1458.047500] Buffer I/O error on device vda1, logical block 1259017
[ 1458.048914] Buffer I/O error on device vda1, logical block 1259018
[ 1458.049587] Buffer I/O error on device vda1, logical block 1259019
[ 1458.049602] EXT4-fs warning (device vda1): ext4_end_bio:345: I/O error 10 writing to inode 403754 starting block 1260385)
[ 1458.050159] Buffer I/O error on device vda1, logical block 1259020
[ 1458.051522] Buffer I/O error on device vda1, logical block 1259021
[ 1458.052133] Buffer I/O error on device vda1, logical block 1259022
[ 1458.052723] Buffer I/O error on device vda1, logical block 1259023
[ 1458.053301] Buffer I/O error on device vda1, logical block 1259024
[ 1458.053934] Buffer I/O error on device vda1, logical block 1259025
[ 1458.054510] Buffer I/O error on device vda1, logical block 1259026
[ 1458.064017] EXT4-fs warning (device vda1): ext4_end_bio:345: I/O error 10 writing to inode 403754 starting block 1265920)
[ 1458.064875] EXT4-fs warning (device vda1): ext4_end_bio:345: I/O error 10 writing to inode 403754 starting block 1266328)
[ 1458.065456] EXT4-fs warning (device vda1): ext4_end_bio:345: I/O error 10 writing to inode 403754 starting block 1266842)

 

[Rastta]

FWIW - I'm also noticing random data corruption since upgrading to 7.1 with qemu 6.2, with Virtio disks. I've tried changing the disks to (VirtIO) SCSI and the problem also seems to persist - random EXT4 IO errors inside the machine, leading to data corruption.
Guests are Arch/NixOS Stable with recent-ish kernels, if it helps.

I have rolled back qemu to 6.0 as mentioned previously and it seems to be back fine.
Also learned the hard lesson about backups with one of those VMS, and also a PSA: don't run fsck inside a VM in such state as IO access is still corrupt which will destroy the filesystems even more.

 

[tonci]

Ok, it seems we've got stuck here :
Now (with 6.0.0-4 version) error messages such as above ones do not appear any more ...

Waiting for some official response about this issue ...

Thanks in advance

BR

Tonci

 

[t.lamprecht]

There's a new kernel package available on the pvetest repository, namely pve-kernel-5.13.19-1-pve with package version 5.13.19-3

It includes some fixes that addresses issues with similar sounding symptoms as described in this thread here, correlating to using io_uring.

We had two reproducers that triggered quite effectively, one was using SATA as disk bus for a Windows VM (froze completely), and another reproducer was an ansible playbook that setup a nested PVE cluster for some internal testing, both work now Ok with that kernel. If you run into IO errors in kernel log or frozen Windows VMs I'd recommend trying that kernel version.

 

[tonci]

updated to 7.1.6 ... but pve-qemu-kvm is still 6.1.0-2 and with this version still have error like this (ubuntu fresh install virtio)


downgrading to 6.0.0-4 solved this out ....

Now I'm afraid upgrading from 6 to 7 ... Did that only once at one client and had some wierd problem with zfs-replication ... I will test little bit more and send you some logs

BR
Tonci

 

[JOduMonT]

@t.lamprecht

today i installed i new proxmox from iso 7-1, I had to downgrade my kernel to 5.11.22-6 otherwize my pci-passthrough didnt work
yes my english is bad, no I dont have log, but im very annoyed since with 5.13 proxmox dont even reboot by itself.

 

[JOduMonT]

again; a new update which dont fix the situation
With kernels 5.13.19-1-pve and 5.13.19-2-pve the situation remains

I'm still stuck with the kernel 5.11.22-6-pve

otherwise my any VM with PCI Passthrough won't boot, such as a VM Ubuntu 20.04LTS which exhibit the message: unable to read tail (got 0 bytes) and a non rebootable server (proxmox system) where I have to do a hard reset to reboot.

since the beginning of that issue, I reinstall from scratch proxmox with the latest version, to eliminate possible issue with the upgrade and miss configuration made with time; I also changed the drives where proxmox is installed and have a new NVM for hosting the pool where the VM are.

what proxmox team propose ?

 

[JOduMonT]

@proxmox team, good job by including an autoremove script while we rebuild kernel

Running hook script 'proxmox-auto-removal'..

I means how I could reinstall an old kernel without being auto remove by your script ???

 

[t.lamprecht]

since the beginning of that issue, I reinstall from scratch proxmox with the latest version, to eliminate possible issue with the upgrade and miss configuration made with time; I also changed the drives where proxmox is installed and have a new NVM for hosting the pool where the VM are.

How can you say that you eliminated a bad configuration under 5.13 if you never got it to work there?
Did you try the 5.15 based kernel we provide?
https://forum.proxmox.com/threads/opt-in-linux-kernel-5-15-for-proxmox-ve-7-x-available.100936/

I means how I could reinstall an old kernel without being auto remove by your script ???

The auto-removal script does not remove the following kernels automatically
- the currently booted version, if still installed
- the kernel version we've been called for
- the latest kernel version (as determined by Debian version number)
- the second-latest kernel version
- the latest kernel version of each series (e.g. 4.13, 4.15, 5.0) by marking the meta-packages

So all sensible cases are covered, if those wouldn't get auto-removed the boot partition would run out of space sooner or later..

If you want to keep a specific kernel I'd suggest to start actually reading the docs first before complaining here...
https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysboot_proxmox_boot_setup

If you cannot do that and just cannot help yourself there's always enterprise support:
https://www.proxmox.com/proxmox-ve/pricing

 

[JOduMonT]

How can you say that you eliminated a bad configuration under 5.13 if you never got it to work there?

what I means, is I reinstall proxmox, to be sure the situation was not cause by a modification (bad configuration) from me such as removing and/or stopping a services and/or editing a file in /etc, ...

Did you try the 5.15 based kernel we provide?

yes, when i reinstall proxmox 7.1, I retrograde one by one from the original kernel (if I remember well 5.13.17 to 5.11.22-60 which is the only one working; the issue started at 5.11.22-7-pve, so if you means if I tried 5.11.15, yes I tried this kernel.

 

[JOduMonT]

If you want to keep a specific kernel I'd suggest to start actually reading the docs first before complaining here...
https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysboot_proxmox_boot_setup

I don't want to keep a specific kernel
I want a normal proxmox, like it was working before, without any tweak and/or what so ever fixed package without a proper explanation

 

[JOduMonT]

If you cannot do that and just cannot help yourself there's always enterprise support:
https://www.proxmox.com/proxmox-ve/pricing

I'm sorry but, at that point, not sure paying for a support will be better

 

[JOduMonT]

Did you try the 5.15 based kernel we provide?

yes I tried, but the kernel 5.15 is not available in the boot partition

root@pve:~# ls /boot/
config-5.11.22-6-pve      initrd.img-5.11.22-7-pve  System.map-5.13.19-2-pve
config-5.11.22-7-pve      initrd.img-5.13.19-1-pve  vmlinuz-5.11.22-6-pve
config-5.13.19-1-pve      initrd.img-5.13.19-2-pve  vmlinuz-5.11.22-7-pve
config-5.13.19-2-pve      pve/                      vmlinuz-5.13.19-1-pve
efi/                      System.map-5.11.22-6-pve  vmlinuz-5.13.19-2-pve
grub/                     System.map-5.11.22-7-pve 
initrd.img-5.11.22-6-pve  System.map-5.13.19-1-pve 
root@pve:~# ls /boot/^C
root@pve:~# kernel list^C
root@pve:~# proxmox-boot-tool kernel add 5.11.22-7^Cve
root@pve:~# apt install pve-headers-5.15.5-1-pve pve-headers-5.15.5-1-pve
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
pve-headers-5.15.5-1-pve is already the newest version (5.15.5-1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
root@pve:~# proxmox-boot-tool refresh^C
root@pve:~# proxmox-boot-tool kernel add 5.15.5-1-pve
E: no kernel image found in /boot for '5.15.5-1-pve', not adding.
root@pve:~#

 

[JOduMonT]

ok I got it;

everything works as expected with 5.15
if I install it like this: apt install pve-headers-5.15 pve-kernel-5.15

sorry to being I pain in your b***
but over time I lost hope and while I taught apt upgrade would automatically upgrade to the last kernel

 

[oguz]

but over time I lost hope and while I taught apt upgrade would automatically upgrade to the last kernel

the 5.15 kernel is at the moment optional, normally when you apt update && apt dist-upgrade (never run apt upgrade!), you should get our newest non-testing kernel