qemu virtio issues after upgrade to 9

D

danstephans

Active Member
Apr 21, 2020
23
2
43
55
Note all of these things worked fine before upgrading to 9.

I've upgraded our production cluster to pve9 and most things are good but there are clearly some problems with virtio networking. Two things stand out (and there may be more).

First, our cluster has compute, storage and hybrid nodes. The storage nodes are all R740XD and the compute nodes are a mix of R740 and R750 and the hybrid node is a R750. The hybrid node is essentially the same as the other R750s in the stack, with a newer bios, and guests with virtio network drivers cannot be successfully migrated to or from this host. When migrating to the host the following error is observed (this is from any other node in the stack)

Code: 
Aug 10 18:16:57 hybr-01-prod systemd[1]: Started 101.scope.
Aug 10 18:17:02 hybr-01-prod QEMU[1473513]: kvm: Features 0x130afffaf unsupported. Allowed features: 0x1c0010179bfffe7
Aug 10 18:17:02 hybr-01-prod QEMU[1473513]: kvm: Failed to load virtio-net:virtio
Aug 10 18:17:02 hybr-01-prod QEMU[1473513]: kvm: error while loading state for instance 0x0 of device '0000:00:12.0/virtio-net'
Aug 10 18:17:02 hybr-01-prod QEMU[1473513]: kvm: Putting registers after init: Failed to set XCRs: Invalid argument


When migrating from the node to any other node in the stack, the receiving node generates the following error.

Code: 
Aug 10 18:51:00 comp-06-prod QEMU[1578269]: kvm: get_pci_config_device: Bad config data: i=0x10 read: a1 device: 1 cmask: ff wmask: c0 w1cmask:0
Aug 10 18:51:00 comp-06-prod QEMU[1578269]: kvm: Failed to load PCIDevice:config
Aug 10 18:51:00 comp-06-prod QEMU[1578269]: kvm: Failed to load virtio-net:virtio
Aug 10 18:51:00 comp-06-prod QEMU[1578269]: kvm: error while loading state for instance 0x0 of device '0000:00:12.0/virtio-net'
Aug 10 18:51:00 comp-06-prod QEMU[1578269]: kvm: load of migration failed: Invalid argument


guests without the virtio network driver migrate fine to and from.

The second issue took me almost 24 hours to identify because it is so odd. We have a load balancer that talks to three vms that live on a different network segment. The vms all seem to work fine and are on the network without issue. From certain network segments an ssl session can be established but then when the vm starts sending data, it disappears / never makes it to the requester. This does not happen across all network segments. We moved the vms to the same network as the load balancer and it then worked. We initially thought it was a firewall issue and spent a lot of time on that but as a test I changed the interface to e1000 on one of the vms and everything worked on its old network segment. There is clearly some weird problem here.

It seems there is some issue with virtio network in the latest qemu. We are running the following across all hosts:

Code: 
QEMU emulator version 10.0.2 (pve-qemu-kvm_10.0.2-4)
Copyright (c) 2003-2025 Fabrice Bellard and the QEMU Project developers

pve-manager/9.0.4/39d8a4de7dfb2c40 (running kernel: 6.14.8-2-pve)
 
Another issue just came up.

We have a mysql server that is used for various production things. Access from the command line works fine from anywhere. A regularly scheduled job in perl that uses DBD loses its connect to mysql every time (and this literally just started this morning, days after the upgrade). I switched the interface type to e1000 from virtio and the issue is resolved.
 
The vms in the first problem above (that sit behind the load balancer) are all running Rocky 8 at the final patch version (8.10) and are a standard build with the application being accessed run in a docker container.

The mysql system is a Rocky 9 system at the latest patch level and is standard.
 
So this is clearly a problem that appears after some time or number of packets or whatever. We have a print server that communicates with a sql server. Everything works except the SQL connections are now breaking. It appears that all these broken connections happen after a key exchange for encrypted connections. I've changed the interface on the print server to E1000 and things work again.

I tested everything after migration and all of these things worked. They are breaking at random times and it's all virtio networking.
 
Here's requested logs on the migration problem. I'll post more on the other problem later.

This particular VM that I tested with (this morning) is Rocky 8.10 -- Kernel 4.18.0-553.58.1.el8_10.x86_64

If I switch the interface to E1000 it works every time. This is the same solution for the odd application problems that we've been seeing. I'll post some packet captures later when I have some time to generate them. We see these migration problems across a spectrum of systems using virtio. Including windows and other distros of linux running various kernels. Without an exception, if the system has a migration problem switching it to E1000 solves it.
 

Attachments

Had another VM (a repo server for distros) experience the weird application layer problem where the box is still online and everything works but application specific connections, I'm going to have to do a full packet capture to see what's wrong with the tcp conversation. Switching the interface (live) to E1000 once again immediately solved the problem.
 
Please also share the full VM configuration qm config <ID> and the configuration of the bridges used by the VM on the host.
 
I would like to reiterate / emphasize that this cluster has been in use without issue for a year. After upgrading is when these virtio problems have appeared across the cluster. If these problems are not being seen on clean installs, clearly something was missed in the upgrade process.

vm config for failing migration vm (there are a lot of them, this is just the one I've been using for testing)

Code: 
agent: 1
balloon: 2048
boot: order=ide2;scsi0;net0
cores: 2
cpu: x86-64-v2-AES
ide2: none,media=cdrom
memory: 8192
meta: creation-qemu=9.0.2,ctime=1751938987
name: idp1-bastion
net0: virtio=BC:24:11:BC:39:12,bridge=vmbr1,tag=1940
numa: 0
onboot: 1
ostype: l26
scsi0: HDD:vm-264-disk-0,iothread=1,size=100G
scsihw: virtio-scsi-single
smbios1: uuid=cd260899-75bc-4048-bd12-7e74f35dc241
sockets: 1
vmgenid: 705cc071-277b-4b35-a8e4-945e04f4c0fd

network config on hybr-01-prod
Code: 
auto lo
iface lo inet loopback

iface idrac inet manual

iface eno8303 inet manual

iface eno8403 inet manual

iface eno12399np0 inet manual
iface eno12409np1 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves eno12399np0 eno12409np1
        bond-miimon 100
        bond-mode 802.3ad
        bond-lacp_rate fast
        bond-xmit_hash_policy layer2+3
        mtu 9100
#25G bond

iface ens6f0 inet manual
iface ens6f1 inet manual

auto bond1
iface bond1
        bond-slaves ens6f0 ens6f1
        bond-miimon 100
        bond-mode 802.3ad
        bond-lacp_rate fast
        bond-xmit_hash_policy layer2+3
        mtu 9100
#10G bond

auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
#25G Trunk

auto vmbr1
iface vmbr1 inet manual
        bridge-ports bond1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
#10G Trunk

auto vmbr1.1504
iface vmbr1.1504 inet static
        address 10.150.5.117/22
        gateway 10.150.4.254
        mtu 1500
#New Servers

auto vmbr0.411
iface vmbr0.411 inet static
        address 192.168.41.117/24
        mtu 9100
#CEPH and cluster backend

auto vmbr1.1510
iface vmbr1.1510 inet static
        address 10.150.10.253/23
        mtu 1500
#test interface

All nodes have similar network / bridge configurations. We use the 25G bond on vmbr0 for all cluster traffic. VMs *can* use this bridge as it supports all the vlans but we generally have VM comms and the management traffic on vmbr1 which is a 10G bond.
 
After a day of troubleshooting I think I'm on to a resolution for the vm level (non-migration) issues we've been seeing with virtio-net. I'll post more once I validate.
 
Ok, this actually solves both problems.

As indicated in our network configs posted above we use jumbo frames on our bridges with an mtu of 9100. The interfaces have always been set to use the mtu of the bridge in config. When the interface type is virtio on the vm and a packet needs to be fragmented the tcp stream stalls. Previously I was just changing the interface to e1000 because that immediately resolved the issue. Leaving the interface as virtio and setting the MTU to 1500 also resolves the issue. Note packet captures show the e1000 interface doing jumbo frames just fine on the same traffic that makes virtio choke.

The migration works just fine if I set the virtio interface to MTU 1500 as well.

This appears to be some sort of regression in virtio-net. We've been running this configuration for over a year now without issue. Now that I have a somewhat acceptable workaround that doesn't require a lot of intervention I will not go through the extensive hassle of downgrading to 8 but it would be nice to have jumbo frames back.
 
  • Like
Reactions: fiona
Why the migration MTU issue only manifested itself on migrations to/from one specific node is also probably interesting for someone who is familiar with the source and can correlate the errors with some MTU mishandling.
 
Also our vmxnet interfaces are trouble-free and are set to use the bridge MTU and you actually cannot change it in config (which I didn't know until just now)
 
I have the same problem. I was able to figure out that the issue occurs when the bridge does not have the same MTU as the VM.

My Case:

Host:
  • vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000
    • tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
VM:

ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel

Bildschirmfoto 2025-09-03 um 18.41.34.png

For MTU: it says “Same as bridge.” However, the MTU inside the VM differs from the MTU of the tap interface or the bridge itself. When I migrate now, an error occurs (ERROR: tunnel replied 'ERR: resume failed - VM 100 qmp command 'query-status' failed - client closed connection' to command 'resume 100')—or in my case, the VM is migrated, but it is powered off, but the state is running, pretty crazy. If I manually set the MTU value to 9000, the migration works. If I first power off the VM, migrate it, and then power it back on, everything works fine. However, in that case, the MTU value actually ends up being 9000.

Is this "Same as bridge" new? All my VMs now have an interface with MTU 9000 cause my vmbr0 is set to 9000 . . .

Shouldn’t 1500 be the default instead of "Same as bridge" ?

Edit:
Is “Same as Bridge” the same as “Use the special value ‘1’ to inherit the MTU value from the underlying bridge”? So, does ‘1’ mean the same as “Same as Bridge” at this time?
 
Last edited:
Hi,
fettfoen said:
For MTU: it says “Same as bridge.” However, the MTU inside the VM differs from the MTU of the tap interface or the bridge itself. When I migrate now, an error occurs (ERROR: tunnel replied 'ERR: resume failed - VM 100 qmp command 'query-status' failed - client closed connection' to command 'resume 100')—or in my case, the VM is migrated, but it is powered off, but the state is running, pretty crazy. If I manually set the MTU value to 9000, the migration works. If I first power off the VM, migrate it, and then power it back on, everything works fine. However, in that case, the MTU value actually ends up being 9000.

Is this "Same as bridge" new? All my VMs now have an interface with MTU 9000 cause my vmbr0 is set to 9000 . . .

Shouldn’t 1500 be the default instead of "Same as bridge" ?

Edit:
Is “Same as Bridge” the same as “Use the special value ‘1’ to inherit the MTU value from the underlying bridge”? So, does ‘1’ mean the same as “Same as Bridge” at this time?
Click to expand...
yes, the default behavior with empty value was changed in Proxmox VE 9 to be "inherit from bridge". The special value 1 is still supported for compatibility.
 
Could you explain why this change was made? Was there a specific technical reason or issue with keeping 1500 as the default? Also, is there a way to restore the old behavior, so that an empty value defaults to MTU 1500 instead of inheriting from the bridge?
 
fettfoen said:
Could you explain why this change was made? Was there a specific technical reason or issue with keeping 1500 as the default?
Click to expand...
One reason behind that change was that e.g. VXLAN requires a lower MTU than 1500 due to overhead, so the bridges there are usually set to 1450 MTU. Another reason was that containers already inherited the MTU of the bridge as default behavior, so that change also unified that behavior between VMs and containers.

fettfoen said:
Also, is there a way to restore the old behavior, so that an empty value defaults to MTU 1500 instead of inheriting from the bridge?
Click to expand...
There's currently no way of restoring the old behavior, apart from manually forcing the MTU to 1500.
 
Thank you for your responses. Now I’m a bit wiser. In my opinion, this change came unexpectedly or was not properly tested, or there was not sufficient warning. In my case, the bridge has an MTU of 9000, since the storage is also connected through it. Simply setting the bridge’s MTU to 1500 would lead to a disconnection from the storage. I therefore had to shut down, migrate, and restart each VM – with the result that now all VMs have an MTU of 9000. Do I really have to manually set each VM to an MTU of 1500, damn?! The changes are understandable, but the implementation has caused major issues. Overall, this is very unfortunate.
 
You must log in or register to reply here.
Top Bottom