Not the person you replied too but I tested and it didn't work with 10.2.1-2 but did with 10.1.2-7, I have a realtek nic though and my problem was internal dns didn't work in my k3s node.
QEMU 11.0 available on pve-test and pve-no-subscription as of now
- Thread starter fiona
- Start date
Between QEMU 10.1 and 10.2 support for negotiating extended VirtIO features was added.
Could you go to the VM monitor
as well as
Does using host kernel 6.17 work as well to mitigate the issue?
Could you go to the VM monitor
qm monitor ID with the numerical ID of the VM and run the following commands
Code:
qm> info virtio
/machine/peripheral/net1/virtio-backend [virtio-net]
/machine/peripheral/virtio0/virtio-backend [virtio-blk]
/machine/peripheral/balloon0/virtio-backend [virtio-balloon]
qm> info virtio-status /machine/peripheral/net1/virtio-backend
/machine/peripheral/net1/virtio-backend:
device_name: virtio-net (vhost)
device_id: 1
...ethtool -k bridge with your actual bridge name used for the VM interface. Both when it's working and when it's not working, so the outputs can be compared. Please indicate the used QEMU and host kernel versions each time.Does using host kernel 6.17 work as well to mitigate the issue?
Last edited:
Ive attached the output of those commands included the ethtool at the top.
Afaik I remember using 6.17 didn't help.
Kernel 7.0.6-2, which will soon be on the
pve-test repository, contains a fix for UDP offloading. Could you check if the issue also exists with that version (once it's available)?
In the VirtIO features I can see that
are new between your 10.1 and 10.2 output, so might be related.
Code:
VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO_CSUM: Device can receive GSO over UDP tunnel packets requiring checksum offload for the outer header,
VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO: Device can receive GSO over UDP tunnel packets,
VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO: Driver can receive GSO over UDP tunnel packets requiring checksum offload for the outer header,
VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO: Driver can receive GSO over UDP tunnel packets,@fiona It's an alpine linux vm (6.18.33-0-lts) running k3s with a flannel network config.
Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 15)
The error DNS is just a timeout, it can't reach the DNS server running on another node over the flannel network.
If I disable tx-checksumming and rx-checksumming via ethtool it starts working again.
Code:
agent: 1
balloon: 0
bios: seabios
boot: order=scsi0;net0
cores: 4
cpu: host
memory: 6144
meta: creation-qemu=7.1.0,ctime=1673013831
name: k3s-node4
net0: virtio=BC:24:11:D4:9D:A9,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local:231/vm-231-disk-0.qcow2,discard=on,size=16G,ssd=1
scsi1: local-lvm-tiny:vm-231-disk-0,discard=on,size=150G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=5e5a8376-4fb9-4fba-b9de-be4e065d3cd0
sockets: 1
usb0: host=10c4:ea60
vmgenid: 49190217-9052-48ca-9f7e-9e23b86e3ee3Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 15)
The error DNS is just a timeout, it can't reach the DNS server running on another node over the flannel network.
If I disable tx-checksumming and rx-checksumming via ethtool it starts working again.
Unfortunately, we were not able to reproduce the issue locally yet. When you have the chance, could you test again with QEMU 11.0 and with the following patch applied:
When using
After testing, to get rid of the patch, you can run
Code:
[I] root@pve9a1 ~# cat tunnel-csum.patch
@@ -1419,6 +1419,10 @@ sub print_netdevice_full {
$tmpstr .= ",romfile=$romfile" if $romfile;
}
+ if ($net->{model} eq 'virtio') {
+ $tmpstr .= ",guest_tunnel_csum=off,host_tunnel_csum=off";
+ }
+
return $tmpstr;
}
[I] root@pve9a1 ~# patch /usr/share/perl5/PVE/QemuServer.pm tunnel-csum.patch
patching file /usr/share/perl5/PVE/QemuServer.pm
[I] root@pve9a1 ~# systemctl reload-or-restart pvedaemon.service pveproxy.serviceqm showcmd ID --pretty | grep net with the numerical ID of the VM, the new options should be visible. Then start the VM fresh (if you do it via CLI qm start ID you don't need the systemctl command) and test again. This will tell us if the issue is related to those new features.After testing, to get rid of the patch, you can run
apt install --reinstall qemu-server.
Last edited:
@shanreich was finally able to reproduce the issue. When the physical NIC does not support the
features, the issue can happen. When the physical NIC does support them, it works. The issue seems to be that the feature negotiation between QEMU/host/guest (AFAICT wrongly) decides to turn on these features for the vNIC even if the underlying physical NIC does not support them. Although they are also turned on for the bridge and tap interfaces. We'll investigate further where exactly the negotiation/setting of features goes wrong.
On the QEMU side, using
Code:
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]On the QEMU side, using
host_tunnel=off for the vNIC makes it work, which we might look into as a stop-gap if we cannot find the real root cause soon.
Last edited:
My environment does have support for(see attachment for more info)
Code:
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: onbut K8S network is not working.
I also tried patching
Rebooted all my Proxmox nodes and it did not solve the issue.
Downgraded QEMU to 10.2.1-2, rebooted all Proxmox hosts - pods in my k8s still can't communicate.
Downgraded QEMU to 10.1.2-7, VMs are failing to come up:
Code:
kvm: -device virtio-net-pci,mac=BC:24:11:AC:99:F2,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=101,host_mtu=9000,guest_tunnel_csum=off,host_tunnel_csum=off: Property 'virtio-net-pci.guest_tunnel_csum' not found
TASK ERROR: start failed: QEMU exited with code 1