Hi!
I must first say i am no seasoned network card virtual functions user but generally feel virtual-functions-and-co would be important thing tomorrow so for no very particular reason i try them out in different combinations. As staring point i had working setup like this with pve v. 8.2 using kernel 6.8.4-3
1. instead having openvswitch-switch i have openvswitch-switch-dpdk (althought i believe my current situation is not related to it since i am not dealing with dpdk flavors)
2. i have such setup script what i apply where network card and ovs is in so to say un-initialized state
i have /etc/network/interface more-or-less empty only temporarily is enp129s0f1np1 configured, and i remove this with 'ifconfig enp129s0f1np1 0'
Then i have such pve kvm virtual machine conf
And this setup works well for me (not ideally performance wise but logically well), for example ovs-vsctl looks like
where eth0 and eth1 are so to say representors, and also
After software upgrade within pve v. 8.2 i moved from kernel 6.8.4 to 6.8.8-2 and when virtual computer starts up and especially arrives to configuring network then pve host gives
I believe it is related not to ovs but specificly to using network card virtual functions. Card firmware etc is
I would appreciate very much if you could comment on this i need to do something differently or maybe just confirm that it is known phenomena and solution is already in works
(Looking around forum i saw also other posts about 6.8.8-2 kernel giving errors.) https://www.auul.pri.ee/wiki/Intel_...emu_virtuaalse_arvuti_juurde_-_vf_representor has longer version on this topic although in Estonian languge. Mostly i based my setup on https://edc.intel.com/content/www/u...ript-b-switchdev-mode-with-ovs-configuration/ text.
Best regards,
Imre
I must first say i am no seasoned network card virtual functions user but generally feel virtual-functions-and-co would be important thing tomorrow so for no very particular reason i try them out in different combinations. As staring point i had working setup like this with pve v. 8.2 using kernel 6.8.4-3
1. instead having openvswitch-switch i have openvswitch-switch-dpdk (althought i believe my current situation is not related to it since i am not dealing with dpdk flavors)
2. i have such setup script what i apply where network card and ovs is in so to say un-initialized state
Code:
# cat vf-setup.sh
ovs-vsctl add-br vmbr0
devlink dev eswitch set pci/0000:81:00.0 mode switchdev
echo 2 > /sys/class/net/enp129s0f0np0/device/sriov_numvfs
sleep 2
ip link set enp129s0f0np0 vf 0 mac 2e:38:4d:3d:61:ed
ip link set enp129s0f0np0 vf 1 mac 2e:38:4d:3d:61:ee
ethtool -K eth0 hw-tc-offload on
ethtool -K eth1 hw-tc-offload on
ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
ovs-vsctl set Open_vSwitch . other_config:tc-policy=skip_sw
systemctl restart openvswitch-switch
ovs-vsctl add-port vmbr0 eth0
ovs-vsctl add-port vmbr0 eth1
ip link set eth0 up
ip link set eth1 up
ovs-vsctl add-port vmbr0 enp129s0f0np0
ip link set enp129s0f0np0 up
ethtool -K enp129s0f0np0 hw-tc-offload on
ovs-vsctl add-port vmbr0 inter -- set interface inter type=internal
ifconfig enp129s0f1np1 0
ifconfig inter 10.40.134.18/29
route add default gw 10.40.134.17
ovs-vsctl set port eth0 tag=3564
ovs-vsctl set port eth1 tag=3564i have /etc/network/interface more-or-less empty only temporarily is enp129s0f1np1 configured, and i remove this with 'ifconfig enp129s0f1np1 0'
Then i have such pve kvm virtual machine conf
Code:
agent: 1
bios: ovmf
boot: order=virtio0;ide2;net0
cores: 8
cpu: host
efidisk0: local-to-valgustaja-1-lvm-over-mdadm:vm-1102-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
ide2: none,media=cdrom
machine: q35
memory: 8192
meta: creation-qemu=9.0.0,ctime=1720472955
name: imre-ubuntu-2404-01
ostype: l26
rng0: source=/dev/urandom
scsihw: virtio-scsi-single
serial0: socket
smbios1: uuid=eff2a3c6-e3e6-4ccf-889b-33babc49c3fc
sockets: 1
tpmstate0: local-to-valgustaja-1-lvm-over-mdadm:vm-1102-disk-1,size=4M,version=v2.0
vga: virtio
virtio0: local-to-valgustaja-1-lvm-over-mdadm:vm-1102-disk-2,backup=0,iothread=1,size=20G
vmgenid: ec8bc8c2-1279-43d2-bfca-9d7b9081c97f
hostpci0: 0000:81:01.0,pcie=1,rombar=0And this setup works well for me (not ideally performance wise but logically well), for example ovs-vsctl looks like
Code:
# ovs-vsctl show
050044f7-ae08-434f-8649-983de8e5ed3d
Bridge vmbr0
Port enp129s0f0np0
Interface enp129s0f0np0
Port inter
Interface inter
type: internal
Port eth0
tag: 3564
Interface eth0
Port vmbr0
Interface vmbr0
type: internal
Port eth1
tag: 3564
Interface eth1
ovs_version: "3.1.0"where eth0 and eth1 are so to say representors, and also
Code:
# devlink dev eswitch show pci/0000:81:00.0
pci/0000:81:00.0: mode switchdev
# driverctl list-devices network
0000:81:00.0 ice
0000:81:00.1 ice
0000:81:01.0 vfio-pci
0000:81:01.1 iavfAfter software upgrade within pve v. 8.2 i moved from kernel 6.8.4 to 6.8.8-2 and when virtual computer starts up and especially arrives to configuring network then pve host gives
Code:
2024-07-15T12:18:49.261183+03:00 valgustaja1 qm[3923]: start VM 1102: UPID:valgustaja1:00000F53:0000582D:6694E979:qmstart:1102:root@pam:
...
2024-07-15T12:18:49.363026+03:00 valgustaja1 kernel: [ 225.838639] iavf 0000:81:01.0: Removing device
2024-07-15T12:18:49.951211+03:00 valgustaja1 systemd[1]: Created slice qemu.slice - Slice /qemu.
2024-07-15T12:18:49.957030+03:00 valgustaja1 systemd[1]: Started 1102.scope.
2024-07-15T12:18:50.475063+03:00 valgustaja1 kernel: [ 226.951037] vfio-pci 0000:81:01.0: enabling device (0000 -> 0002)
2024-07-15T12:18:50.769838+03:00 valgustaja1 qm[3920]: <root@pam> end task UPID:valgustaja1:00000F53:0000582D:6694E979:qmstart:1102:root@pam: OK
2024-07-15T12:18:52.216805+03:00 valgustaja1 pvedaemon[3999]: starting vnc proxy UPID:valgustaja1:00000F9F:00005954:6694E97C:vncproxy:1102:root@pam:
2024-07-15T12:18:52.217078+03:00 valgustaja1 pvedaemon[2665]: <root@pam> starting task UPID:valgustaja1:00000F9F:00005954:6694E97C:vncproxy:1102:root@pam:
2024-07-15T12:19:34.456984+03:00 valgustaja1 kernel: [ 270.755914] BUG: kernel NULL pointer dereference, address: 000000000000003f
2024-07-15T12:19:34.456998+03:00 valgustaja1 kernel: [ 270.756271] #PF: supervisor read access in kernel mode
2024-07-15T12:19:34.456999+03:00 valgustaja1 kernel: [ 270.756554] #PF: error_code(0x0000) - not-present page
2024-07-15T12:19:34.457000+03:00 valgustaja1 kernel: [ 270.756831] PGD 355f71067 P4D 0
2024-07-15T12:19:34.457000+03:00 valgustaja1 kernel: [ 270.757101] Oops: 0000 [#1] PREEMPT SMP NOPTI
2024-07-15T12:19:34.457001+03:00 valgustaja1 kernel: [ 270.757365] CPU: 31 PID: 491 Comm: kworker/31:1 Tainted: P O 6.8.8-2-pve #1
2024-07-15T12:19:34.457001+03:00 valgustaja1 kernel: [ 270.757628] Hardware name: Supermicro AS -1115CS-TNR/H13SSW, BIOS 1.6 10/05/2023
2024-07-15T12:19:34.457001+03:00 valgustaja1 kernel: [ 270.757888] Workqueue: ice ice_service_task [ice]
2024-07-15T12:19:34.457002+03:00 valgustaja1 kernel: [ 270.758170] RIP: 0010:ice_lag_move_new_vf_nodes+0x82/0x1c0 [ice]
2024-07-15T12:19:34.457002+03:00 valgustaja1 kernel: [ 270.758452] Code: 68 01 49 89 c4 0f 85 49 01 00 00 4d 8b 75 28 49 8d 86 e8 0a 00 00 4d 8b ae a0 47 00 00 48 89 c7 48 89 44 24 08 e8 1e 46 ba e8 <41> f6 45 3f 01 0f 84 e1 00 00 00 49 83 7d 10 00 45 0f b6 7d 3e 48
2024-07-15T12:19:34.457003+03:00 valgustaja1 kernel: [ 270.758987] RSP: 0018:ff4352c94227fbc0 EFLAGS: 00010246
2024-07-15T12:19:34.457003+03:00 valgustaja1 kernel: [ 270.759252] RAX: 0000000000000000 RBX: 00000000ffffffff RCX: 0000000000000000
2024-07-15T12:19:34.457004+03:00 valgustaja1 kernel: [ 270.759515] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
2024-07-15T12:19:34.457004+03:00 valgustaja1 kernel: [ 270.759773] RBP: ff4352c94227fc18 R08: 0000000000000000 R09: 0000000000000000
2024-07-15T12:19:34.457004+03:00 valgustaja1 kernel: [ 270.760033] R10: 0000000000000000 R11: 0000000000000000 R12: ff3cd473d2593028
2024-07-15T12:19:34.457005+03:00 valgustaja1 kernel: [ 270.760296] R13: 0000000000000000 R14: ff3cd471ba8581a0 R15: 0000000000000000
2024-07-15T12:19:34.457005+03:00 valgustaja1 kernel: [ 270.760561] FS: 0000000000000000(0000) GS:ff3cd4a00c980000(0000) knlGS:0000000000000000
2024-07-15T12:19:34.457005+03:00 valgustaja1 kernel: [ 270.760828] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
2024-07-15T12:19:34.457005+03:00 valgustaja1 kernel: [ 270.761093] CR2: 000000000000003f CR3: 0000000346806002 CR4: 0000000000f71ef0
2024-07-15T12:19:34.457006+03:00 valgustaja1 kernel: [ 270.761364] PKRU: 55555554
2024-07-15T12:19:34.457006+03:00 valgustaja1 kernel: [ 270.761630] Call Trace:
2024-07-15T12:19:34.457007+03:00 valgustaja1 kernel: [ 270.761898] <TASK>
2024-07-15T12:19:34.457007+03:00 valgustaja1 kernel: [ 270.762162] ? show_regs+0x6d/0x80
2024-07-15T12:19:34.457007+03:00 valgustaja1 kernel: [ 270.762429] ? __die+0x24/0x80
2024-07-15T12:19:34.457008+03:00 valgustaja1 kernel: [ 270.762690] ? page_fault_oops+0x176/0x500
2024-07-15T12:19:34.457008+03:00 valgustaja1 kernel: [ 270.762950] ? srso_alias_return_thunk+0x5/0xfbef5
2024-07-15T12:19:34.457008+03:00 valgustaja1 kernel: [ 270.763214] ? ice_sched_find_node_by_teid+0x71/0xb0 [ice]
2024-07-15T12:19:34.457009+03:00 valgustaja1 kernel: [ 270.763499] ? do_user_addr_fault+0x2f9/0x6b0
2024-07-15T12:19:34.457009+03:00 valgustaja1 kernel: [ 270.763757] ? exc_page_fault+0x83/0x1b0
2024-07-15T12:19:34.457009+03:00 valgustaja1 kernel: [ 270.764014] ? asm_exc_page_fault+0x27/0x30
2024-07-15T12:19:34.457009+03:00 valgustaja1 kernel: [ 270.764266] ? ice_lag_move_new_vf_nodes+0x82/0x1c0 [ice]
2024-07-15T12:19:34.457009+03:00 valgustaja1 kernel: [ 270.764534] ? srso_alias_return_thunk+0x5/0xfbef5
2024-07-15T12:19:34.457009+03:00 valgustaja1 kernel: [ 270.764778] ice_vc_cfg_qs_msg+0xa3/0x680 [ice]
2024-07-15T12:19:34.457010+03:00 valgustaja1 kernel: [ 270.765035] ? ice_vc_send_msg_to_vf+0x39/0xa0 [ice]
2024-07-15T12:19:34.457010+03:00 valgustaja1 kernel: [ 270.765288] ice_vc_process_vf_msg+0x5b4/0xb60 [ice]
2024-07-15T12:19:34.457010+03:00 valgustaja1 kernel: [ 270.765539] __ice_clean_ctrlq+0x2e7/0xa90 [ice]
2024-07-15T12:19:34.457011+03:00 valgustaja1 kernel: [ 270.765787] ? srso_alias_return_thunk+0x5/0xfbef5
2024-07-15T12:19:34.457011+03:00 valgustaja1 kernel: [ 270.766017] ice_service_task+0xade/0x10b0 [ice]
2024-07-15T12:19:34.457011+03:00 valgustaja1 kernel: [ 270.766256] ? add_timer+0x20/0x40
2024-07-15T12:19:34.457011+03:00 valgustaja1 kernel: [ 270.766473] ? srso_alias_return_thunk+0x5/0xfbef5
2024-07-15T12:19:34.457011+03:00 valgustaja1 kernel: [ 270.766684] ? srso_alias_return_thunk+0x5/0xfbef5
2024-07-15T12:19:34.457012+03:00 valgustaja1 kernel: [ 270.766889] process_one_work+0x16a/0x350
2024-07-15T12:19:34.457012+03:00 valgustaja1 kernel: [ 270.767089] worker_thread+0x306/0x440
2024-07-15T12:19:34.457012+03:00 valgustaja1 kernel: [ 270.767289] ? __pfx_worker_thread+0x10/0x10
2024-07-15T12:19:34.457012+03:00 valgustaja1 kernel: [ 270.767492] kthread+0xef/0x120
2024-07-15T12:19:34.457012+03:00 valgustaja1 kernel: [ 270.767694] ? __pfx_kthread+0x10/0x10
2024-07-15T12:19:34.457012+03:00 valgustaja1 kernel: [ 270.767899] ret_from_fork+0x44/0x70
2024-07-15T12:19:34.457013+03:00 valgustaja1 kernel: [ 270.768101] ? __pfx_kthread+0x10/0x10
2024-07-15T12:19:34.457013+03:00 valgustaja1 kernel: [ 270.768301] ret_from_fork_asm+0x1b/0x30
2024-07-15T12:19:34.457014+03:00 valgustaja1 kernel: [ 270.768502] </TASK>
2024-07-15T12:19:34.457014+03:00 valgustaja1 kernel: [ 270.768697] Modules linked in: tcp_diag inet_diag vfio_pci vfio_pci_core vfio_iommu_type1 vfio iommufd act_vlan act_mirred act_skbedit cls_matchall act_gact cls_flower sch_ingress iavf nfnetlink_cttimeout ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter sctp ip6_udp_tunnel udp_tunnel nf_tables bonding tls openvswitch nsh nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 softdog nfnetlink_log binfmt_misc nfnetlink rpcrdma intel_rapl_msr intel_rapl_common sunrpc amd64_edac edac_mce_amd rdma_ucm ib_iser libiscsi scsi_transport_iscsi rdma_cm iw_cm kvm_amd ib_cm ipmi_ssif kvm irqbypass crct10dif_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha256_ssse3 sha1_ssse3 aesni_intel crypto_simd cryptd dax_hmem cxl_acpi rapl cxl_core pcspkr irdma i40e ib_uverbs acpi_ipmi ast ib_core i2c_algo_bit ipmi_si k10temp ccp ipmi_devintf ipmi_msghandler joydev input_leds mac_hid vhost_net vhost vhost_iotlb tap efi_pstore dmi_sysfs ip_tables x_tables autofs4
2024-07-15T12:19:34.457016+03:00 valgustaja1 kernel: [ 270.768776] rndis_host cdc_ether usbnet mii hid_generic usbmouse usbhid hid zfs(PO) spl(O) btrfs blake2b_generic raid10 raid1 raid0 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c xhci_pci xhci_pci_renesas crc32_pclmul ice nvme ahci gnss xhci_hcd nvme_core libahci i2c_piix4 nvme_auth
2024-07-15T12:19:34.457016+03:00 valgustaja1 kernel: [ 270.771408] CR2: 000000000000003f
2024-07-15T12:19:34.457017+03:00 valgustaja1 kernel: [ 270.771646] ---[ end trace 0000000000000000 ]---
2024-07-15T12:19:34.457017+03:00 valgustaja1 kernel: [ 270.930129] RIP: 0010:ice_lag_move_new_vf_nodes+0x82/0x1c0 [ice]
2024-07-15T12:19:34.457017+03:00 valgustaja1 kernel: [ 270.930457] Code: 68 01 49 89 c4 0f 85 49 01 00 00 4d 8b 75 28 49 8d 86 e8 0a 00 00 4d 8b ae a0 47 00 00 48 89 c7 48 89 44 24 08 e8 1e 46 ba e8 <41> f6 45 3f 01 0f 84 e1 00 00 00 49 83 7d 10 00 45 0f b6 7d 3e 48
2024-07-15T12:19:34.457018+03:00 valgustaja1 kernel: [ 270.930950] RSP: 0018:ff4352c94227fbc0 EFLAGS: 00010246
2024-07-15T12:19:34.457018+03:00 valgustaja1 kernel: [ 270.931203] RAX: 0000000000000000 RBX: 00000000ffffffff RCX: 0000000000000000
2024-07-15T12:19:34.457019+03:00 valgustaja1 kernel: [ 270.931457] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
2024-07-15T12:19:34.457019+03:00 valgustaja1 kernel: [ 270.931710] RBP: ff4352c94227fc18 R08: 0000000000000000 R09: 0000000000000000
2024-07-15T12:19:34.457019+03:00 valgustaja1 kernel: [ 270.931964] R10: 0000000000000000 R11: 0000000000000000 R12: ff3cd473d2593028
2024-07-15T12:19:34.457020+03:00 valgustaja1 kernel: [ 270.932222] R13: 0000000000000000 R14: ff3cd471ba8581a0 R15: 0000000000000000
2024-07-15T12:19:34.457020+03:00 valgustaja1 kernel: [ 270.932482] FS: 0000000000000000(0000) GS:ff3cd4a00c980000(0000) knlGS:0000000000000000
2024-07-15T12:19:34.457021+03:00 valgustaja1 kernel: [ 270.932743] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
2024-07-15T12:19:34.457021+03:00 valgustaja1 kernel: [ 270.933002] CR2: 000000000000003f CR3: 0000000346806002 CR4: 0000000000f71ef0
2024-07-15T12:19:34.457021+03:00 valgustaja1 kernel: [ 270.933264] PKRU: 55555554
2024-07-15T12:19:34.457022+03:00 valgustaja1 kernel: [ 270.933523] note: kworker/31:1[491] exited with irqs disabledI believe it is related not to ovs but specificly to using network card virtual functions. Card firmware etc is
Code:
# devlink dev info pci/0000:81:00.0
pci/0000:81:00.0:
driver ice
serial_number 00-01-00-ff-ff-00-00-00
versions:
fixed:
board.id K58132-000
running:
fw.mgmt 7.2.4
fw.mgmt.api 1.7.10
fw.mgmt.build 0xe49dbde8
fw.undi 1.3346.0
fw.psid.api 4.20
fw.bundle_id 0x800177ba
fw.app.name ICE OS Default Package
fw.app 1.3.36.0
fw.app.bundle_id 0xc0000001
fw.netlist 2.40.5000-2.f.0
fw.netlist.build 0x85001ebf
stored:
fw.undi 1.3346.0
fw.psid.api 4.20
fw.bundle_id 0x800177ba
fw.netlist 2.40.5000-2.f.0
fw.netlist.build 0x85001ebfI would appreciate very much if you could comment on this i need to do something differently or maybe just confirm that it is known phenomena and solution is already in works
(Looking around forum i saw also other posts about 6.8.8-2 kernel giving errors.) https://www.auul.pri.ee/wiki/Intel_...emu_virtuaalse_arvuti_juurde_-_vf_representor has longer version on this topic although in Estonian languge. Mostly i based my setup on https://edc.intel.com/content/www/u...ript-b-switchdev-mode-with-ovs-configuration/ text.Best regards,
Imre
Last edited: