WARNING: drivers/iommu/dma-iommu.c:1953 at dma_iova_link

R

RobertKih

Member
Mar 14, 2023
2
0
6
I just started a Proxmox 8 to 9 upgrade on a three node cluster.
After a seemingly successful upgrade of the first node I started live migrating VMs to it to continue with the second node.
Most VMs have VM disks stored on Ceph, but some have storage on a local NVME disk on each node for various reasons.
During the upgrade the instances with local storage are moved to Ceph during the upgrade.

Upon changing disks back from Ceph to local storage those VMs started acting up and the following error was found on the first node, after the upgrade and after a reboot:

Code: 
[Tue Sep 15 13:01:43 2026] ------------[ cut here ]------------                                                                                                                                                                                                                                                                                            [Tue Sep 15 13:01:43 2026] WARNING: drivers/iommu/dma-iommu.c:1953 at dma_iova_link+0x1ab/0x360, CPU#31: kvm/36157                                                                                                                                                                                                                                         [Tue Sep 15 13:01:43 2026] Modules linked in: tcp_diag inet_diag ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter scsi_transport_iscsi nf_tables ceph libceph krb5enc authenc camellia_generic camellia_aesni_avx2 camellia_aesni_avx_x86_64 camellia_x86_64 cmac krb5 netfs nfnetlink_cttimeout bonding softdog openvswitch nsh nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 binfmt_misc nfnetlink_log ipmi_ssif sch_fq_codel amd_atl intel_rapl_msr intel_rapl_common amd64_edac edac_mce_amd kvm_amd kvm acpi_ipmi irqbypass ghash_clmulni_intel jc42 ipmi_si mlx5_fwctl joydev aesni_intel ast input_leds vga16fb ipmi_devintf wmi_bmof rapl pcspkr vgastate i2c_algo_bit ccp ee1004 bnxt_re k10temp fwctl ipmi_msghandler mac_hid zfs(PO) spl(O) msr auth_rpcgss vhost_net vhost vhost_iotlb tap nvme_fabrics efi_pstore sunrpc nfnetlink dmi_sysfs ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid0 linear mlx5_ib ib_uverbs macsec
[Tue Sep 15 13:01:43 2026]  ib_core raid1 hid_generic usbmouse rndis_host usbhid cdc_ether hid usbnet mii mlx5_core nvme nvme_core mlxfw xhci_pci nvme_keyring psample nvme_auth ahci hkdf bnxt_en xhci_hcd tls libahci i2c_piix4 i2c_smbus ptdma pci_hyperv_intf wmi
[Tue Sep 15 13:01:43 2026] CPU: 31 UID: 0 PID: 36157 Comm: kvm Tainted: P           O        7.0.14-17-pve #1 PREEMPT(lazy)
[Tue Sep 15 13:01:43 2026] Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE
[Tue Sep 15 13:01:43 2026] Hardware name: Supermicro AS -1114S-WN10RT/H12SSW-NTR, BIOS 2.6a 08/28/2023
[Tue Sep 15 13:01:43 2026] RIP: 0010:dma_iova_link+0x1ab/0x360
[Tue Sep 15 13:01:43 2026] Code: 4d d0 49 89 c3 75 51 48 8b 45 d0 49 29 c6 4c 89 75 c8 0f 85 d7 00 00 00 48 83 7d d0 00 0f 85 8a 00 00 00 31 c0 e9 43 ff ff ff <0f> 0b 48 8d 65 d8 b8 fb ff ff ff 5b 41 5c 41 5d 41 5e 41 5f 5d 31
[Tue Sep 15 13:01:43 2026] RSP: 0018:ffffd48d5648f6d0 EFLAGS: 00010206
[Tue Sep 15 13:01:43 2026] RAX: 0000000000001000 RBX: 000000278d758400 RCX: 0000000000000fff
[Tue Sep 15 13:01:43 2026] RDX: 0000000000000400 RSI: ffff8de371988388 RDI: 0000000000000000
[Tue Sep 15 13:01:43 2026] RBP: ffffd48d5648f730 R08: 0000000000000400 R09: 0000000000000001
[Tue Sep 15 13:01:43 2026] R10: ffff8de371988388 R11: 0000000000000000 R12: 0000000000000800
[Tue Sep 15 13:01:43 2026] R13: ffff8de3443830d0 R14: 0000000000000400 R15: 0000000000000001
[Tue Sep 15 13:01:43 2026] FS:  000074cbd4e78880(0000) GS:ffff8e60d308d000(0000) knlGS:0000000000000000
[Tue Sep 15 13:01:43 2026] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Tue Sep 15 13:01:43 2026] CR2: 00007fff0f3df4d8 CR3: 00000003a6dcf038 CR4: 0000000000f70ef0
[Tue Sep 15 13:01:43 2026] PKRU: 55555554
[Tue Sep 15 13:01:43 2026] Call Trace:
[Tue Sep 15 13:01:43 2026]  <TASK>
[Tue Sep 15 13:01:43 2026]  ? srso_alias_return_thunk+0x5/0xfbef5
[Tue Sep 15 13:01:43 2026]  ? dma_iova_try_alloc+0xcb/0x130
[Tue Sep 15 13:01:43 2026]  blk_dma_map_iter_start+0x2d6/0x360
[Tue Sep 15 13:01:43 2026]  blk_rq_dma_map_iter_start+0x5e/0xe0
[Tue Sep 15 13:01:43 2026]  nvme_prep_rq.part.0+0x473/0xb50 [nvme]
[Tue Sep 15 13:01:43 2026]  ? __entry_text_end+0x1024b9/0x1024bd
[Tue Sep 15 13:01:43 2026]  nvme_queue_rqs+0x11f/0x240 [nvme]
[Tue Sep 15 13:01:43 2026]  blk_mq_dispatch_queue_requests+0x19a/0x1d0
[Tue Sep 15 13:01:43 2026]  blk_mq_flush_plug_list+0xbf/0x1d0
[Tue Sep 15 13:01:43 2026]  __blk_flush_plug+0xef/0x150
[Tue Sep 15 13:01:43 2026]  __submit_bio+0x196/0x250
[Tue Sep 15 13:01:43 2026]  submit_bio_noacct_nocheck+0x136/0x370
[Tue Sep 15 13:01:43 2026]  submit_bio_noacct+0x1b5/0x5e0
[Tue Sep 15 13:01:43 2026]  submit_bio+0xb1/0x110
[Tue Sep 15 13:01:43 2026]  blkdev_direct_IO+0x305/0x730
[Tue Sep 15 13:01:43 2026]  blkdev_write_iter+0x228/0x350
[Tue Sep 15 13:01:43 2026]  ? rw_verify_area+0x57/0x190
[Tue Sep 15 13:01:43 2026]  aio_write+0x161/0x2a0
[Tue Sep 15 13:01:43 2026]  ? fdget+0xd0/0x100
[Tue Sep 15 13:01:43 2026]  io_submit_one+0x6b7/0xa10
[Tue Sep 15 13:01:43 2026]  ? srso_alias_return_thunk+0x5/0xfbef5
[Tue Sep 15 13:01:43 2026]  ? io_submit_one+0x6b7/0xa10
[Tue Sep 15 13:01:43 2026]  ? srso_alias_return_thunk+0x5/0xfbef5
[Tue Sep 15 13:01:43 2026]  ? srso_alias_return_thunk+0x5/0xfbef5
[Tue Sep 15 13:01:43 2026]  __x64_sys_io_submit+0x90/0x1f0
[Tue Sep 15 13:01:43 2026]  ? srso_alias_return_thunk+0x5/0xfbef5
[Tue Sep 15 13:01:43 2026]  x64_sys_call+0x2258/0x2390
[Tue Sep 15 13:01:43 2026]  do_syscall_64+0x10b/0x14e0
[Tue Sep 15 13:01:43 2026]  ? srso_alias_return_thunk+0x5/0xfbef5
[Tue Sep 15 13:01:43 2026]  ? irqentry_exit+0xb2/0x710
[Tue Sep 15 13:01:43 2026]  ? srso_alias_return_thunk+0x5/0xfbef5
[Tue Sep 15 13:01:43 2026]  ? srso_alias_return_thunk+0x5/0xfbef5
[Tue Sep 15 13:01:43 2026]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[Tue Sep 15 13:01:43 2026] RIP: 0033:0x74cbd81114f9
[Tue Sep 15 13:01:43 2026] Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d e7 68 0d 00 f7 d8 64 89 01 48
[Tue Sep 15 13:01:43 2026] RSP: 002b:00007ffc569527f8 EFLAGS: 00000246 ORIG_RAX: 00000000000000d1
[Tue Sep 15 13:01:43 2026] RAX: ffffffffffffffda RBX: 000074cbd4e762b8 RCX: 000074cbd81114f9
[Tue Sep 15 13:01:43 2026] RDX: 00007ffc56952840 RSI: 0000000000000001 RDI: 000074cbd42b6000
[Tue Sep 15 13:01:43 2026] RBP: 000074cbd42b6000 R08: 0000000000000000 R09: 0000000000000400
[Tue Sep 15 13:01:43 2026] R10: 00007ffc56952840 R11: 0000000000000246 R12: 0000000000000001
[Tue Sep 15 13:01:43 2026] R13: 000000000000000b R14: 00007ffc56952840 R15: 00005f19a6fdd670
[Tue Sep 15 13:01:43 2026]  </TASK>
[Tue Sep 15 13:01:43 2026] ---[ end trace 0000000000000000 ]---

And the local NVME disk is reporting errors:
Code: 
[Tue Sep 15 13:01:43 2026] I/O error, dev nvme4n1, sector 867825668 op 0x1:(WRITE) flags 0xc800 phys_seg 33 prio class 2
[Tue Sep 15 13:01:43 2026] I/O error, dev nvme4n1, sector 867825738 op 0x1:(WRITE) flags 0xc800 phys_seg 33 prio class 2
[Tue Sep 15 13:01:43 2026] I/O error, dev nvme4n1, sector 867825668 op 0x1:(WRITE) flags 0xc800 phys_seg 33 prio class 2
[Tue Sep 15 13:01:43 2026] I/O error, dev nvme4n1, sector 867825738 op 0x1:(WRITE) flags 0xc800 phys_seg 33 prio class 2
[Tue Sep 15 13:01:43 2026] I/O error, dev nvme4n1, sector 867825668 op 0x1:(WRITE) flags 0xc800 phys_seg 33 prio class 2
[Tue Sep 15 13:01:43 2026] I/O error, dev nvme4n1, sector 867825738 op 0x1:(WRITE) flags 0xc800 phys_seg 33 prio class 2
[Tue Sep 15 13:01:43 2026] I/O error, dev nvme4n1, sector 867825668 op 0x1:(WRITE) flags 0xc800 phys_seg 33 prio class 2
[Tue Sep 15 13:01:43 2026] I/O error, dev nvme4n1, sector 867825738 op 0x1:(WRITE) flags 0xc800 phys_seg 33 prio class 2
[Tue Sep 15 13:01:43 2026] I/O error, dev nvme4n1, sector 867825668 op 0x1:(WRITE) flags 0xc800 phys_seg 33 prio class 2
[Tue Sep 15 13:01:43 2026] I/O error, dev nvme4n1, sector 867825738 op 0x1:(WRITE) flags 0xc800 phys_seg 33 prio class 2

But the blast radius was also beyond the VMs that had their disks migrated to local storage. (Some VMs with Ceph based virtual disks also froze and had to be hard reset. Some were fine and could be live migrated to the other hosts.)

We are still early in the troubleshooting, but the nvme4n1 disk seems fine and we've managed to evacuate the node again.

System info:
- Supermicro H12SSW
- AMD EPYC 7313P
- Micron 7450 NVME disks for both local storage and Ceph.

Code: 
# pveversion -v
proxmox-ve: 9.2.0 (running kernel: 7.0.14-17-pve)
pve-manager: 9.2.20 (running version: 9.2.20/49318c671b82f31e)
proxmox-kernel-helper: 9.2.0
proxmox-kernel-7.0.14-17-pve-signed: 7.0.14-17
proxmox-kernel-7.0: 7.0.14-17
proxmox-kernel-6.8.12-43-pve-signed: 6.8.12-43
proxmox-kernel-6.8: 6.8.12-43
proxmox-kernel-6.8.12-17-pve-signed: 6.8.12-17
ceph: 19.2.6-pve4
ceph-fuse: 19.2.6-pve4
corosync: 3.1.10-pve3
criu: 4.1.1-1
frr-pythontools: 10.6.1-1+pve3
ifupdown: residual config
ifupdown2: 3.3.0-1+pmx12
libjs-extjs: 7.0.0-7
libproxmox-acme-perl: 1.7.2
libproxmox-backup-qemu0: 2.0.2
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.1.1
libpve-apiclient-perl: 3.4.3
libpve-cluster-api-perl: 9.1.6
libpve-cluster-perl: 9.1.6
libpve-common-perl: 9.2.2
libpve-guest-common-perl: 6.0.5
libpve-http-server-perl: 6.0.5
libpve-network-perl: 1.6.7
libpve-notify-perl: 9.1.6
libpve-rs-perl: 0.15.3
libpve-storage-perl: 9.1.10
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2+pmx1
lxc-pve: 7.0.0-2
lxcfs: 7.0.0-pve1
novnc-pve: 1.7.0-2
openvswitch-switch: 3.5.0-1+b1
proxmox-backup-client: 4.2.5-1
proxmox-backup-file-restore: 4.2.5-1
proxmox-backup-restore-image: 1.0.0
proxmox-enterprise-support-keyring: 1.1
proxmox-firewall: 1.2.3
proxmox-kernel-helper: 9.2.0
proxmox-mail-forward: 1.0.3
proxmox-mini-journalreader: 1.7
proxmox-offline-mirror-helper: 0.7.4
proxmox-widget-toolkit: 5.2.8
pve-cluster: 9.1.6
pve-container: 6.1.14
pve-docs: 9.2.10
pve-edk2-firmware: not correctly installed
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.6
pve-firmware: 3.18-6
pve-ha-manager: 5.2.5
pve-i18n: 3.10.0
pve-qemu-kvm: 11.0.3-3
pve-xtermjs: 6.0.0-2
qemu-server: 9.2.7
smartmontools: 7.5-pve2
spiceterm: 3.4.2
swtpm: 0.8.0+pve3
vncterm: 1.9.2
zfsutils-linux: 2.4.4-pve1

# nvme smart-log /dev/nvme4n1
Smart Log for NVME device:nvme4n1 namespace-id:ffffffff
critical_warning                        : 0
temperature                             : 80 °F (300 K)
available_spare                         : 100%
available_spare_threshold               : 10%
percentage_used                         : 0%
endurance group critical warning summary: 0
Data Units Read                         : 175317523 (89.76 TB)
Data Units Written                      : 137649418 (70.48 TB)
host_read_commands                      : 2467882661
host_write_commands                     : 4225430125
controller_busy_time                    : 5161
power_cycles                            : 41
power_on_hours                          : 23535
unsafe_shutdowns                        : 27
media_errors                            : 0
num_err_log_entries                     : 0
Warning Temperature Time                : 0
Critical Composite Temperature Time     : 0
Temperature Sensor 1                    : 91 °F (306 K)
Temperature Sensor 2                    : 82 °F (301 K)
Temperature Sensor 3                    : 80 °F (300 K)
Thermal Management T1 Trans Count       : 0
Thermal Management T2 Trans Count       : 0
Thermal Management T1 Total Time        : 0
Thermal Management T2 Total Time        : 0

Also, this is not optimal, that the Bookworm repo has only Ceph 19.2.5 and Trixie has 19.2.6, but I don't think it has anything to do with this issue...

Code: 
# ceph versions
{
    "mon": {
        "ceph version 19.2.5 (8b5bd86e28fcdf705ab8ac294202c1a26de9ac5a) squid (stable)": 2,
        "ceph version 19.2.6 (7198987e0613d69ffb68c62f59cf70ce51a10eb7) squid (stable)": 1
    },
    "mgr": {
        "ceph version 19.2.5 (8b5bd86e28fcdf705ab8ac294202c1a26de9ac5a) squid (stable)": 2,
        "ceph version 19.2.6 (7198987e0613d69ffb68c62f59cf70ce51a10eb7) squid (stable)": 1
    },
    "osd": {
        "ceph version 19.2.5 (8b5bd86e28fcdf705ab8ac294202c1a26de9ac5a) squid (stable)": 8,
        "ceph version 19.2.6 (7198987e0613d69ffb68c62f59cf70ce51a10eb7) squid (stable)": 4
    },
    "overall": {
        "ceph version 19.2.5 (8b5bd86e28fcdf705ab8ac294202c1a26de9ac5a) squid (stable)": 12,
        "ceph version 19.2.6 (7198987e0613d69ffb68c62f59cf70ce51a10eb7) squid (stable)": 6
    }
}
 
Hi @RobertKih

Good news... It is not the disk. Those EIOs come from the kernel's DMA mapping code, before the write ever reaches the drive. That's why your smart-log is clean.

This looks like a block layer bug in 7.0.14. Split bios lose their segment gap tracking, NVMe tries to coalesce unaligned segments, and dma_iova_link bails. This was fixed upstream in Aug: https://git.kernel.org/pub/scm/linu.../?id=96c8ea3c5add7920b3c43840d1ea76b3354c8d2d

The WARN only prints once, every following hit is just an I/O error line.

Your Ceph freezes are probably the same thing since the OSDs on that node sit on the same NVMe. You could try adding iommu=pt to the cmdline (PCI passthrough still works with it) until Proxmox picks up the fix.

Cheers

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Thanks, that is good information.

I've set the iommu=pt and rebooted the node. No issues so far after that but we'll pause the upgrade and keep the node unloaded until a patched kernel exists.
 
You must log in or register to reply here.
Top Bottom
Back