Updated to 8.2 - DMA error

[JayTee75]

I just upgrded HPE Microserver gen8 to new version (8.2), and got this after while with system stuck:




Any ideas?

 

[JayTee75]

I add some params to grub:

BOOT_IMAGE=/boot/vmlinuz-6.8.4-2-pve root=/dev/mapper/pve-root ro quiet intel_iommu = on, igfx_off iommu=pt intremap=off

I will see and pray

 

[SpotNick]

Same here!

Gen8 microserver with Smart Array P222 as RAID controller. ...i have some bad experiences downgrading kernels. may be a RAID Controller malfunction or something like this...
You solution eliminates DMAR ERROR?

 

[JayTee75]

Adding kernel boot parameter fixed it for me.

(I am using HP smart array P410)

 

[SpotNick]

Adding kernel boot parameter fixed it for me.

(I am using HP smart array P410)

In my case I had to downgrade to kernel version 6.5.13-5-pve, and seems it works!
Thanks!

 

[OtxiMOU]

hello. The same thing has happened to me too. deactivating the vt-t solves it for me

 

[Supaman]

Same error here, i tried #6 from OtxiMOU: boot was ok, root login possible, but after a few mnites same eror on shell.

My Solutio nhas been:
Enter Bios - System - Processor Options

intel vt: default enabled - unchanged
intel vd-d: default enabled -> changed to disabled

Hope this heps.

 

[OtxiMOU]

Hello supaman, in the end I decided to leave and fix the version 8.1 kernel so it doesn't give errors and I can have vt and vd activated

 

[Supaman]

Dear Staff,

this error still occurs with the lastest kernal 6.8.8-2-pve.

After system boot, root login appears and around 5 seconds later i get 17 lines "... DMAR: ERROR: PTE for vPFN ..." like in the screenshot from inital post here.

System is a HP Microserver Gen 8, 16 GB Ram, PCIe Slot has a Dual Intel 10GBe Card, latest J06 Bios and BMC v2.82 iLO Firmware.
System had run fine with PVE 8.1 / Kernal 6.5.x.

Solution until kernal 6.8.8-2-pve had been is disabling vt-d in bios.
System and VMs had been a little bit slower, but worked for me.

Now wtih 6.8.8-2-pve + disabled vt-d i need to switch off KVM virtualisation and can not select host-cpu anymore,
i need to switch to "CPU Typ= Ivy Bridge or something like that", wich causes the VM runnig really slow.

Not good.


PVE System Log for the suspecting part:

Jul 03 16:21:36 pve4 kernel: DMAR: ERROR: DMA PTE for vPFN 0xedf7f already set (to edf7f003 not 127c4f001)
Jul 03 16:21:36 pve4 kernel: ------------[ cut here ]------------
Jul 03 16:21:36 pve4 kernel: WARNING: CPU: 6 PID: 220 at drivers/iommu/intel/iommu.c:2214 __domain_mapping+0x375/0x4f0
Jul 03 16:21:36 pve4 kernel: Modules linked in: ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter nf_tables 8021q garp mrp bonding tls softdog sunrpc nfnetlink_log binfmt_misc nfnetlink intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha256_ssse3 sha1_ssse3 aesni_intel crypto_simd cryptd ipmi_ssif rapl intel_cstate pcspkr serio_raw acpi_power_meter acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler mgag200 input_leds joydev i2c_algo_bit hpilo ie31200_edac mac_hid zfs(PO) spl(O) vhost_net vhost vhost_iotlb tap efi_pstore dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq hid_generic usbkbd usbmouse usbhid hid dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c xhci_pci ahci xhci_pci_renesas gpio_ich crc32_pclmul uhci_hcd i40e psmouse ehci_pci xhci_hcd lpc_ich tg3 libahci ehci_hcd
Jul 03 16:21:36 pve4 kernel: CPU: 6 PID: 220 Comm: kworker/u128:5 Tainted: P W IO 6.8.8-2-pve #1
Jul 03 16:21:36 pve4 kernel: Hardware name: HP ProLiant MicroServer Gen8, BIOS J06 04/04/2019
Jul 03 16:21:36 pve4 kernel: Workqueue: writeback wb_workfn (flush-252:1)
Jul 03 16:21:36 pve4 kernel: RIP: 0010:__domain_mapping+0x375/0x4f0
Jul 03 16:21:36 pve4 kernel: Code: 48 89 c2 4c 89 4d b0 48 c7 c7 b8 5d 63 ae e8 82 6e 6d ff 8b 05 a0 6b 9e 01 4c 8b 4d b0 85 c0 74 09 83 e8 01 89 05 8f 6b 9e 01 <0f> 0b e9 fe fe ff ff 8b 45 c4 4c 89 ee 4c 89 f7 8d 58 01 48 8b 45
Jul 03 16:21:36 pve4 kernel: RSP: 0018:ffffb5e78101f260 EFLAGS: 00010006
Jul 03 16:21:36 pve4 kernel: RAX: 0000000000000003 RBX: 0000000000000001 RCX: 0000000000000000
Jul 03 16:21:36 pve4 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Jul 03 16:21:36 pve4 kernel: RBP: ffffb5e78101f2f0 R08: 0000000000000000 R09: ffff90ab8819e800
Jul 03 16:21:36 pve4 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff90ab8819ebf8
Jul 03 16:21:36 pve4 kernel: R13: ffff90ab81b09500 R14: 0000000000000027 R15: 0000000127c4f001
Jul 03 16:21:36 pve4 kernel: FS: 0000000000000000(0000) GS:ffff90ae7df00000(0000) knlGS:0000000000000000
Jul 03 16:21:36 pve4 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 03 16:21:36 pve4 kernel: CR2: 0000629c2e354b88 CR3: 0000000104236001 CR4: 00000000001726f0
Jul 03 16:21:36 pve4 kernel: Call Trace:
Jul 03 16:21:36 pve4 kernel: <TASK>
Jul 03 16:21:36 pve4 kernel: ? show_regs+0x6d/0x80
Jul 03 16:21:36 pve4 kernel: ? __warn+0x89/0x160
Jul 03 16:21:36 pve4 kernel: ? __domain_mapping+0x375/0x4f0
Jul 03 16:21:36 pve4 kernel: ? report_bug+0x17e/0x1b0
Jul 03 16:21:36 pve4 kernel: ? handle_bug+0x46/0x90
Jul 03 16:21:36 pve4 kernel: ? exc_invalid_op+0x18/0x80
Jul 03 16:21:36 pve4 kernel: ? asm_exc_invalid_op+0x1b/0x20
Jul 03 16:21:36 pve4 kernel: ? __domain_mapping+0x375/0x4f0
Jul 03 16:21:36 pve4 kernel: ? check_preempt_wakeup_fair+0x165/0x1d0
Jul 03 16:21:36 pve4 kernel: intel_iommu_map_pages+0xe1/0x140
Jul 03 16:21:36 pve4 kernel: __iommu_map+0x121/0x280
Jul 03 16:21:36 pve4 kernel: iommu_map_sg+0xbf/0x1f0
Jul 03 16:21:36 pve4 kernel: iommu_dma_map_sg+0x45f/0x4f0
Jul 03 16:21:36 pve4 kernel: ? __blk_rq_map_sg+0x30a/0x4f0
Jul 03 16:21:36 pve4 kernel: __dma_map_sg_attrs+0x35/0xd0
Jul 03 16:21:36 pve4 kernel: dma_map_sg_attrs+0xe/0x30
Jul 03 16:21:36 pve4 kernel: ata_qc_issue+0x170/0x2c0
Jul 03 16:21:36 pve4 kernel: ? __pfx_ata_scsi_rw_xlat+0x10/0x10
Jul 03 16:21:36 pve4 kernel: ? __pfx_ata_scsi_rw_xlat+0x10/0x10
Jul 03 16:21:36 pve4 kernel: __ata_scsi_queuecmd+0x21c/0x520
Jul 03 16:21:36 pve4 kernel: ata_scsi_queuecmd+0x44/0x80
Jul 03 16:21:36 pve4 kernel: scsi_queue_rq+0x3d2/0xc40
Jul 03 16:21:36 pve4 kernel: blk_mq_dispatch_rq_list+0x137/0x810
Jul 03 16:21:36 pve4 kernel: ? sbitmap_get+0x73/0x180
Jul 03 16:21:36 pve4 kernel: __blk_mq_sched_dispatch_requests+0x41f/0x5d0
Jul 03 16:21:36 pve4 kernel: ? sbitmap_get_shallow+0x68/0x140
Jul 03 16:21:36 pve4 kernel: blk_mq_sched_dispatch_requests+0x2c/0x70
Jul 03 16:21:36 pve4 kernel: blk_mq_run_hw_queue+0x1bf/0x210
Jul 03 16:21:36 pve4 kernel: blk_mq_flush_plug_list.part.0+0x187/0x5c0
Jul 03 16:21:36 pve4 kernel: blk_add_rq_to_plug+0x14d/0x1b0
Jul 03 16:21:36 pve4 kernel: blk_mq_submit_bio+0x634/0x750
Jul 03 16:21:36 pve4 kernel: __submit_bio+0xb3/0x1c0
Jul 03 16:21:36 pve4 kernel: submit_bio_noacct_nocheck+0x17a/0x390
Jul 03 16:21:36 pve4 kernel: submit_bio_noacct+0x1f3/0x650
Jul 03 16:21:36 pve4 kernel: submit_bio+0xb2/0x110
Jul 03 16:21:36 pve4 kernel: ext4_io_submit+0x24/0x50
Jul 03 16:21:36 pve4 kernel: ext4_do_writepages+0x564/0xe10
Jul 03 16:21:36 pve4 kernel: ext4_writepages+0xb5/0x190
Jul 03 16:21:36 pve4 kernel: do_writepages+0xd0/0x1f0
Jul 03 16:21:36 pve4 kernel: __writeback_single_inode+0x44/0x370
Jul 03 16:21:36 pve4 kernel: writeback_sb_inodes+0x211/0x510
Jul 03 16:21:36 pve4 kernel: __writeback_inodes_wb+0x54/0x100
Jul 03 16:21:36 pve4 kernel: ? queue_io+0x115/0x120
Jul 03 16:21:36 pve4 kernel: wb_writeback+0x2a8/0x320
Jul 03 16:21:36 pve4 kernel: wb_workfn+0x2c7/0x4d0
Jul 03 16:21:36 pve4 kernel: process_one_work+0x16d/0x350
Jul 03 16:21:36 pve4 kernel: worker_thread+0x306/0x440
Jul 03 16:21:36 pve4 kernel: ? __pfx_worker_thread+0x10/0x10
Jul 03 16:21:36 pve4 kernel: kthread+0xf2/0x120
Jul 03 16:21:36 pve4 kernel: ? __pfx_kthread+0x10/0x10
Jul 03 16:21:36 pve4 kernel: ret_from_fork+0x47/0x70
Jul 03 16:21:36 pve4 kernel: ? __pfx_kthread+0x10/0x10
Jul 03 16:21:36 pve4 kernel: ret_from_fork_asm+0x1b/0x30
Jul 03 16:21:36 pve4 kernel: </TASK>
Jul 03 16:21:36 pve4 kernel: ---[ end trace 0000000000000000 ]---
Jul 03 16:21:36 pve4 kernel: DMAR: ERROR: DMA PTE for vPFN 0xedf80 already set (to edf80003 not 127c50001)

 

[fiona]

Hi,

Dear Staff,

this error still occurs with the lastest kernal 6.8.8-2-pve.

After system boot, root login appears and around 5 seconds later i get 17 lines "... DMAR: ERROR: PTE for vPFN ..." like in the screenshot from inital post here.

System is a HP Microserver Gen 8, 16 GB Ram, PCIe Slot has a Dual Intel 10GBe Card, latest J06 Bios and BMC v2.82 iLO Firmware.
System had run fine with PVE 8.1 / Kernal 6.5.x.

Solution until kernal 6.8.8-2-pve had been is disabling vt-d in bios.
System and VMs had been a little bit slower, but worked for me.

Now wtih 6.8.8-2-pve + disabled vt-d i need to switch off KVM virtualisation and can not select host-cpu anymore,
i need to switch to "CPU Typ= Ivy Bridge or something like that", wich causes the VM runnig really slow.

Not good.


PVE System Log for the suspecting part:

did you already try disabling the intel_iommu setting on the kernel commandline? The default for the setting changed in kernel 6.8 and your backtrace mentions IOMMU: https://pve.proxmox.com/wiki/Roadmap#8.2-known-issues

 

[mitch55]

Hi all.

Someone could help me here ?

I had the same problem and trying to resolve my problem other ones appeared...

 

[Supaman]

So far, 2 worarounds possible:

1) disabling the intel_iommu setting on the GRUB kernel commandline
2) pin kernel to 6.5

With #1, System is slower, with #2 there may be incomaptibilities.

Still hoping for a working 6.8+ kernal, because the HP Gen8 microserver is still a very good piece of hardware.

 

[mitch55]

For my part, I have disabled the intel_iommu.

With #1, System is slower,

Are VMs slowed down too?
Is the system significantly slowed down?

By the way, what is the use of "intel_iommu=on"?

 

[pintero]

Hello!
Just noticed this error too, but contrary to others, my Gen8 microserver with 2 VM's is working just fine. Running 8.2.4 with all the latest updates. No kernel downgrades and no features disabled in Bios.
Remote console seems to be stuck, but Proxmox Gui shell and SSH is working.
TBH I rarely access the host console, so I guess it's been like this for months. No RAID, just AHCI.
Interesting!