[SOLVED] Please Help Interpret This Log Output

[Nollimox]

There are two items from a journalctl output that I am trying to wrap my head around to understand why? Please see below:

Oct 24 09:47:40 nolliprivatecloud kernel: Policy zone: Normal
Oct 24 09:47:40 nolliprivatecloud kernel: Kernel command line: initrd=\EFI\proxmox\6.2.16-3-pve\initrd.img-6.2.16-3-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs in>
Oct 24 09:47:40 nolliprivatecloud kernel: DMAR: IOMMU enabled
Oct 24 09:47:40 nolliprivatecloud kernel: DMAR: Unknown option - 'pt'
Oct 24 09:47:40 nolliprivatecloud kernel: Unknown kernel command line parameters "boot=zfs", will be passed to user space.

Here is what I have:
GNU nano 7.2 /etc/kernel/cmdline
root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on intel_iommu=pt

AND
Oct 24 10:21:23 nolliprivatecloud kernel: ------------[ cut here ]------------
Oct 24 10:21:23 nolliprivatecloud kernel: WARNING: CPU: 16 PID: 2527 at arch/x86/kvm/mmu/mmu.c:6949 kvm_nx_huge_page_recovery_worker+0x3c4/0x410 [kvm]
Oct 24 10:21:23 nolliprivatecloud kernel: Modules linked in: tcp_diag inet_diag ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_t>
Oct 24 10:21:23 nolliprivatecloud kernel: dell_wmi_descriptor intel_wmi_thunderbolt pcspkr wmi_bmof snd_timer mei_me snd soundcore mei ioatdma acpi_tad joyd>
Oct 24 10:21:23 nolliprivatecloud kernel: CPU: 16 PID: 2527 Comm: kvm-nx-lpage-re Tainted: P O 6.2.16-3-pve #1
Oct 24 10:21:23 nolliprivatecloud kernel: Hardware name: Dell Inc. Precision 5820 Tower/002KVM, BIOS 2.31.0 07/06/2023
Oct 24 10:21:23 nolliprivatecloud kernel: RIP: 0010:kvm_nx_huge_page_recovery_worker+0x3c4/0x410 [kvm]
Oct 24 10:21:23 nolliprivatecloud kernel: Code: ff 48 8b 45 c0 4c 39 e0 0f 85 e6 fd ff ff 48 89 df e8 e0 e7 f9 ff e9 ed fd ff ff 49 bc ff ff ff ff ff ff ff 7>
Oct 24 10:21:23 nolliprivatecloud kernel: RSP: 0018:ffffba022053be40 EFLAGS: 00010246
Oct 24 10:21:23 nolliprivatecloud kernel: RAX: 0000000000000000 RBX: ffffba02202f1000 RCX: 0000000000000000
Oct 24 10:21:23 nolliprivatecloud kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Oct 24 10:21:23 nolliprivatecloud kernel: RBP: ffffba022053bec0 R08: 0000000000000000 R09: 0000000000000000
Oct 24 10:21:23 nolliprivatecloud kernel: R10: ffff9781ba0bac38 R11: 0000000000000000 R12: ffffba022053be80
Oct 24 10:21:23 nolliprivatecloud kernel: R13: 0000000000000001 R14: 0000000000000001 R15: ffff9781ba0bacc8
Oct 24 10:21:23 nolliprivatecloud kernel: FS: 0000000000000000(0000) GS:ffff97a060000000(0000) knlGS:0000000000000000
Oct 24 10:21:23 nolliprivatecloud kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 24 10:21:23 nolliprivatecloud kernel: CR2: fffffe00d726a000 CR3: 0000001558a10001 CR4: 00000000003726e0
Oct 24 10:21:23 nolliprivatecloud kernel: Call Trace:
Oct 24 10:21:23 nolliprivatecloud kernel: <TASK>
Oct 24 10:21:23 nolliprivatecloud kernel: ? __pfx_kvm_nx_huge_page_recovery_worker+0x10/0x10 [kvm]
Oct 24 10:21:23 nolliprivatecloud kernel: kvm_vm_worker_thread+0x9d/0x1b0 [kvm]
Oct 24 10:21:23 nolliprivatecloud kernel: ? __pfx_kvm_vm_worker_thread+0x10/0x10 [kvm]
Oct 24 10:21:23 nolliprivatecloud kernel: kthread+0xe6/0x110
Oct 24 10:21:23 nolliprivatecloud kernel: ? __pfx_kthread+0x10/0x10
Oct 24 10:21:23 nolliprivatecloud kernel: ret_from_fork+0x29/0x50
Oct 24 10:21:23 nolliprivatecloud kernel: </TASK>
Oct 24 10:21:23 nolliprivatecloud kernel: ---[ end trace 0000000000000000 ]---

The second log entry didn't appear in my first built before the certificate accident that wipe out my webgui on this pass Sunday trying to pass a self-signed certificate to a Docker container which cause me to rebuild the PVE.

 

[leesteken]

Oct 24 09:47:40 nolliprivatecloud kernel: DMAR: Unknown option - 'pt'
Oct 24 09:47:40 nolliprivatecloud kernel: Unknown kernel command line parameters "boot=zfs", will be passed to user space.

Here is what I have:
GNU nano 7.2 /etc/kernel/cmdline
root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on intel_iommu=pt

intel_iommu=pt is indeed invalid. Maybe you wanted to write iommu=pt, which makes the IOMMU use the identity mapping for non-passed through devices? The remark about boot=zfs is normal, as it is not a valid kernel parameter but added by Proxmox.

 

[Nollimox]

intel_iommu=pt is indeed invalid. Maybe you wanted to write iommu=pt, which makes the IOMMU use the identity mapping for non-passed through devices? The remark about boot=zfs is normal, as it is not a valid kernel parameter but added by Proxmox.

Thank you Leesteken for responding. So, should it be: root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt?
Any idea on the second log entry?

 

[Nollimox]

Well, @leesteken , it seems that the proper way as of PVE8 is: root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt
No more complaining nor getting the second alert message in my first post.

Oct 24 19:40:16 nolliprivatecloud kernel: Policy zone: Normal
Oct 24 19:40:16 nolliprivatecloud kernel: Kernel command line: initrd=\EFI\proxmox\6.2.16-3-pve\initrd.img-6.2.16-3-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs in>
Oct 24 19:40:16 nolliprivatecloud kernel: DMAR: IOMMU enabled
Oct 24 19:40:16 nolliprivatecloud kernel: Unknown kernel command line parameters "boot=zfs", will be passed to user space.
Oct 24 19:40:16 nolliprivatecloud kernel: Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes, linear)
Oct 24 19:40:16 nolliprivatecloud kernel: Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes, linear)
Oct 24 19:40:16 nolliprivatecloud kernel: mem auto-init: stack:all(zero), heap allocn, heap freeff
Oct 24 19:40:16 nolliprivatecloud kernel: software IO TLB: area num 32.

Now, after the fix, got this below:

ct 24 19:40:16 nolliprivatecloud systemd[1]: Mounted sys-kernel-debug.mount - Kernel Debug File System.
Oct 24 19:40:16 nolliprivatecloud kernel: vfio_pci: unknown parameter '1028:0738' ignored
Oct 24 19:40:16 nolliprivatecloud kernel: vfio-pci 0000:04:00.0: device denylist disabled - allowing device 8086:0435.
Oct 24 19:40:16 nolliprivatecloud kernel: vfio_pci: add [8086:0435[ffffffff:ffffffff]] class 0x000000/00000000
Oct 24 19:40:16 nolliprivatecloud kernel: vfio_pci: device denylist disabled.
Oct 24 19:40:16 nolliprivatecloud systemd[1]: Mounted sys-kernel-tracing.mount - Kernel Trace File System

That's Dell's driver for the Intel QAT...I'll remove it.

 

[Nollimox]

It appeared again in the log, can someone please explain what this is alerting? I am not using IPv6

Oct 25 00:31:27 nolliprivatecloud kernel: ------------[ cut here ]------------
Oct 25 00:31:27 nolliprivatecloud kernel: WARNING: CPU: 15 PID: 8911 at arch/x86/kvm/mmu/mmu.c:6949 kvm_nx_huge_page_recovery_worker+0x3c4/0x410 [kvm]
Oct 25 00:31:27 nolliprivatecloud kernel: Modules linked in: tcp_diag inet_diag ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_t>
Oct 25 00:31:27 nolliprivatecloud kernel: dell_wmi_descriptor intel_wmi_thunderbolt wmi_bmof pcspkr snd_timer snd mei_me soundcore mei ioatdma acpi_tad joyd>
Oct 25 00:31:27 nolliprivatecloud kernel: CPU: 15 PID: 8911 Comm: kvm-nx-lpage-re Tainted: P O 6.2.16-3-pve #1
Oct 25 00:31:27 nolliprivatecloud kernel: Hardware name: Dell Inc. Precision 5820 Tower/002KVM, BIOS 2.31.0 07/06/2023
Oct 25 00:31:27 nolliprivatecloud kernel: RIP: 0010:kvm_nx_huge_page_recovery_worker+0x3c4/0x410 [kvm]
Oct 25 00:31:27 nolliprivatecloud kernel: Code: ff 48 8b 45 c0 4c 39 e0 0f 85 e6 fd ff ff 48 89 df e8 e0 e7 f9 ff e9 ed fd ff ff 49 bc ff ff ff ff ff ff ff 7>
Oct 25 00:31:27 nolliprivatecloud kernel: RSP: 0018:ffffb46545487e40 EFLAGS: 00010246
Oct 25 00:31:27 nolliprivatecloud kernel: RAX: 0000000000000000 RBX: ffffb46545429000 RCX: 0000000000000000
Oct 25 00:31:27 nolliprivatecloud kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Oct 25 00:31:27 nolliprivatecloud kernel: RBP: ffffb46545487ec0 R08: 0000000000000000 R09: 0000000000000000
Oct 25 00:31:27 nolliprivatecloud kernel: R10: ffff9e0fbc40f590 R11: 0000000000000000 R12: ffffb46545487e80
Oct 25 00:31:27 nolliprivatecloud kernel: R13: 0000000000000001 R14: 0000000000000001 R15: ffff9e0fbc40f620
Oct 25 00:31:27 nolliprivatecloud kernel: FS: 0000000000000000(0000) GS:ffff9e2e5ffc0000(0000) knlGS:0000000000000000
Oct 25 00:31:27 nolliprivatecloud kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 25 00:31:27 nolliprivatecloud kernel: CR2: 00007fde14c73438 CR3: 0000000549810001 CR4: 00000000003726e0
Oct 25 00:31:27 nolliprivatecloud kernel: Call Trace:
Oct 25 00:31:27 nolliprivatecloud kernel: <TASK>
Oct 25 00:31:27 nolliprivatecloud kernel: ? __pfx_kvm_nx_huge_page_recovery_worker+0x10/0x10 [kvm]
Oct 25 00:31:27 nolliprivatecloud kernel: kvm_vm_worker_thread+0x9d/0x1b0 [kvm]
Oct 25 00:31:27 nolliprivatecloud kernel: ? __pfx_kvm_vm_worker_thread+0x10/0x10 [kvm]
Oct 25 00:31:27 nolliprivatecloud kernel: kthread+0xe6/0x110
Oct 25 00:31:27 nolliprivatecloud kernel: ? __pfx_kthread+0x10/0x10
Oct 25 00:31:27 nolliprivatecloud kernel: ret_from_fork+0x29/0x50
Oct 25 00:31:27 nolliprivatecloud kernel: </TASK>
Oct 25 00:31:27 nolliprivatecloud kernel: ---[ end trace 0000000000000000 ]---

 

[Nollimox]

It's a bug: https://bugzilla.proxmox.com/show_bug.cgi?id=4833#c2

That was July...patch status says available...where? Any link?

 

[fabian]

you don't mention which kernel version you are using..

 

[beisser]

actually its in his last output:

CPU: 15 PID: 8911 Comm: kvm-nx-lpage-re Tainted: P O 6.2.16-3-pve #1

so 6.2.16-3-pve i would presume

 

[fabian]

that version predates the fix (see the linked bug report, which also contains a link to the commit fixing it in our kernel packages).

 

[Nollimox]

that version predates the fix (see the linked bug report, which also contains a link to the commit fixing it in our kernel packages).

The thing is I installed PVE 8 just this passed Sunday and had upgraded immediately...so I assumed any repair to the kernel should have been included. Just like, I get the CVE 2018-3646 warning, despite the computer was made January 2020...Dell must have fixed the issue.

 

[beisser]

installing pve just installs whatever is in the iso.
if you want to be up to date, you will need to make sure you have the correct repositories configured (no-subscription repo if you dont have a subscription) and then run an refresh/update from within the webgui. that will install the latest updates.

 

[fabian]

and also keep in mind - kernel updates require a reboot to become effective!

 

[Nollimox]

installing pve just installs whatever is in the iso.
if you want to be up to date, you will need to make sure you have the correct repositories configured (no-subscription repo if you dont have a subscription) and then run an refresh/update from within the webgui. that will install the latest updates.

I did, see post #10 (and had upgraded immediately).

 

[Nollimox]

and also keep in mind - kernel updates require a reboot to become effective!

But, I did also...yet the date is showing 06-17-2023

CPU(s) 20 x Intel(R) Xeon(R) W-2155 CPU @ 3.30GHz (1 Socket)

Kernel Version Linux 6.2.16-3-pve #1 SMP PREEMPT_DYNAMIC PVE 6.2.16-3 (2023-06-17T05:58Z)

PVE Manager Version pve-manager/8.0.3/bbf3993334bfa916

 

[leesteken]

But, I did also...yet the date is showing 06-17-2023

CPU(s) 20 x Intel(R) Xeon(R) W-2155 CPU @ 3.30GHz (1 Socket)

Kernel Version Linux 6.2.16-3-pve #1 SMP PREEMPT_DYNAMIC PVE 6.2.16-3 (2023-06-17T05:58Z)

PVE Manager Version pve-manager/8.0.3/bbf3993334bfa916

Yes that looks old, if you're using the no-subscription repository (I don't know the current version on the enterprise repository).

 

[Nollimox]

Yes that looks old, if you're using the no-subscription repository (I don't know the current version on the enterprise repository).

Well, I checked while you were typing and indeed there was another upgrade available for no-subscription, so I upgraded...that should resolve the issue.

CPU(s)

20 x Intel(R) Xeon(R) W-2155 CPU @ 3.30GHz (1 Socket)
Kernel Version

Linux 6.2.16-15-pve #1 SMP PREEMPT_DYNAMIC PMX 6.2.16-15 (2023-09-28T13:53Z)
PVE Manager Version

pve-manager/8.0.4/d258a813cfa6b390
Repository Status

Proxmox VE updates Non production-ready repository enabled!

 

[Nollimox]

To confirm, the kernel upgrade appeared to have resolved the issue as it had been running for a while and would have reappeared if it wasn't by now.