VM freezes irregularly

[catcraft]

Hello Guys,

I want you to inform, that my Toptom N6005 Box seems now to be stable. I will test the 6.1 Kernel soon.

One limitation: I run opnsense without suricata enabled since my last reboot.

My current uptime is 27 days and ~ 16h

Kernel 5.19.17-1-pve
Diabled C-States (bios and bootloader)
microcode : 0x2400001f

 

[MrBlowfish]

6.1 seems to be stable on Intel NUC11ATKPE (N6005)
5 days running, previously it crashed around 0.5-1.5 days

 

[juched]

Having upgraded to 6.1 on Wednesday and fully rebooting for other updates yesterday, had a random reboot of my hosted opnsense VM today. Cannot tell yet why, but 6.1 may not be solid on the Topton N5105 firewall device.

Edit: Ran 5.19 for weeks with no restart/crash. Decided to now switch my box to bare metal OPNSense, little value in chasing this for now.

Looks similar to the older 5.15 crashes to me:

Spoiler: Logs from inside FreeBSD / Opnsense

Fatal double fault
rip 0xffffffff8115cbd4 rsp 0xfffffe000eda3000 rbp 0xfffffe000eda3000
rax 0 rdx 0 rbx 0xfffffe000eda31f0
rcx 0x1f rsi 0xfffffe000eda3010 rdi 0xc
r8 0 r9 0xfffff800031ac000 r10 0xfffff800031ac040
r11 0x1 r12 0xfffff800036e1a90 r13 0x1
r14 0xfffffe00105ee3a0 r15 0xc rflags 0x10006
cs 0x20 ss 0x28 ds 0x3b es 0x3b fs 0x13 gs 0x1b
fsbase 0x800b9e120 gsbase 0xffffffff82c10000 kgsbase 0
cpuid = 0; apic id = 00
panic: double fault
cpuid = 0
time = 1671916006
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xffffffff81f5f4c0
vpanic() at vpanic+0x17f/frame 0xffffffff81f5f510
panic() at panic+0x43/frame 0xffffffff81f5f570
dblfault_handler() at dblfault_handler+0x1ce/frame 0xffffffff81f5f630
Xdblfault() at Xdblfault+0xd7/frame 0xffffffff81f5f630
--- trap 0x17, rip = 0xffffffff8115cbd4, rsp = 0xfffffe000eda3000, rbp = 0xfffffe000eda3000 ---
trap_check() at trap_check+0x4/frame 0xfffffe000eda3000
calltrap() at calltrap+0x8/frame 0xfffffe000eda3000

.... repeated many times.

--- interrupt, rip = 0xffffffff8111b9d6, rsp = 0xfffffe000eda6dd0, rbp = 0xfffffe000eda6dd0 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6/frame 0xfffffe000eda6dd0
acpi_cpu_idle() at acpi_cpu_idle+0x2ef/frame 0xfffffe000eda6e10
cpu_idle_acpi() at cpu_idle_acpi+0x3e/frame 0xfffffe000eda6e30
cpu_idle() at cpu_idle+0x9f/frame 0xfffffe000eda6e50
sched_idletd() at sched_idletd+0x4e1/frame 0xfffffe000eda6ef0
fork_exit() at fork_exit+0x7e/frame 0xfffffe000eda6f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe000eda6f30
--- trap 0xee250b8, rip = 0xffffffff80c313af, rsp = 0, rbp = 0xffffffff8133a248 ---
mi_startup() at mi_startup+0xdf/frame 0xffffffff8133a248
KDB: enter: panic

 

[R1CH]

VM is up for 36 days now with the host using the powersave governor. It certainly seems to be related to C-states or other CPU power state features.

 

[Mincka]

It's great to see people manage to overcome this issue. However, it does not seem to be the same fix for everyone.
In my case, I did not have to change anything in the BIOS or install software to manage frequency.
I installed the Intel Microcode from the current branch + 6.1 Linux Kernel and it did the trick.
I also made sure the temperatures stay low with good thermal paste and a fan.

 

[R1CH]

Unfortunately 6.1 seems to have made things worse for me, my OpenWrt VM froze after< 48 hours. Such a frustrating issue, really wish we knew what the root cause was.

 

[s0x]

Hi

On my end, still on kernel 5.19.17-1-pve, with 32 days uptime, two VMs, OPNsense ( 3 (1 sockets, 3 cores) [host,flags=-pcid;-spec-ctrl;-ssbd;+aes] [cpuunits=2048] ) with VirtIO NICs and HomeAssistant, and two LXC containers (PiHole and TP-Link Omada Controller, based on Ubuntu 22.04).

root@pve:~# last reboot | head -n 1
reboot system boot 5.19.17-1-pve Sat Nov 26 20:14 still running
root@pve:~# uptime
11:48:25 up 32 days, 15:33, 1 user, load average: 0.39, 0.32, 0.29
root@pve:~# uname -a
Linux pve 5.19.17-1-pve #1 SMP PREEMPT_DYNAMIC PVE 5.19.17-1 (Mon, 14 Nov 2022 20:25:12 x86_64 GNU/Linux

Host is a Topton N5105 (CW-6000) with i225 B3 NICs, BIOS date 29/09/2022, 2x8GB RAM, 1x NVMe SSD WD SN530. Extra Noctua 40mm fan 12v (NF-A4x10 PWM) as exhaust is inaudible (as intake the noise would be noticeable).

But I've applied several options to the kernel cmdline, see below.

Kernel cmdline options:

intel_idle.max_cstate=1 (disable C-states below 1 (such as C3))
intel_iommu=on iommu=pt (Enable iommu, since at the begining I was going to use passthrough NICs to the OPNsense VM, but ended up using Virtio NICs, while testing for the crashes, and kept them)
mitigations=off (Self explanatory)
i915.enable_guc=2 ( Enable low-power H264 encoding, https://01.org/linuxgraphics/downloads/firmware , https://jellyfin.org/docs/general/administration/hardware-acceleration/#intel-gen9-and-gen11-igpus )
initcall_blacklist=sysfb_init ( GPU passthrough , https://wiki.tozo.info/books/server/page/proxmox-gpu-passthrough )
nvme_core.default_ps_max_latency_us=14900 ( https://esc.sh/blog/nvme-ssd-and-linux-freezing/ )

Also, due to i2c-6 NAK errors ( [Sat Nov 26 20:14:37 2022] i2c i2c-6: sendbytes: NAK bailout. ) related to the iGPU I've connected a dummy HDMI dongle after confirming that with a monitor plugged in the errors stoppped and so did system crashes, but by then I've had already applied other kernel parameters.

Didn't test if those were related to the enabling of i915 GuC/HuC or not.

And due to errors related to the NVMe SSD (WD SN530 M.2 2242) I've applied the nvme_core.default_ps_max_latency_us parameter as well.

[Tue Nov 29 11:46:52 2022] nvme 0000:01:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[Tue Nov 29 11:46:52 2022] nvme 0000:01:00.0: device [15b7:5008] error status/mask=00000001/0000e000
[Tue Nov 29 11:46:52 2022] nvme 0000:01:00.0: [ 0] RxErr

- edit -
Also updated the intel microcode:

root@pve:~# dmesg -T | grep microcode
[Sat Nov 26 20:14:32 2022] microcode: microcode updated early to revision 0x24000023, date = 2022-02-19
[Sat Nov 26 20:14:32 2022] SRBDS: Vulnerable: No microcode
[Sat Nov 26 20:14:33 2022] microcode: sig=0x906c0, pf=0x1, revision=0x24000023
[Sat Nov 26 20:14:33 2022] microcode: Microcode Update Driver: v2.2.

- edit -

Hopefully someone else finds this information helpful.

 

[magingale]

Bought a Topton N5105 (model A) FMI07 board, latest so it seems. Also facing crashed of VM's, its random selected.

Upgraded to the latest microcode available:

Old:

root@pve:~# cat /proc/cpuinfo | grep micro
microcode       : 0x1d
microcode       : 0x1d
microcode       : 0x1d
microcode       : 0x1d

New:

root@pve:~# cat /proc/cpuinfo | grep micro
microcode       : 0x24000023
microcode       : 0x24000023
microcode       : 0x24000023
microcode       : 0x24000023

wget http://http.us.debian.org/debian/pool/non-free/i/intel-microcode/intel-microcode_3.20221108.1_amd64.deb
dpkg -i intel-microcode_3.20221108.1_amd64.deb
update-initramfs -u -k all
reboot

Running kernel 5.19 opt-in.

 

[magingale]

Still random vm reboots and crashes. Any way to get Proxmox involved to look deeper? All the vms has run for weeks on an intel NUC without a glitch.

Looking forward how we can help? If we all move to bare metal installations it won’t help solving the root cause.

 

[magingale]

since this seems to be hardware related and we don't have any affected hardware - could any of you try the steps from my last comment in the bug?

https://bugzilla.proxmox.com/show_bug.cgi?id=4188#c14

I’m new to bug reading. I’m confused which commands you refer to? Could you read my last post; we need help, really appreciate your time

 

[enricoross98]

Same problem here, I have a Topton N5105 machine with 4xi225 interfaces from Aliexpress and my VMs still freezes after upgrading to the "5.19.17-1-pve" Kernel. Today I have upgraded also the microcode as described in post #348, let's see if this changes something.
My Proxmox machine runs: an OPNsense VM with 2 NICs (PCI passthrough), OpenMediaVault and a Ubuntu Server VM with Docker and a lot of containers with some heavy ones such as FrigateNVR (TensorFlow recognition on RTSP streams). The last one seems to freeze the most.

Last month i found a temporary solution to this problem by creating a script in Python that stops and start VMs that gets stuck, if you have another machine (such as a Pi) you can try it: https://github.com/EnricoRoss98/ProxmoxWatchdog

 

[nikybiasion]

I've the same experience on two server, ML380 G9 and ML160 G10. All worked fine until i've upgraded from 7.2 to 7.3 and kernel 5.19, then the vms randomly freeze with 100%cpu usage. Now i've revert to kernel 5.15, i report back if this fixes the freezes

 

[AdriftAtlas]

HW:
HUNSN RJ03
4 Core Intel N5105
4 x Intel 2.5GbE I226-V LAN

SW:
Proxmox 7.3-4
Linux prox 5.15.83-1-pve #1 SMP PVE 5.15.83-1 (2022-12-15T00:00Z) x86_64 GNU/Linux


VM Config:
pfSense Plus 22.05
2 Cores, 4GB RAM assigned
WAN/LAN in PCI Passthrough
Hardware checksum, TCP segmentation, and large receive offload enabled
No traffic shaping enabled

I do have powersave governor enabled to reduce heat.

Got a kernel panic from pfSense today, looks similar to others. Something to do with CPU sleep states.

Some guy on another forum claims to have fixed it by using the following CPU settings, haven't tried it yet:

kvm64: -mitigation flags +aes = stable
2 (1 sockets, 2 cores) [kvm64,flags=-pcid;-spec-ctrl;-ssbd;-ibpb;-virt-ssbd;-amd-ssbd;-amd-no-ssb;+aes]
https://forum.opnsense.org/index.php?PHPSESSID=fjen1nhgf7l4jm6vn6u9la014h&topic=30230.15

kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address    = 0x1008e
fault code        = supervisor write data, page not present
instruction pointer    = 0x20:0xffffffff80da2d71
stack pointer            = 0x28:0xfffffe0025782b00
frame pointer            = 0x28:0xfffffe0025782b60
code segment        = base 0x0, limit 0xfffff, type 0x1b
            = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags    = resume, IOPL = 0
current process        = 11 (idle: cpu0)
trap number        = 12
panic: page fault
cpuid = 0
time = 1672654637
KDB: enter: panic

db:0:kdb.enter.default>  bt

Tracing pid 11 tid 100003 td 0xfffff8000520d000
kdb_enter() at kdb_enter+0x37/frame 0xfffffe00257828c0
vpanic() at vpanic+0x194/frame 0xfffffe0025782910
panic() at panic+0x43/frame 0xfffffe0025782970
trap_fatal() at trap_fatal+0x38f/frame 0xfffffe00257829d0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe0025782a30
calltrap() at calltrap+0x8/frame 0xfffffe0025782a30
--- trap 0xc, rip = 0xffffffff80da2d71, rsp = 0xfffffe0025782b00, rbp = 0xfffffe0025782b60 ---
callout_process() at callout_process+0x1b1/frame 0xfffffe0025782b60
handleevents() at handleevents+0x188/frame 0xfffffe0025782ba0
cpu_activeclock() at cpu_activeclock+0x70/frame 0xfffffe0025782bd0
cpu_idle() at cpu_idle+0xa8/frame 0xfffffe0025782bf0
sched_idletd() at sched_idletd+0x326/frame 0xfffffe0025782cb0
fork_exit() at fork_exit+0x7e/frame 0xfffffe0025782cf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0025782cf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

db:0:kdb.enter.default>  alltrace

Tracing command sleep pid 35878 tid 100632 td 0xfffff80057237740
sched_switch() at sched_switch+0x606/frame 0xfffffe003671b9c0
mi_switch() at mi_switch+0xdb/frame 0xfffffe003671b9f0
sleepq_catch_signals() at sleepq_catch_signals+0x3f3/frame 0xfffffe003671ba40
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame 0xfffffe003671ba80
_sleep() at _sleep+0x1c6/frame 0xfffffe003671bb00
kern_clock_nanosleep() at kern_clock_nanosleep+0x1c1/frame 0xfffffe003671bb80
sys_nanosleep() at sys_nanosleep+0x3b/frame 0xfffffe003671bbc0
amd64_syscall() at amd64_syscall+0x387/frame 0xfffffe003671bcf0
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe003671bcf0
--- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x80038c9fa, rsp = 0x7fffffffec18, rbp = 0x7fffffffec60 ---

Tracing command sh pid 15762 tid 100600 td 0xfffff80016b8e000
sched_switch() at sched_switch+0x606/frame 0xfffffe00366cb970
mi_switch() at mi_switch+0xdb/frame 0xfffffe00366cb9a0
sleepq_catch_signals() at sleepq_catch_signals+0x3f3/frame 0xfffffe00366cb9f0
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00366cba20
_sleep() at _sleep+0x1f1/frame 0xfffffe00366cbaa0
pipe_read() at pipe_read+0x3fe/frame 0xfffffe00366cbb10
dofileread() at dofileread+0x95/frame 0xfffffe00366cbb50
sys_read() at sys_read+0xc0/frame 0xfffffe00366cbbc0
amd64_syscall() at amd64_syscall+0x387/frame 0xfffffe00366cbcf0
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00366cbcf0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80044f03a, rsp = 0x7fffffffe3d8, rbp = 0x7fffffffe900 ---

Tracing command sh pid 15703 tid 100633 td 0xfffff80057237000
sched_switch() at sched_switch+0x606/frame 0xfffffe0036720800
mi_switch() at mi_switch+0xdb/frame 0xfffffe0036720830
sleepq_catch_signals() at sleepq_catch_signals+0x3f3/frame 0xfffffe0036720880
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe00367208b0
_sleep() at _sleep+0x1f1/frame 0xfffffe0036720930
kern_wait6() at kern_wait6+0x59e/frame 0xfffffe00367209c0
sys_wait4() at sys_wait4+0x7d/frame 0xfffffe0036720bc0
amd64_sy

 

[Chelmet]

Just to throw my hat in too, I'm getting regular crashes with OPNsense on PVE 7.3-3. It was typically about 23 hours between crashes, so I upgradede the linux kernel to 6.1.0-1; managed 3 days before a crash, but crash it did. I wouldn't mind quite so much if I could set up a watchdog to start the VM on failure, but I can't get it to work.

Using a Topton N6005 unit with 4x i226 and similar settings to everyone else experiencing this issue.

 

[AdriftAtlas]

Kernel cmdline options:

intel_idle.max_cstate=1 (disable C-states below 1 (such as C3))

So if you reenable C-States in the kernel then your VMs will crash? Have you tried it after all of your other changes?

So far I installed the intel-microcode package from the non-free Debian repository and will see what happens. Kernel is stock for now.

I really don't want to disable C-states as it will increase power usage and heat. This is what I'm getting at idle with no fans, enabled Enhanced C-States, enabled ASPM, and powersave governor:

coretemp-isa-0000
Adapter: ISA adapter
Package id 0:  +35.0°C  (high = +105.0°C, crit = +105.0°C)
Core 0:        +31.0°C  (high = +105.0°C, crit = +105.0°C)
Core 1:        +31.0°C  (high = +105.0°C, crit = +105.0°C)
Core 2:        +31.0°C  (high = +105.0°C, crit = +105.0°C)
Core 3:        +31.0°C  (high = +105.0°C, crit = +105.0°C)

acpitz-acpi-0
Adapter: ACPI interface
temp1:        +39.0°C  (crit = +119.0°C)

nvme-pci-0100
Adapter: PCI adapter
Composite:    +35.9°C  (low  = -273.1°C, high = +81.8°C)
                       (crit = +84.8°C)
Sensor 1:     +35.9°C  (low  = -273.1°C, high = +65261.8°C)
Sensor 2:     +36.9°C  (low  = -273.1°C, high = +65261.8°C)

 

[misery]

Still random vm reboots and crashes. Any way to get Proxmox involved to look deeper? All the vms has run for weeks on an intel NUC without a glitch.

Looking forward how we can help? If we all move to bare metal installations it won’t help solving the root cause.

Maybe we need to create a bugreport somewhere else.
I don't use Proxmox but I manually use AlpineLinux with libvirt/qemu/kvm. I have the same crashes with OPNSense VM. It takes some days until the crash. I upgraded to 6.1.1 Kernel now (microcode updated, C-States enabled) and it runs 5 days at the moment.
I have a CWWK N5105 (v5) with 4 i226 NICs.

So the crash seems not Proxmox related.

KVM internal error. Suberror: 3
extra data[0]: 0x0000000080000b0e
extra data[1]: 0x0000000000000031
extra data[2]: 0x0000000000000083
extra data[3]: 0x0000000800a68fe0
extra data[4]: 0x0000000000000000
RAX=0000000800a68120 RBX=fffffe000916a090 RCX=00000000c0000101 RDX=00000000ffffffff
RSI=0000000000000015 RDI=fffffe000916a090 RBP=fffffe000916a080 RSP=fffffe0009169fb0
R8 =0000000000000000 R9 =00000000ffffffff R10=0000000000000000 R11=0000000800a61b40
R12=00007fffffff6540 R13=0000000000000000 R14=0000000800bb60a0 R15=0000000800a68120
RIP=ffffffff811338c1 RFL=00010082 [--S----] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =003b 0000000000000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
CS =0020 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]
SS =0000 0000000000000000 ffffffff 00c00000
DS =003b 0000000000000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
FS =0013 0000000800288120 ffffffff 00c0f300 DPL=3 DS   [-WA]
GS =001b ffffffff82612000 ffffffff 00c0f300 DPL=3 DS   [-WA]
LDT=0000 0000000000000000 ffffffff 00c00000
TR =0048 ffffffff82612384 00002068 00008b00 DPL=0 TSS64-busy
GDT=     ffffffff826123ec 00000067
IDT=     ffffffff81f5d710 00000fff
CR0=80050033 CR2=ffffffff811338c1 CR3=0000000800a68120 CR4=003506e0
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000d01
Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
2022-12-28T10:44:48.612670Z qemu-system-x86_64: terminating on signal 15 from pid 2968 (/usr/sbin/libvirtd)
2022-12-28 10:44:49.229+0000: shutting down, reason=destroyed
2022-12-28 10:44:52.406+0000: starting up libvirt version: 8.9.0, qemu version: 7.1.0, kernel: 5.15.85-0-lts,

 

[tcamargo]

I had constant VM reboots (OPNsense) and at least 2 hard freezes (need to manually restart VM). Upgrade to kernel 5.19 improved situation a bit: no hard freezes and eventual VM rebootsevery 2-3 days. Enabled non-free repo and installed intel-microde. System is running flawlessly for more than 8d now. Keep you posted.

Thanks for all your support!

 

[magingale]

I had constant VM reboots (OPNsense) and at least 2 hard freezes (need to manually restart VM). Upgrade to kernel 5.19 improved situation a bit: no hard freezes and eventual VM rebootsevery 2-3 days. Enabled non-free repo and installed intel-microde. System is running flawlessly for more than 8d now. Keep you posted.

Thanks for all your support!

Which microcode is loaded?

 

[magingale]

Maybe we need to create a bugreport somewhere else.
I don't use Proxmox but I manually use AlpineLinux with libvirt/qemu/kvm. I have the same crashes with OPNSense VM. It takes some days until the crash. I upgraded to 6.1.1 Kernel now (microcode updated, C-States enabled) and it runs 5 days at the moment.
I have a CWWK N5105 (v5) with 4 i226 NICs.

So the crash seems not Proxmox related.

KVM internal error. Suberror: 3
extra data[0]: 0x0000000080000b0e
extra data[1]: 0x0000000000000031
extra data[2]: 0x0000000000000083
extra data[3]: 0x0000000800a68fe0
extra data[4]: 0x0000000000000000
RAX=0000000800a68120 RBX=fffffe000916a090 RCX=00000000c0000101 RDX=00000000ffffffff
RSI=0000000000000015 RDI=fffffe000916a090 RBP=fffffe000916a080 RSP=fffffe0009169fb0
R8 =0000000000000000 R9 =00000000ffffffff R10=0000000000000000 R11=0000000800a61b40
R12=00007fffffff6540 R13=0000000000000000 R14=0000000800bb60a0 R15=0000000800a68120
RIP=ffffffff811338c1 RFL=00010082 [--S----] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =003b 0000000000000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
CS =0020 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]
SS =0000 0000000000000000 ffffffff 00c00000
DS =003b 0000000000000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
FS =0013 0000000800288120 ffffffff 00c0f300 DPL=3 DS   [-WA]
GS =001b ffffffff82612000 ffffffff 00c0f300 DPL=3 DS   [-WA]
LDT=0000 0000000000000000 ffffffff 00c00000
TR =0048 ffffffff82612384 00002068 00008b00 DPL=0 TSS64-busy
GDT=     ffffffff826123ec 00000067
IDT=     ffffffff81f5d710 00000fff
CR0=80050033 CR2=ffffffff811338c1 CR3=0000000800a68120 CR4=003506e0
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000d01
Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
2022-12-28T10:44:48.612670Z qemu-system-x86_64: terminating on signal 15 from pid 2968 (/usr/sbin/libvirtd)
2022-12-28 10:44:49.229+0000: shutting down, reason=destroyed
2022-12-28 10:44:52.406+0000: starting up libvirt version: 8.9.0, qemu version: 7.1.0, kernel: 5.15.85-0-lts,

Could you elaborate more on your statement? I have VMS random rebooting the hypervisor remains up. only VMs hanging or rebooting randomly every 2-3 days or 1 day. I updated to kernel 6.1 today.

If your hypervisor (the host OS) reboots it might be hardware related?

 

[misery]

Could you elaborate more on your statement? I have VMS random rebooting the hypervisor remains up. only VMs hanging or rebooting randomly every 2-3 days or 1 day. I updated to kernel 6.1 today.

If your hypervisor (the host OS) reboots it might be hardware related?

My hypervisor (host OS) does not crash - only the VM.
Why not proxmox related? Because I do not use proxmox and I have the same problem. So it is probably the underlying system that is used by Proxmox: qemu or kvm.