vm's constantly rebooting ever 6 minutes

[doanne]

Running Virtual Environment 8.1.4

Two out of the three VM's I have running are constantly rebooting every 6 minutes. There are fairly new machines with no cron set up.
101 is running Ubuntu 22.04.4
102 is running Rocky Linux release 9.3


I used to run kvm directly on a cenos before switching to proxmox. I have never had such a strange issue like this before.

Here is the messages from 102
Mar 6 21:59:47 server chronyd[701]: Selected source 192.48.105.15 (2.rocky.pool.ntp.org)
Mar 6 22:00:35 server /usr/sbin/irqbalance[697]: Cannot change IRQ 28 affinity: Input/output error
Mar 6 22:00:35 server /usr/sbin/irqbalance[697]: IRQ 28 affinity is now unmanaged
Mar 6 22:03:03 server chronyd[701]: Selected source 147.182.158.78 (2.rocky.pool.ntp.org)
Mar 6 22:04:25 server kernel: Linux version 5.14.0-362.18.1.el9_3.0.1.x86_64 (mockbuild@iad1-prod-build001.bld.equ.rockylinux.org) (gcc (GCC) 11.4.1 20230605 (Red Hat 11.4.1-2), GNU ld version 2.35.2-42.el9) #1 SMP PREEMPT_DYNAMIC Sun Feb 11 13:49:23 UTC 2024
Mar 6 22:04:25 server kernel: The list of certified hardware and cloud instances for Enterprise Linux 9 can be viewed at the Red Hat Ecosystem Catalog, https://catalog.redhat.com.
Mar 6 22:04:25 server kernel: Command line: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-362.18.1.el9_3.0.1.x86_64 root=/dev/mapper/rl-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rl-swap rd.lvm.lv=rl/root rd.lvm.lv=rl/swap
Mar 6 22:04:25 server kernel: x86/fpu: x87 FPU will use FXSAVE
Mar 6 22:04:25 server kernel: signal: max sigframe size: 1440
Mar 6 22:04:25 server kernel: BIOS-provided physical RAM map:
Mar 6 22:04:25 server kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
Mar 6 22:04:25 server kernel: BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
Mar 6 22:04:25 server kernel: BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
Mar 6 22:04:25 server kernel: BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable
Mar 6 22:04:25 server kernel: BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved
Mar 6 22:04:25 server kernel: BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
Mar 6 22:04:25 server kernel: BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
Mar 6 22:04:25 server kernel: BIOS-e820: [mem 0x0000000100000000-0x000000013fffffff] usable
Mar 6 22:04:25 server kernel: NX (Execute Disable) protection: active

With no real indication of why it rebooted.

reboot system boot Thu Mar 7 08:52 still running 5.14.0-362.18.1.el9_3.0.1.x86_64
reboot system boot Thu Mar 7 08:46 still running 5.14.0-362.18.1.el9_3.0.1.x86_64
reboot system boot Thu Mar 7 08:40 still running 5.14.0-362.18.1.el9_3.0.1.x86_64
reboot system boot Thu Mar 7 08:34 still running 5.14.0-362.18.1.el9_3.0.1.x86_64
reboot system boot Thu Mar 7 08:28 still running 5.14.0-362.18.1.el9_3.0.1.x86_64
reboot system boot Thu Mar 7 08:22 still running 5.14.0-362.18.1.el9_3.0.1.x86_64
reboot system boot Thu Mar 7 08:16 still running 5.14.0-362.18.1.el9_3.0.1.x86_64
reboot system boot Thu Mar 7 08:11 still running 5.14.0-362.18.1.el9_3.0.1.x86_64
reboot system boot Thu Mar 7 08:05 still running 5.14.0-362.18.1.el9_3.0.1.x86_64
reboot system boot Thu Mar 7 07:59 still running 5.14.0-362.18.1.el9_3.0.1.x86_64
reboot system boot Thu Mar 7 07:53 still running 5.14.0-362.18.1.el9_3.0.1.x86_64
reboot system boot Thu Mar 7 07:47 still running 5.14.0-362.18.1.el9_3.0.1.x86_64
reboot system boot Thu Mar 7 07:41 still running 5.14.0-362.18.1.el9_3.0.1.x86_64
reboot system boot Thu Mar 7 07:35 still running 5.14.0-362.18.1.el9_3.0.1.x86_64
reboot system boot Thu Mar 7 07:29 still running 5.14.0-362.18.1.el9_3.0.1.x86_64
reboot system boot Thu Mar 7 07:24 still running 5.14.0-362.18.1.el9_3.0.1.x86_64
reboot system boot Thu Mar 7 07:18 still running 5.14.0-362.18.1.el9_3.0.1.x86_64
reboot system boot Thu Mar 7 07:12 still running 5.14.0-362.18.1.el9_3.0.1.x86_64
reboot system boot Thu Mar 7 07:06 still running 5.14.0-362.18.1.el9_3.0.1.x86_64
reboot system boot Thu Mar 7 07:00 still running 5.14.0-362.18.1.el9_3.0.1.x86_64
reboot system boot Thu Mar 7 06:54 still running 5.14.0-362.18.1.el9_3.0.1.x86_64
reboot system boot Thu Mar 7 06:48 still running 5.14.0-362.18.1.el9_3.0.1.x86_64
reboot system boot Thu Mar 7 06:42 still running 5.14.0-362.18.1.el9_3.0.1.x86_64

 

[BobhWasatch]

About 90% of the time this is because the admin has installed some kind of monitoring script and didn't set up the qemu-agent correctly in the VM.

 

[doanne]

No monitoring, but also no qemu-agent, is qemu-agent required on the VM to be stable?

 

[leesteken]

It's not uncommon that people forget or don't know because they followed an unofficial guide. Something is stopping and starting the VM (via the Proxmox API or command-line), and this is not normal Proxmox behavior. How did you install Proxmox. What third-party scripts did you run )and you and we don't know what they did)?

No monitoring, but also no qemu-agent, is qemu-agent required on the VM to be stable?

No that is not needed for normal operations. But some third-party monitoring software might need it...

 

[doanne]

I installed using a ISO loaded onto USB.
Good news is after I followed https://pve.proxmox.com/wiki/Qemu-guest-agent and set up the agent I have been up for 60 minutes now so it looks like adding the agent stopped the constant reboots.