New VM's crashing on startup?

[Tivum]

Hello there, I was hoping you guys could shed some light on what is happening.

I run PVE on a Dell Poweredge r710 with 2 Xeon CPU's and 128gb of RAM. Currently I have a media server, apps server and a backup server I'm attempting to get working.



I launch the backup VM and boot into Windows Server 2022 install, get a few minutes into it and the VM crashes. I don't see anything out of the ordinary in the syslog, or at least anything that points to what would be wrong.

Here is the syslog at the time of crash, it happened around the 19:46 mark.

Jan 14 19:45:47 ADK-XENON pvedaemon[1188]: worker 1025739 finished
Jan 14 19:45:47 ADK-XENON pvedaemon[1188]: starting 1 worker(s)
Jan 14 19:45:47 ADK-XENON pvedaemon[1188]: worker 1044334 started
Jan 14 19:46:11 ADK-XENON smartd[847]: Device: /dev/bus/2 [megaraid_disk_00] [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 80 to 81
Jan 14 19:46:11 ADK-XENON smartd[847]: Device: /dev/bus/2 [megaraid_disk_00] [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 66 to 68
Jan 14 19:46:11 ADK-XENON smartd[847]: Device: /dev/bus/2 [megaraid_disk_00] [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 34 to 32
Jan 14 19:46:11 ADK-XENON smartd[847]: Device: /dev/bus/2 [megaraid_disk_00] [SAT], SMART Usage Attribute: 195 Hardware_ECC_Recovered changed from 80 to 81
Jan 14 19:46:11 ADK-XENON smartd[847]: Device: /dev/bus/2 [megaraid_disk_01] [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 117 to 119
Jan 14 19:46:11 ADK-XENON smartd[847]: Device: /dev/bus/2 [megaraid_disk_02] [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 114 to 115
Jan 14 19:46:11 ADK-XENON smartd[847]: Device: /dev/bus/2 [megaraid_disk_04] [SAT], 1 Currently unreadable (pending) sectors
Jan 14 19:46:11 ADK-XENON smartd[847]: Device: /dev/bus/2 [megaraid_disk_04] [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 118 to 119
Jan 14 19:46:12 ADK-XENON smartd[847]: Device: /dev/bus/2 [megaraid_disk_05] [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 82 to 83
Jan 14 19:46:12 ADK-XENON smartd[847]: Device: /dev/bus/2 [megaraid_disk_05] [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 66 to 67
Jan 14 19:46:12 ADK-XENON smartd[847]: Device: /dev/bus/2 [megaraid_disk_05] [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 34 to 33
Jan 14 19:46:12 ADK-XENON smartd[847]: Device: /dev/bus/2 [megaraid_disk_05] [SAT], SMART Usage Attribute: 195 Hardware_ECC_Recovered changed from 82 to 83
Jan 14 19:46:32 ADK-XENON qm[1044694]: VM 102 qmp command failed - VM 102 qmp command 'guest-ping' failed - got timeout
Jan 14 19:46:39 ADK-XENON qm[1044849]: <root@pam> starting task UPID:ADK-XENON:000FF18F:0193FE34:65A4806F:qmstop:102:root@pam:
Jan 14 19:46:39 ADK-XENON qm[1044879]: stop VM 102: UPID:ADK-XENON:000FF18F:0193FE34:65A4806F:qmstop:102:root@pam:
Jan 14 19:46:39 ADK-XENON kernel: tap102i0: left allmulticast mode
Jan 14 19:46:39 ADK-XENON kernel: fwbr102i0: port 2(tap102i0) entered disabled state
Jan 14 19:46:40 ADK-XENON kernel: fwbr102i0: port 1(fwln102i0) entered disabled state
Jan 14 19:46:40 ADK-XENON kernel: vmbr0: port 4(fwpr102p0) entered disabled state
Jan 14 19:46:40 ADK-XENON kernel: fwln102i0 (unregistering): left allmulticast mode
Jan 14 19:46:40 ADK-XENON kernel: fwln102i0 (unregistering): left promiscuous mode
Jan 14 19:46:40 ADK-XENON kernel: fwbr102i0: port 1(fwln102i0) entered disabled state
Jan 14 19:46:40 ADK-XENON kernel: fwpr102p0 (unregistering): left allmulticast mode
Jan 14 19:46:40 ADK-XENON kernel: fwpr102p0 (unregistering): left promiscuous mode
Jan 14 19:46:40 ADK-XENON kernel: vmbr0: port 4(fwpr102p0) entered disabled state
Jan 14 19:46:40 ADK-XENON qmeventd[846]: read: Connection reset by peer
Jan 14 19:46:40 ADK-XENON pvedaemon[1038665]: <root@pam> end task UPID:ADK-XENON:000FEAA2:019397F4:65A47F69:vncproxy:102:root@pam: OK
Jan 14 19:46:40 ADK-XENON systemd[1]: 102.scope: Deactivated successfully.
Jan 14 19:46:40 ADK-XENON systemd[1]: 102.scope: Consumed 2min 36.130s CPU time.
Jan 14 19:46:40 ADK-XENON qm[1044849]: <root@pam> end task UPID:ADK-XENON:000FF18F:0193FE34:65A4806F:qmstop:102:root@pam: OK
Jan 14 19:46:41 ADK-XENON qmeventd[1044911]: Starting cleanup for 102
Jan 14 19:46:41 ADK-XENON qmeventd[1044911]: Finished cleanup for 102
Jan 14 19:46:46 ADK-XENON qm[1044938]: <root@pam> starting task UPID:ADK-XENON:000FF1CD:019400F5:65A48076:qmstart:102:root@pam:
Jan 14 19:46:46 ADK-XENON qm[1044941]: start VM 102: UPID:ADK-XENON:000FF1CD:019400F5:65A48076:qmstart:102:root@pam:
Jan 14 19:46:46 ADK-XENON systemd[1]: Started 102.scope.
Jan 14 19:46:47 ADK-XENON kernel: tap102i0: entered promiscuous mode
Jan 14 19:46:47 ADK-XENON kernel: vmbr0: port 4(fwpr102p0) entered blocking state

Could you guys help? Thanks

 

[fiona]

Hi,

Jan 14 19:46:32 ADK-XENON qm[1044694]: VM 102 qmp command failed - VM 102 qmp command 'guest-ping' failed - got timeout
Jan 14 19:46:39 ADK-XENON qm[1044849]: <root@pam> starting task UPID:ADK-XENON:000FF18F:0193FE34:65A4806F:qmstop:102:root@pam:
Jan 14 19:46:39 ADK-XENON qm[1044879]: stop VM 102: UPID:ADK-XENON:000FF18F:0193FE34:65A4806F:qmstop:102:root@pam:

that rather sounds like it was a controlled stop. If it were a crash, there would be no such task.

XENON:000FF1CD:019400F5:65A48076:qmstart:102:root@pam:
Jan 14 19:46:46 ADK-XENON qm[1044941]: start VM 102: UPID:ADK-XENON:000FF1CD:019400F5:65A48076:qmstart:102:root@pam:
Jan 14 19:46:46 ADK-XENON systemd[1]: Started 102.scope.

Based on the timing I assume this was also triggered automatically?

Do you have any monitoring script that will just kill/restart VMs if they don't respond to guest-agent ping? E.g. something like: https://forum.proxmox.com/threads/vm-resetting-after-every-6-minutes.127110/post-556621