Proxmox often doesn't start the machines after back-up

[Czam]

stop mode
This mode provides the highest consistency of the backup, at the cost of a short downtime in the VM operation. It works by executing an orderly shutdown of the VM, and then runs a background QEMU process to backup the VM data. After the backup is started, the VM goes to full operation mode if it was previously running. Consistency is guaranteed by using the live backup feature.

So in this case we are using Stop mode...
Is it possible to hang when it receives the stop mode....and that to disrupt the other 3 machines in the cluster?
But on the other hand the back-up was created....I don't know if back-up will start at all without the machine to be restated.

This is the backup from today, that is passing without any problems...

starting task UPIDroxmaster:0014DEBC:01F91197:6446EDD5:qmshutdown:10224:root@pam:
Apr 25 00:00:05 proxmaster qm[1367740]: shutdown VM 10224: UPIDroxmaster:0014DEBC:01F91197:6446EDD5:qmshutdown:10224:root@pam:
Apr 25 00:00:11 proxmaster pveproxy[1367728]: send HUP to 2089
Apr 25 00:00:11 proxmaster pveproxy[2089]: received signal HUP
Apr 25 00:00:11 proxmaster pveproxy[2089]: server closing
Apr 25 00:00:11 proxmaster pveproxy[2089]: server shutdown (restart)
Apr 25 00:00:11 proxmaster systemd[1]: Reloaded PVE API Proxy Server.
Apr 25 00:00:11 proxmaster systemd[1]: Reloading PVE SPICE Proxy Server.
Apr 25 00:00:12 proxmaster spiceproxy[1367768]: send HUP to 2097
Apr 25 00:00:12 proxmaster spiceproxy[2097]: received signal HUP
Apr 25 00:00:12 proxmaster spiceproxy[2097]: server closing
Apr 25 00:00:12 proxmaster spiceproxy[2097]: server shutdown (restart)
Apr 25 00:00:12 proxmaster systemd[1]: Reloaded PVE SPICE Proxy Server.
Apr 25 00:00:13 proxmaster systemd[1]: Stopping Proxmox VE firewall logger...
Apr 25 00:00:13 proxmaster pvefw-logger[1002710]: received terminate request (signal)
Apr 25 00:00:13 proxmaster pvefw-logger[1002710]: stopping pvefw logger
Apr 25 00:00:13 proxmaster spiceproxy[2097]: restarting server
Apr 25 00:00:13 proxmaster spiceproxy[2097]: starting 1 worker(s)
Apr 25 00:00:13 proxmaster spiceproxy[2097]: worker 1367799 started
Apr 25 00:00:13 proxmaster systemd[1]: pvefw-logger.service: Succeeded.
Apr 25 00:00:13 proxmaster systemd[1]: Stopped Proxmox VE firewall logger.
Apr 25 00:00:13 proxmaster systemd[1]: pvefw-logger.service: Consumed 9.092s CPU time.
Apr 25 00:00:13 proxmaster systemd[1]: Starting Proxmox VE firewall logger...
Apr 25 00:00:13 proxmaster pvefw-logger[1367801]: starting pvefw logger
Apr 25 00:00:13 proxmaster systemd[1]: Started Proxmox VE firewall logger.
Apr 25 00:00:14 proxmaster pveproxy[2089]: restarting server
Apr 25 00:00:14 proxmaster pveproxy[2089]: starting 3 worker(s)
Apr 25 00:00:14 proxmaster pveproxy[2089]: worker 1367803 started
Apr 25 00:00:14 proxmaster pveproxy[2089]: worker 1367804 started
Apr 25 00:00:14 proxmaster pveproxy[2089]: worker 1367805 started
Apr 25 00:00:14 proxmaster systemd[1]: logrotate.service: Succeeded.
Apr 25 00:00:14 proxmaster systemd[1]: Finished Rotate log files.
Apr 25 00:00:15 proxmaster kernel: pcieport 0000:00:03.0: PME: Spurious native interrupt!
Apr 25 00:00:18 proxmaster spiceproxy[291025]: worker exit
Apr 25 00:00:18 proxmaster spiceproxy[2097]: worker 291025 finished
Apr 25 00:00:19 proxmaster pveproxy[1277007]: worker exit
Apr 25 00:00:19 proxmaster pveproxy[1265371]: worker exit
Apr 25 00:00:19 proxmaster pveproxy[1267960]: worker exit
Apr 25 00:00:19 proxmaster pveproxy[2089]: worker 1277007 finished
Apr 25 00:00:19 proxmaster pveproxy[2089]: worker 1267960 finished
Apr 25 00:00:19 proxmaster pveproxy[2089]: worker 1265371 finished
Apr 25 00:00:21 proxmaster systemd[1]: man-db.service: Succeeded.
Apr 25 00:00:21 proxmaster systemd[1]: Finished Daily man-db regeneration.
Apr 25 00:00:32 proxmaster kernel: pcieport 0000:00:03.0: PME: Spurious native interrupt!
Apr 25 00:00:55 proxmaster QEMU[1002996]: kvm: terminating on signal 15 from pid 1280 (/usr/sbin/qmeventd)
Apr 25 00:00:55 proxmaster kernel: vmbr0: port 4(tap10224i0) entered disabled state
Apr 25 00:00:56 proxmaster systemd[1]: 10224.scope: Succeeded.
Apr 25 00:00:56 proxmaster systemd[1]: 10224.scope: Consumed 9h 23min 55.768s CPU time.
Apr 25 00:00:56 proxmaster qm[1367739]: <root@pam> end task UPIDroxmaster:0014DEBC:01F91197:6446EDD5:qmshutdown:10224:root@pam: OK
Apr 25 00:00:56 proxmaster systemd[1]: Started 10224.scope.
Apr 25 00:00:57 proxmaster systemd-udevd[1367971]: Using default interface naming scheme 'v247'.
Apr 25 00:00:57 proxmaster systemd-udevd[1367971]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Apr 25 00:00:57 proxmaster qmeventd[1367972]: Starting cleanup for 10224
Apr 25 00:00:57 proxmaster qmeventd[1367972]: trying to acquire lock...
Apr 25 00:00:58 proxmaster kernel: device tap10224i0 entered promiscuous mode
Apr 25 00:00:58 proxmaster kernel: vmbr0: port 4(tap10224i0) entered blocking state
Apr 25 00:00:58 proxmaster kernel: vmbr0: port 4(tap10224i0) entered disabled state
Apr 25 00:00:58 proxmaster kernel: vmbr0: port 4(tap10224i0) entered blocking state
Apr 25 00:00:58 proxmaster kernel: vmbr0: port 4(tap10224i0) entered forwarding state
Apr 25 00:00:58 proxmaster qmeventd[1367972]: OK
Apr 25 00:00:58 proxmaster qmeventd[1367972]: vm still running
Apr 25 00:02:39 proxmaster kernel: usb 1-1.7.4: reset low-speed USB device number 7 using ehci-pci
Apr 25 00:03:41 proxmaster smartd[1279]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 67 to 70
Apr 25 00:03:41 proxmaster smartd[1279]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 123 to 128
Apr 25 00:03:41 proxmaster smartd[1279]: Device: /dev/sdc [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 66 to 69
Apr 25 00:03:44 proxmaster pmxcfs[1626]: [dcdb] notice: data verification successful
Apr 25 00:17:01 proxmaster CRON[1371676]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Apr 25 00:17:01 proxmaster CRON[1371677]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Apr 25 00:17:01 proxmaster CRON[1371676]: pam_unix(cron:session): session closed for user root
Apr 25 00:19:17 proxmaster pvescheduler[1367707]: INFO: Finished Backup of VM 10224 (00:19:14)
Apr 25 00:19:17 proxmaster pvescheduler[1367707]: INFO: Backup job finished successfully


And this is from 21st when it was shutdown and didnt came back-up

<root@pam> starting task UPIDroxmaster:003B8403:0CC91D58:6441A7DD:qmshutdown:10224:root@pam:
Apr 21 00:00:13 proxmaster qm[3900419]: shutdown VM 10224: UPIDroxmaster:003B8403:0CC91D58:6441A7DD:qmshutdown:10224:root@pam:
Apr 21 00:00:13 proxmaster spiceproxy[3567553]: worker exit
Apr 21 00:00:13 proxmaster spiceproxy[2095]: worker 3567553 finished
Apr 21 00:00:13 proxmaster pveproxy[3567558]: worker exit
Apr 21 00:00:13 proxmaster pveproxy[3567559]: worker exit
Apr 21 00:00:13 proxmaster pveproxy[3567560]: worker exit
Apr 21 00:00:14 proxmaster pveproxy[2087]: worker 3567560 finished
Apr 21 00:00:14 proxmaster pveproxy[2087]: worker 3567558 finished
Apr 21 00:00:14 proxmaster pveproxy[2087]: worker 3567559 finished
Apr 21 00:00:15 proxmaster kernel: pcieport 0000:00:03.0: PME: Spurious native interrupt!
Apr 21 00:00:26 proxmaster systemd[1]: man-db.service: Succeeded.
Apr 21 00:00:26 proxmaster systemd[1]: Finished Daily man-db regeneration.
Apr 21 00:00:54 proxmaster kernel: pcieport 0000:00:03.0: PME: Spurious native interrupt!
Apr 21 00:01:01 proxmaster QEMU[3567775]: kvm: terminating on signal 15 from pid 1279 (/usr/sbin/qmeventd)
Apr 21 00:01:02 proxmaster kernel: vmbr0: port 3(tap10224i0) entered disabled state
Apr 21 00:01:02 proxmaster systemd[1]: 10224.scope: Succeeded.
Apr 21 00:01:02 proxmaster systemd[1]: 10224.scope: Consumed 9h 48min 25.119s CPU time.
Apr 21 00:01:03 proxmaster qm[3900407]: <root@pam> end task UPIDroxmaster:003B8403:0CC91D58:6441A7DD:qmshutdown:10224:root@pam: OK
Apr 21 00:01:03 proxmaster systemd[1]: Started 10224.scope.
Apr 21 00:01:03 proxmaster systemd-udevd[3900599]: Using default interface naming scheme 'v247'.
Apr 21 00:01:03 proxmaster systemd-udevd[3900599]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Apr 21 00:01:03 proxmaster qmeventd[3900601]: Starting cleanup for 10224
Apr 21 00:01:04 proxmaster qmeventd[3900601]: trying to acquire lock...
Apr 21 00:01:04 proxmaster kernel: device tap10224i0 entered promiscuous mode
Apr 21 00:01:04 proxmaster kernel: vmbr0: port 3(tap10224i0) entered blocking state
Apr 21 00:01:04 proxmaster kernel: vmbr0: port 3(tap10224i0) entered disabled state
Apr 21 00:01:04 proxmaster kernel: vmbr0: port 3(tap10224i0) entered blocking state
Apr 21 00:01:04 proxmaster kernel: vmbr0: port 3(tap10224i0) entered forwarding state
Apr 21 00:01:05 proxmaster qmeventd[3900601]: OK
Apr 21 00:01:05 proxmaster qmeventd[3900601]: vm still running
Apr 21 00:09:21 proxmaster smartd[1278]: Device: /dev/sdb [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 123 to 125
Apr 21 00:09:21 proxmaster smartd[1278]: Device: /dev/sdc [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 66 to 65
Apr 21 00:09:24 proxmaster pmxcfs[1617]: [dcdb] notice: data verification successful
Apr 21 00:17:01 proxmaster CRON[3903828]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Apr 21 00:17:01 proxmaster CRON[3903829]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Apr 21 00:17:01 proxmaster CRON[3903828]: pam_unix(cron:session): session closed for user root
Apr 21 00:21:00 proxmaster pvescheduler[3900396]: INFO: Finished Backup of VM 10224 (00:20:49)
Apr 21 00:21:00 proxmaster pvescheduler[3900396]: INFO: Backup job finished successfully




Thank's to everyone that is still helping and helped with this, I got to understand a lot of how proxmox works in matter of days.

 

[Czam]

Found the problem, the kernel of Proxmox is restarting on that particular machine for some reason....it has nothing to do with the back-ups.
The reason they (VM's) are not starting is because the option "Start at boot" is not market as "yes"

Mar 22 03:43:01 unknow:
-- Reboot --
Mar 22 03:45:09 proxmaster kernel: Linux version 5.15.39-2-pve (build@proxmox) (gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP PVE 5.15.39-2 (Wed, 20 Jul 2022 17:22:19 +0200) ()

So now I have to search for reason, the kernel is restarting (mainly in the night).

 

[Nuke Bloodaxe]

How about this: https://forum.proxmox.com/threads/p...rly-since-7-3-7-4-upgrade.125499/#post-547997

 

[Czam]

Thank's for the reply, but the machine is with Intel CPU - Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz...
Also the crashes are happening 2-3 times a month.... (sometimes 2 weeks between, sometimes 3....sometimes 1...)..
The other (physical) machine is with Proxmox 7.1-7 -Linux 5.13.19-2-pve (Intel(R) Xeon(R) CPU X5650 @ 2.67GHz).
No problems.
I'm wondering if should rollback the kernel to 5.13 ( now is : 5.15.39-2 as from the logs above)...