VM just stopped and process is still running

S

Semmo

Well-Known Member
May 27, 2019
35
4
48
39
Hi there!

I had this issue multiple times now and just want to find out whats going on. When i log into my proxmox host I see an unusual load on my node and several VMs had stopped. I can not start them again because the kvm (id 504) is still running with a high cpu load (see screenshot). An other VM (501) just stopped without any process still running.

Is there any way to find out in a logfile what happened here? It happened with oder VMs too. So it's not just that two VMs with the problem.

Thanks in advance!

screen.png
 
what does 'qm status 504 --verbose' say in this state?
check the syslog (/var/log/syslog or journalctl) and search for that vmid (504)
 
dcsapak said:
what does 'qm status 504 --verbose' say in this state?
check the syslog (/var/log/syslog or journalctl) and search for that vmid (504)
Click to expand...

Code: 
root@proxmox:/etc# qm status 504 --verbose
cpus: 1
disk: 0
diskread: 0
diskwrite: 0
maxdisk: 34359738368
maxmem: 536870912
mem: 0
name: syncthing
netin: 1278583348
netout: 1113400037
nics:
        tap504i0:
                netout: 102865911
                netin: 337972211
        tap504i2:
                netout: 2105098
                netin: 8820759
        tap504i1:
                netout: 1008429028
                netin: 931790378
qmpstatus: stopped
status: stopped
template:
uptime: 0
vmid: 504

The last journalctl entry to the machines are the start of the vm and my attempt to restart it today:
Code: 
Oct 16 08:39:55 proxmox pvedaemon[14510]: starting vnc proxy UPID:proxmox:000038AE:03986261:5DA6BB3B:vncproxy:504:sebastian@pam:
Oct 16 08:39:55 proxmox pvedaemon[4789]: <sebastian@pam> starting task UPID:proxmox:000038AE:03986261:5DA6BB3B:vncproxy:504:sebastian@pam:
Oct 16 08:39:56 proxmox qm[14512]: VM 504 qmp command failed - VM 504 not running
Oct 16 08:39:56 proxmox pvedaemon[4789]: <sebastian@pam> end task UPID:proxmox:000038AE:03986261:5DA6BB3B:vncproxy:504:sebastian@pam: Failed to run vncproxy.
Oct 16 08:39:59 proxmox pvedaemon[4788]: <sebastian@pam> starting task UPID:proxmox:0000399C:039863F5:5DA6BB3F:qmstart:504:sebastian@pam:
Oct 16 08:39:59 proxmox pvedaemon[14748]: start VM 504: UPID:proxmox:0000399C:039863F5:5DA6BB3F:qmstart:504:sebastian@pam:
Oct 16 08:39:59 proxmox systemd[1]: 504.scope: Succeeded.
Oct 16 08:39:59 proxmox systemd[1]: Stopped 504.scope.
Oct 16 08:40:04 proxmox pvedaemon[4788]: <sebastian@pam> end task UPID:proxmox:0000399C:039863F5:5DA6BB3F:qmstart:504:sebastian@pam: timeout waiting on systemd
[CODE]Oct 09 09:07:10 proxmox kernel: clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
Oct 09 09:09:48 proxmox pvesh[4821]: Starting VM 504
Oct 09 09:09:48 proxmox pve-guests[13054]: start VM 504: UPID:proxmox:000032FE:000044C1:5D9D87BC:qmstart:504:root@pam:
Oct 09 09:09:48 proxmox pve-guests[4822]: <root@pam> starting task UPID:proxmox:000032FE:000044C1:5D9D87BC:qmstart:504:root@pam:
Oct 09 09:09:48 proxmox systemd[1]: Started 504.scope.
Oct 09 09:09:48 proxmox systemd-udevd[13065]: Could not generate persistent MAC address for tap504i0: No such file or directory
Oct 09 09:09:48 proxmox kernel: device tap504i0 entered promiscuous mode
Oct 09 09:09:49 proxmox systemd-udevd[13063]: Could not generate persistent MAC address for fwbr504i0: No such file or directory
Oct 09 09:09:49 proxmox systemd-udevd[13065]: Could not generate persistent MAC address for fwpr504p0: No such file or directory
Oct 09 09:09:49 proxmox systemd-udevd[13062]: Could not generate persistent MAC address for fwln504i0: No such file or directory
Oct 09 09:09:49 proxmox kernel: fwbr504i0: port 1(fwln504i0) entered blocking state
Oct 09 09:09:49 proxmox kernel: fwbr504i0: port 1(fwln504i0) entered disabled state
Oct 09 09:09:49 proxmox kernel: device fwln504i0 entered promiscuous mode
Oct 09 09:09:49 proxmox kernel: fwbr504i0: port 1(fwln504i0) entered blocking state
Oct 09 09:09:49 proxmox kernel: fwbr504i0: port 1(fwln504i0) entered forwarding state
Oct 09 09:09:49 proxmox kernel: vmbr5: port 4(fwpr504p0) entered blocking state
Oct 09 09:09:49 proxmox kernel: vmbr5: port 4(fwpr504p0) entered disabled state
Oct 09 09:09:49 proxmox kernel: device fwpr504p0 entered promiscuous mode
Oct 09 09:09:49 proxmox kernel: vmbr5: port 4(fwpr504p0) entered blocking state
Oct 09 09:09:49 proxmox kernel: vmbr5: port 4(fwpr504p0) entered forwarding state
Oct 09 09:09:49 proxmox kernel: fwbr504i0: port 2(tap504i0) entered blocking state
Oct 09 09:09:49 proxmox kernel: fwbr504i0: port 2(tap504i0) entered disabled state
Oct 09 09:09:49 proxmox kernel: fwbr504i0: port 2(tap504i0) entered blocking state
Oct 09 09:09:49 proxmox kernel: fwbr504i0: port 2(tap504i0) entered forwarding state
Oct 09 09:09:49 proxmox systemd-udevd[13063]: Could not generate persistent MAC address for tap504i1: No such file or directory
Oct 09 09:09:49 proxmox kernel: device tap504i1 entered promiscuous mode
Oct 09 09:09:49 proxmox systemd-udevd[13063]: Could not generate persistent MAC address for fwbr504i1: No such file or directory
Oct 09 09:09:49 proxmox systemd-udevd[13065]: Could not generate persistent MAC address for fwpr504p1: No such file or directory
Oct 09 09:09:49 proxmox systemd-udevd[13062]: Could not generate persistent MAC address for fwln504i1: No such file or directory
Oct 09 09:09:49 proxmox kernel: fwbr504i1: port 1(fwln504i1) entered blocking state
Oct 09 09:09:49 proxmox kernel: fwbr504i1: port 1(fwln504i1) entered disabled state
Oct 09 09:09:49 proxmox kernel: device fwln504i1 entered promiscuous mode
Oct 09 09:09:49 proxmox kernel: fwbr504i1: port 1(fwln504i1) entered blocking state
Oct 09 09:09:49 proxmox kernel: fwbr504i1: port 1(fwln504i1) entered forwarding state
Oct 09 09:09:49 proxmox kernel: vmbr2: port 2(fwpr504p1) entered blocking state
Oct 09 09:09:49 proxmox kernel: vmbr2: port 2(fwpr504p1) entered disabled state
Oct 09 09:09:49 proxmox kernel: device fwpr504p1 entered promiscuous mode
Oct 09 09:09:49 proxmox kernel: vmbr2: port 2(fwpr504p1) entered blocking state
Oct 09 09:09:49 proxmox kernel: vmbr2: port 2(fwpr504p1) entered forwarding state
Oct 09 09:09:49 proxmox kernel: fwbr504i1: port 2(tap504i1) entered blocking state
Oct 09 09:09:49 proxmox kernel: fwbr504i1: port 2(tap504i1) entered disabled state
Oct 09 09:09:49 proxmox kernel: fwbr504i1: port 2(tap504i1) entered blocking state
Oct 09 09:09:49 proxmox kernel: fwbr504i1: port 2(tap504i1) entered forwarding state
Oct 09 09:09:49 proxmox systemd-udevd[13063]: Could not generate persistent MAC address for tap504i2: No such file or directory
Oct 09 09:09:49 proxmox kernel: device tap504i2 entered promiscuous mode
Oct 09 09:09:49 proxmox systemd-udevd[13063]: Could not generate persistent MAC address for fwbr504i2: No such file or directory
Oct 09 09:09:49 proxmox systemd-udevd[13062]: Could not generate persistent MAC address for fwln504i2: No such file or directory
Oct 09 09:09:49 proxmox systemd-udevd[13065]: Could not generate persistent MAC address for fwpr504p2: No such file or directory
Oct 09 09:09:49 proxmox kernel: fwbr504i2: port 1(fwln504i2) entered blocking state
Oct 09 09:09:49 proxmox kernel: fwbr504i2: port 1(fwln504i2) entered disabled state
Oct 09 09:09:49 proxmox kernel: device fwln504i2 entered promiscuous mode
Oct 09 09:09:49 proxmox kernel: fwbr504i2: port 1(fwln504i2) entered blocking state
Oct 09 09:09:49 proxmox kernel: fwbr504i2: port 1(fwln504i2) entered forwarding state
Oct 09 09:09:49 proxmox kernel: vmbr3: port 3(fwpr504p2) entered blocking state
Oct 09 09:09:49 proxmox kernel: vmbr3: port 3(fwpr504p2) entered disabled state
Oct 09 09:09:49 proxmox kernel: device fwpr504p2 entered promiscuous mode
Oct 09 09:09:49 proxmox kernel: vmbr3: port 3(fwpr504p2) entered blocking state
Oct 09 09:09:49 proxmox kernel: vmbr3: port 3(fwpr504p2) entered forwarding state
Oct 09 09:09:49 proxmox kernel: fwbr504i2: port 2(tap504i2) entered blocking state
Oct 09 09:09:49 proxmox kernel: fwbr504i2: port 2(tap504i2) entered disabled state
Oct 09 09:09:49 proxmox kernel: fwbr504i2: port 2(tap504i2) entered blocking state
Oct 09 09:09:49 proxmox kernel: fwbr504i2: port 2(tap504i2) entered forwarding state
Oct 09 09:25:51 proxmox pvedaemon[4788]: <sebastian@pam> starting task UPID:proxmox:00003098:0001BCF4:5D9D8B7F:vncproxy:504:sebastian@pam:
Oct 09 09:25:51 proxmox pvedaemon[12440]: starting vnc proxy UPID:proxmox:00003098:0001BCF4:5D9D8B7F:vncproxy:504:sebastian@pam:
Oct 09 09:26:00 proxmox pvedaemon[4788]: <sebastian@pam> end task UPID:proxmox:00003098:0001BCF4:5D9D8B7F:vncproxy:504:sebastian@pam: OK
...........
Oct 16 08:39:55 proxmox pvedaemon[14510]: starting vnc proxy UPID:proxmox:000038AE:03986261:5DA6BB3B:vncproxy:504:sebastian@pam:
Oct 16 08:39:55 proxmox pvedaemon[4789]: <sebastian@pam> starting task UPID:proxmox:000038AE:03986261:5DA6BB3B:vncproxy:504:sebastian@pam:
Oct 16 08:39:56 proxmox qm[14512]: VM 504 qmp command failed - VM 504 not running
Oct 16 08:39:56 proxmox pvedaemon[4789]: <sebastian@pam> end task UPID:proxmox:000038AE:03986261:5DA6BB3B:vncproxy:504:sebastian@pam: Failed to run vncproxy.
Oct 16 08:39:59 proxmox pvedaemon[4788]: <sebastian@pam> starting task UPID:proxmox:0000399C:039863F5:5DA6BB3F:qmstart:504:sebastian@pam:
Oct 16 08:39:59 proxmox pvedaemon[14748]: start VM 504: UPID:proxmox:0000399C:039863F5:5DA6BB3F:qmstart:504:sebastian@pam:
Oct 16 08:39:59 proxmox systemd[1]: 504.scope: Succeeded.
Oct 16 08:39:59 proxmox systemd[1]: Stopped 504.scope.
Oct 16 08:40:04 proxmox pvedaemon[4788]: <sebastian@pam> end task UPID:proxmox:0000399C:039863F5:5DA6BB3F:qmstart:504:sebastian@pam: timeout waiting on systemd
 
are there any .pid files in /var/run/qemu-server?
if yes, do the pids inside match with the running processes?
 
dcsapak said:
are there any .pid files in /var/run/qemu-server?
if yes, do the pids inside match with the running processes?
Click to expand...

For my 501 machine (also stopped but without a process still running or a log entry) there is no PID file. The 504 has a PID which do not match the processes.

screen2.png
screen3.png
 
ok it seems the main qemu process died somehow (13072) but the childs still run (13073)

you could try to run the vm in the foreground ('qm showcmd ID' gives the qemu commandline and from that remove the '--daemonize' flag to run in foreground)
to see why it dies/get killed if nothing in the log is found...
 
dcsapak said:
ok it seems the main qemu process died somehow (13072) but the childs still run (13073)

you could try to run the vm in the foreground ('qm showcmd ID' gives the qemu commandline and from that remove the '--daemonize' flag to run in foreground)
to see why it dies/get killed if nothing in the log is found...
Click to expand...

So you mean i kill the process right now, restart it in the foreground and just wait for the next crash? Thank you for the help btw :)
 
Semmo said:
So you mean i kill the process right now, restart it in the foreground and just wait for the next crash? Thank you for the help btw :)
Click to expand...
yes this is what i meant, but only if the logs do not tell you anything else
is the vm still working btw? maybe you can shut it down from ssh or something like that? (instead of killing it)
 
dcsapak said:
yes this is what i meant, but only if the logs do not tell you anything else
is the vm still working btw? maybe you can shut it down from ssh or something like that? (instead of killing it)
Click to expand...

No it's not reachable. I also tried it over my internal switch only > no route to host
Nothing helpful in the logs so far. Is there anyway to increase the loglevel maybe?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom