KVM guest dies silently

[Alex Kr]

Hello !

I'm trying to figure out why some of my KVM guests dies, no logs, messages in dmesg , or any useful information — guest just shutting down silently and then I need to turn it on via web-interface. This problem happens with Windows and Ubuntu guests with GUI.

At this moment I launched one of guest directly via console without daemonize option and waiting for the next shutdown.

Proxmox version and other useful information:

Spoiler: pveversion -v

proxmox-ve: 6.1-2 (running kernel: 5.3.13-1-pve)
pve-manager: 6.1-5 (running version: 6.1-5/9bf06119)
pve-kernel-5.3: 6.1-1
pve-kernel-helper: 6.1-1
pve-kernel-5.0: 6.0-11
pve-kernel-5.3.13-1-pve: 5.3.13-1
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.21-4-pve: 5.0.21-9
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 1.2.8-1+pve4
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-5
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-9
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.1-3
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve3
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-1
pve-cluster: 6.1-2
pve-container: 3.0-15
pve-docs: 6.1-3
pve-edk2-firmware: 2.20191127-1
pve-firewall: 4.0-9
pve-firmware: 3.0-4
pve-ha-manager: 3.0-8
pve-i18n: 2.0-3
pve-qemu-kvm: 4.1.1-2
pve-xtermjs: 3.13.2-1
qemu-server: 6.1-4
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.2-pve2

Spoiler: kvm guest configuration

balloon: 0
bios: ovmf
bootdisk: ide0
cores: 8
ide0: local-ssd:vm-175-disk-0,cache=writeback,discard=on,format=raw,size=127G,ssd=1
ide1: local-ssd:vm-175-disk-1,cache=writeback,discard=on,format=raw,size=100G,ssd=1
memory: 8192
name: win04
net0: virtio=4A:24:12:C1:A1:2A,bridge=vmbr0
numa: 0
onboot: 1
ostype: win8
scsihw: virtio-scsi-pci
smbios1: uuid=e5eb4ec4-3b15-423b-a679-3c840d2121e9
sockets: 2
vmgenid: a29c946f-bfcc-4b7f-b1da-e84b21565779

Spoiler: guest command line

kvm
-id 175
-name win04
-chardev socket,id=qmp,path=/var/run/qemu-server/175.qmp,server,nowait
-mon chardev=qmp,mode=control
-chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5
-mon chardev=qmp-event,mode=control
-pidfile /var/run/qemu-server/175.pid
-smbios type=1,uuid=e5eb4ec4-3b15-423b-a679-3c840d2121e9
-drive if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd
-drive if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/tmp/175-ovmf.fd
-smp 16,sockets=2,cores=8,maxcpus=16
-nodefaults
-boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg
-vnc unix:/var/run/qemu-server/175.vnc,password
-no-hpet
-cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,hv_ipi,enforce
-m 8192
-device pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e
-device pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f
-device vmgenid,guid=a29c946f-bfcc-4b7f-b1da-e84b21565779
-device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2
-device usb-tablet,id=tablet,bus=uhci.0,port=1
-device VGA,id=vga,bus=pci.0,addr=0x2
-iscsi initiator-name=iqn.1993-08.org.debian:01:abbcf1a16b49
-drive file=/dev/pve-ssd/vm-175-disk-0,if=none,id=drive-ide0,format=raw,cache=writeback,discard=on,aio=threads,detect-zeroes=unmap
-device ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,rotation_rate=1,bootindex=100
-drive file=/dev/pve-ssd/vm-175-disk-1,if=none,id=drive-ide1,format=raw,cache=writeback,discard=on,aio=threads,detect-zeroes=unmap
-device ide-hd,bus=ide.0,unit=1,drive=drive-ide1,id=ide1,rotation_rate=1
-netdev type=tap,id=net0,ifname=tap175i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on
-device virtio-net-pci,mac=4A:24:12:C1:A1:2A,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300
-rtc driftfix=slew,base=localtime
-machine type=pc+pve1
-global kvm-pit.lost_tick_policy=discard

Can you help me solve this problem ? I would really appreciate it if you could give me some advice.

Thank you !

 

[spirit]

Maybe out of memory on Host && oom killer ?

 

[Alex Kr]

If it was oom killer then oom message will apear in dmesg , but it’s not ,only messages like

Spoiler: dmesg

[Fri Jan 10 17:10:21 2020] fwbr183i0: port 2(tap183i0) entered disabled state
[Fri Jan 10 17:10:21 2020] fwbr183i0: port 2(tap183i0) entered disabled state
[Fri Jan 10 17:10:22 2020] fwbr183i0: port 1(fwln183i0) entered disabled state
[Fri Jan 10 17:10:22 2020] vmbr0: port 4(fwpr183p0) entered disabled state
[Fri Jan 10 17:10:22 2020] device fwln183i0 left promiscuous mode
[Fri Jan 10 17:10:22 2020] fwbr183i0: port 1(fwln183i0) entered disabled state
[Fri Jan 10 17:10:22 2020] device fwpr183p0 left promiscuous mode
[Fri Jan 10 17:10:22 2020] vmbr0: port 4(fwpr183p0) entered disabled state

 

[spirit]

If it was oom killer then oom message will apear in dmesg , but it’s not ,only messages like

Spoiler: dmesg

[Fri Jan 10 17:10:21 2020] fwbr183i0: port 2(tap183i0) entered disabled state
[Fri Jan 10 17:10:21 2020] fwbr183i0: port 2(tap183i0) entered disabled state
[Fri Jan 10 17:10:22 2020] fwbr183i0: port 1(fwln183i0) entered disabled state
[Fri Jan 10 17:10:22 2020] vmbr0: port 4(fwpr183p0) entered disabled state
[Fri Jan 10 17:10:22 2020] device fwln183i0 left promiscuous mode
[Fri Jan 10 17:10:22 2020] fwbr183i0: port 1(fwln183i0) entered disabled state
[Fri Jan 10 17:10:22 2020] device fwpr183p0 left promiscuous mode
[Fri Jan 10 17:10:22 2020] vmbr0: port 4(fwpr183p0) entered disabled state

This log is when the vm is shutdown or killed (when the kvm process is halted).

Now, why, I really don't known. (Until the kvm process crash, but I don't have see this since years).

do you use HA on your cluster ?

 

[Alex Kr]

No ,we have not configured HA ,just a big cluster with 32 members .

 

[Alex Kr]

By the way ,problem appears only on kvm guests with gui(linux or windows ) ,regular linux guests without gui work's fine .

 

[vazir]

I have the same problem for certain CT's - it just dies. HA immediately restarts it. Only one, single CT, very basic setup, debian 10, but 2 GRE tunnels.