[SOLVED] Proxmox switching off power of other VM while performing backup

[Ales_R]

Hello,
I run Proxmox PVE 6.4-12 with 3 VM - Ubuntu Server and 2 VM of Windows Server 2019 (VM103 and VM104). I have scheduled a backup of all VM (stop mode) and it runs fine. However, sometimes happens that when the scheduled backup up VM104 starts then the VM103 switches off (like power cut off) unexpectedly. The VM103 wont start by itself then and Windows log shows critical failure with power supply.
How can I solve this? Where can I find more information what and why it happened?
Thank you.

 

[Stefan_R]

Please post your 'pveversion -v' and 'qm config <vmid>' output for your Windows VMs. For any error messages, check the task log (at the bottom of the GUI for your backup job) and the syslog ('journalctl -e').

 

[Ales_R]

Hello,
posts below.
-----------------------------------
root@pve:~# pveversion -v
proxmox-ve: 6.4-1 (running kernel: 5.4.128-1-pve)
pve-manager: 6.4-13 (running version: 6.4-13/9f411e79)
pve-kernel-5.4: 6.4-5
pve-kernel-helper: 6.4-5
pve-kernel-5.4.128-1-pve: 5.4.128-1
pve-kernel-5.4.124-1-pve: 5.4.124-2
pve-kernel-5.4.114-1-pve: 5.4.114-1
pve-kernel-5.4.106-1-pve: 5.4.106-1
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.1.0-1
libpve-access-control: 6.4-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-3
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-3
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.12-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.6-1
pve-cluster: 6.4-1
pve-container: 3.3-6
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-4
pve-firmware: 3.2-4
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.5-pve1~bpo10+1
root@pve:~#
-----------------------------------
root@pve:~# qm config 103
agent: 1
boot: order=scsi0;net0
cores: 4
cpu: host
description:
ide2: none,media=cdrom
memory: 10240
name: WS2019-AD-DHCP-DNS
net0: e1000=E2:A0:3C:BD:67:70,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: win10
scsi0: local-zfs:vm-103-disk-0,size=100G
scsi1: local-zfs:vm-103-disk-1,size=500G
scsi2: local-zfs:vm-103-disk-2,backup=0,size=600G
scsihw: virtio-scsi-pci
smbios1: uuid=b5a44c94-5d89-4a59-a5f6-f80d9bd45798
sockets: 1
startup: order=1
vmgenid: e3d891e7-6767-4ac7-b3f7-ca61fe3b216f
root@pve:~#
-----------------------------------
root@pve:~# qm config 104
agent: 1
boot: order=scsi0
cores: 4
cpu: host
description:
memory: 10240
name: WS2019-RDS
net0: e1000=F6:34:3D:FE:27:10,bridge=vmbr0
numa: 0
onboot: 1
ostype: win10
scsi0: local-zfs:vm-104-disk-0,size=100G
scsihw: virtio-scsi-pci
smbios1: uuid=4bc974ca-ace0-4002-8f24-74cf9cd00fbc
sockets: 1
startup: order=3,up=90
vmgenid: c588f87c-4b9f-401b-9f67-e7f1ce5b9584
root@pve:~#
-----------------------------------
Backup job finished successfully, no error log in task history.
-----------------------------------
'journalctl -e' doesnt show this issue because I have updated yesterday and rebooted the machine :-(

BR

 

[Stefan_R]

'journalctl -e' doesnt show this issue because I have updated yesterday and rebooted the machine :-(

Then I would advice to see if the issue persists and is reproducible, if so, post the logs, if not, well, fixed

 

[Ales_R]

Hello,
logs attached.

 

[Stefan_R]

That's an OOM killer log, you're running out of memory on the host. Reduce RAM of the VMs or add more RAM to the physical server.

 

[Ales_R]

Hi,
my VM never reach maximum available RAM. This shutdown happens during backup. I am using ZFS - could this be the problem? How to limit RAM used by ZFS?
BR
Ales

 

[Stefan_R]

You're not running out of RAM in the VM, the PVE host is running out. Limiting ZFS will not help, as its ARC cache is already shrinking automatically to cope. Can you post the output of a qm list && free command on PVE while both VMs are running?

 

[Ales_R]

Here is the output:

---

VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
101 Webserver-Ubuntu-Server-20.04-LTS running 1536 40.00 23304
103 WS2019-AD-DHCP-DNS running 8192 100.00 12215
104 WS2019-RDS running 8192 100.00 7763
105 Windows-XP stopped 1536 32.00 0
total used free shared buff/cache available
Mem: 30816280 25745464 4491412 432224 579404 4303832
Swap: 0 0 0

---

I reduced the 103 and 104 VM's memory from 10GB to 8GB yesterday.