backup job finished and VM stop working

[Kompex]

We have been using proxmox on 3 physical servers for many years
We backup VM twice a week from one machine to another.
We have a total of a clique of VMs on one server running 4 to 7 virtuals.
For the past several months, two of the VMs have been shutting down at night after the backup is finished. This does not always happen once or twice a month. I can't figure out what is the reason for these VMs shutting down after a task is completed.

VM when stop today is VM =105
Mchine from backup Start is Proxmox2 >> backa save to Proxmox3.
We have problem stop only VM = 105,109 another VM is ok

Details​

VMID​	Name​	Status​	Time​	Size​	Filename​
100	unbound	ok	25min 48s	4.048 GiB	/mnt/pve/proxmox3_nfs/dump/vzdump-qemu-100-2024_12_13-01_15_03.vma.zst
105	hoster	ok	21min 29s	22.379 GiB	/mnt/pve/proxmox3_nfs/dump/vzdump-qemu-105-2024_12_13-01_40_51.vma.zst
109	LMSnew	ok	34min 27s	71.232 GiB	/mnt/pve/proxmox3_nfs/dump/vzdump-qemu-109-2024_12_13-02_02_20.vma.zst
116	huawei	ok	2min 27s	11.217 GiB	/mnt/pve/proxmox3_nfs/dump/vzdump-qemu-116-2024_12_13-02_36_48.vma.zst
118	asterisk	ok	5min 44s	7.361 GiB	/mnt/pve/proxmox3_nfs/dump/vzdump-qemu-118-2024_12_13-02_39_15.vma.zst
133	kaczor	ok	11min 29s	23.608 GiB	/mnt/pve/proxmox3_nfs/dump/vzdump-qemu-133-2024_12_13-02_44_59.vma.zst

Total running time: 1h 41min 25s
Total size: 139.846 GiB

Logs​

vzdump 109 100 118 105 116 133 --mode snapshot --storage proxmox3_nfs --mailnotification always --fleecing 0 --mailto gabi@kompex.pl --bwlimit 122880 --compress zstd --node proxmox2 --quiet 1


105: 2024-12-13 01:40:51 INFO: Starting Backup of VM 105 (qemu)
105: 2024-12-13 01:40:51 INFO: status = running
105: 2024-12-13 01:40:51 INFO: VM Name: hoster
105: 2024-12-13 01:40:51 INFO: include disk 'scsi0' 'ssd3:vm-105-disk-0' 150G
105: 2024-12-13 01:40:51 INFO: backup mode: snapshot
105: 2024-12-13 01:40:51 INFO: bandwidth limit: 122880 KiB/s
105: 2024-12-13 01:40:51 INFO: ionice priority: 7
105: 2024-12-13 01:40:51 INFO: creating vzdump archive '/mnt/pve/proxmox3_nfs/dump/vzdump-qemu-105-2024_12_13-01_40_51.vma.zst'
105: 2024-12-13 01:40:51 INFO: started backup task '7ccedc44-f06a-492d-a3a6-516a91892523'
105: 2024-12-13 01:40:51 INFO: resuming VM again
105: 2024-12-13 01:40:54 INFO: 0% (360.0 MiB of 150.0 GiB) in 3s, read: 120.0 MiB/s, write: 35.8 MiB/s
105: 2024-12-13 01:41:04 INFO: 1% (1.5 GiB of 150.0 GiB) in 13s, read: 120.1 MiB/s, write: 97.0 MiB/s
105: 2024-12-13 01:41:17 INFO: 2% (3.0 GiB of 150.0 GiB) in 26s, read: 119.5 MiB/s, write: 81.4 MiB/s
105: 2024-12-13 01:41:30 INFO: 3% (4.6 GiB of 150.0 GiB) in 39s, read: 119.7 MiB/s, write: 55.6 MiB/s

.........................
105: 2024-12-13 02:01:53 INFO: 98% (147.0 GiB of 150.0 GiB) in 21m 2s, read: 119.4 MiB/s, write: 1.2 MiB/s
105: 2024-12-13 02:02:07 INFO: 99% (148.5 GiB of 150.0 GiB) in 21m 16s, read: 111.1 MiB/s, write: 19.7 MiB/s
105: 2024-12-13 02:02:20 INFO: 100% (150.0 GiB of 150.0 GiB) in 21m 29s, read: 116.4 MiB/s, write: 630.0 B/s
105: 2024-12-13 02:02:20 INFO: backup is sparse: 101.99 GiB (67%) total zero data
105: 2024-12-13 02:02:20 INFO: transferred 150.00 GiB in 1289 seconds (119.2 MiB/s)
105: 2024-12-13 02:02:20 INFO: archive file size: 22.38GB
105: 2024-12-13 02:02:20 INFO: prune older backups with retention: keep-daily=2, keep-hourly=2, keep-last=15, keep-monthly=9, keep-weekly=4, keep-yearly=5
105: 2024-12-13 02:02:20 INFO: removing backup 'proxmox3_nfs:backup/vzdump-qemu-105-2024_07_05-01_16_53.vma.zst'
105: 2024-12-13 02:02:20 INFO: pruned 1 backup(s) not covered by keep-retention policy
105: 2024-12-13 02:02:20 INFO: Finished Backup of VM 105 (00:21:29)

109: 2024-12-13 02:02:20 INFO: Starting Backup of VM 109 (qemu)
109: 2024-12-13 02:02:20 INFO: status = running
109: 2024-12-13 02:02:20 INFO: VM Name: LMSnew
109: 2024-12-13 02:02:20 INFO: include disk 'scsi0' 'ssd1:vm-109-disk-0' 240G
109: 2024-12-13 02:02:20 INFO: backup mode: snapshot
109: 2024-12-13 02:02:20 INFO: bandwidth limit: 122880 KiB/s

 

[Kompex]

kernel.log

 

[fiona]

Hi,
the kernel log shows that your system ran out of RAM and had to kill off a process to recover. That process was the VM.