Backup stopping VM´s, when selected snapshoot mode

[Marcos Trigo]

Hello Proxmox community,

i'm having a problem with my proxmox, i set the backup to be done once a day, in snapshot mode, saving to an external NAS drive, it does correctly every day, when sunday arrives the system switches the backup mode to stopped, and the machine is turned off after the backup.

I've been having this problem for a while, it used to happen once every two weeks, now it's dropped to a week, if I don't restart the proxmox server, it keeps shutting down every day.

This happens in both VMS that I have 100 and 101, and after it happens in one the next day, it's the other and sometimes both are turned off





INFO: starting new backup job: vzdump 101 --compress zstd --quiet 1 --mode snapshot --mailto --mailnotification always --storage NAS
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2022-04-18 04:30:02
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: W2019-SERVER
INFO: include disk 'virtio0' 'HD_VM_101:vm-101-disk-0' 150G
INFO: include disk 'virtio1' 'Arquivos:vm-101-disk-1' 600G
INFO: include disk 'virtio2' 'Backup:vm-101-disk-0' 400G
INFO: creating vzdump archive '/mnt/pve/NAS/dump/vzdump-qemu-101-2022_04_18-04_30_02.vma.zst'
INFO: starting kvm to execute backup task
INFO: started backup task '043c72b2-5b47-494a-9c42-4b6cc85fea04'
INFO: 0% (369.1 MiB of 1.1 TiB) in 3s, read: 123.0 MiB/s, write: 108.9 MiB/s
INFO: 1% (11.7 GiB of 1.1 TiB) in 1m 20s, read: 150.2 MiB/s, write: 147.7 MiB/s
INFO: 2% (23.1 GiB of 1.1 TiB) in 2m 34s, read: 157.8 MiB/s, write: 157.8 MiB/s
INFO: 3% (34.6 GiB of 1.1 TiB) in 3m 46s, read: 163.9 MiB/s, write: 163.9 MiB/s
INFO: 4% (46.0 GiB of 1.1 TiB) in 4m 58s, read: 162.7 MiB/s, write: 162.7 MiB/s
INFO: 5% (57.6 GiB of 1.1 TiB) in 6m 8s, read: 168.7 MiB/s, write: 168.7 MiB/s
INFO: 6% (69.0 GiB of 1.1 TiB) in 7m 22s, read: 158.9 MiB/s, write: 158.9 MiB/s
INFO: 7% (80.6 GiB of 1.1 TiB) in 9m 7s, read: 112.8 MiB/s, write: 112.8 MiB/s
INFO: 8% (92.0 GiB of 1.1 TiB) in 10m 21s, read: 158.1 MiB/s, write: 158.1 MiB/s
INFO: 9% (103.6 GiB of 1.1 TiB) in 11m 34s, read: 162.0 MiB/s, write: 162.0 MiB/s
INFO: 10% (115.1 GiB of 1.1 TiB) in 12m 44s, read: 168.0 MiB/s, write: 168.0 MiB/s
INFO: 11% (126.6 GiB of 1.1 TiB) in 13m 59s, read: 157.4 MiB/s, write: 157.4 MiB/s
INFO: 12% (138.1 GiB of 1.1 TiB) in 15m 11s, read: 163.7 MiB/s, write: 163.7 MiB/s
INFO: 13% (149.6 GiB of 1.1 TiB) in 16m 24s, read: 161.1 MiB/s, write: 161.1 MiB/s
INFO: 14% (161.1 GiB of 1.1 TiB) in 17m 37s, read: 161.5 MiB/s, write: 161.5 MiB/s
INFO: 15% (172.6 GiB of 1.1 TiB) in 18m 54s, read: 152.4 MiB/s, write: 152.4 MiB/s
INFO: 16% (184.0 GiB of 1.1 TiB) in 21m 30s, read: 75.2 MiB/s, write: 75.2 MiB/s
INFO: 17% (195.5 GiB of 1.1 TiB) in 25m 41s, read: 46.9 MiB/s, write: 46.9 MiB/s
INFO: 18% (207.2 GiB of 1.1 TiB) in 27m 4s, read: 143.7 MiB/s, write: 143.7 MiB/s
INFO: 19% (218.5 GiB of 1.1 TiB) in 28m 15s, read: 164.0 MiB/s, write: 164.0 MiB/s
INFO: 20% (230.1 GiB of 1.1 TiB) in 29m 26s, read: 167.1 MiB/s, write: 167.1 MiB/s
INFO: 21% (241.6 GiB of 1.1 TiB) in 31m 24s, read: 99.6 MiB/s, write: 99.6 MiB/s
INFO: 22% (253.0 GiB of 1.1 TiB) in 33m 46s, read: 82.6 MiB/s, write: 82.6 MiB/s
INFO: 23% (264.6 GiB of 1.1 TiB) in 35m 38s, read: 105.4 MiB/s, write: 105.4 MiB/s
INFO: 24% (276.0 GiB of 1.1 TiB) in 38m 50s, read: 61.0 MiB/s, write: 61.0 MiB/s
INFO: 25% (287.5 GiB of 1.1 TiB) in 45m 1s, read: 31.7 MiB/s, write: 31.7 MiB/s
INFO: 26% (299.0 GiB of 1.1 TiB) in 47m 15s, read: 87.9 MiB/s, write: 87.9 MiB/s
INFO: 27% (310.6 GiB of 1.1 TiB) in 49m 32s, read: 86.3 MiB/s, write: 86.3 MiB/s
INFO: 28% (322.0 GiB of 1.1 TiB) in 51m 20s, read: 108.5 MiB/s, write: 108.5 MiB/s
INFO: 29% (333.5 GiB of 1.1 TiB) in 52m 32s, read: 163.5 MiB/s, write: 163.5 MiB/s
INFO: 30% (345.1 GiB of 1.1 TiB) in 54m, read: 134.5 MiB/s, write: 134.5 MiB/s
INFO: 31% (356.5 GiB of 1.1 TiB) in 55m 55s, read: 102.0 MiB/s, write: 102.0 MiB/s
INFO: 32% (368.0 GiB of 1.1 TiB) in 58m 42s, read: 70.5 MiB/s, write: 70.5 MiB/s
INFO: 33% (379.5 GiB of 1.1 TiB) in 1h 16s, read: 125.6 MiB/s, write: 125.6 MiB/s
INFO: 34% (391.2 GiB of 1.1 TiB) in 1h 3m 3s, read: 71.1 MiB/s, write: 71.1 MiB/s
INFO: 35% (402.6 GiB of 1.1 TiB) in 1h 4m 18s, read: 156.1 MiB/s, write: 155.5 MiB/s
INFO: 36% (414.0 GiB of 1.1 TiB) in 1h 7m 13s, read: 67.0 MiB/s, write: 66.8 MiB/s
INFO: 37% (425.5 GiB of 1.1 TiB) in 1h 13m 8s, read: 33.1 MiB/s, write: 33.1 MiB/s
INFO: 38% (437.0 GiB of 1.1 TiB) in 1h 17m 38s, read: 43.7 MiB/s, write: 43.7 MiB/s
INFO: 39% (448.5 GiB of 1.1 TiB) in 1h 21m 58s, read: 45.3 MiB/s, write: 45.3 MiB/s
INFO: 40% (460.0 GiB of 1.1 TiB) in 1h 25m 21s, read: 58.0 MiB/s, write: 57.9 MiB/s
INFO: 41% (471.6 GiB of 1.1 TiB) in 1h 29m 22s, read: 49.1 MiB/s, write: 49.0 MiB/s
INFO: 42% (483.0 GiB of 1.1 TiB) in 1h 31m 35s, read: 88.2 MiB/s, write: 87.8 MiB/s
INFO: 43% (494.5 GiB of 1.1 TiB) in 1h 33m 40s, read: 94.0 MiB/s, write: 93.6 MiB/s
INFO: 44% (506.1 GiB of 1.1 TiB) in 1h 36m, read: 85.1 MiB/s, write: 85.0 MiB/s
INFO: 45% (517.5 GiB of 1.1 TiB) in 1h 38m 24s, read: 81.1 MiB/s, write: 81.0 MiB/s
INFO: 46% (529.1 GiB of 1.1 TiB) in 1h 41m 4s, read: 74.0 MiB/s, write: 74.0 MiB/s
INFO: 47% (540.5 GiB of 1.1 TiB) in 1h 42m 24s, read: 146.1 MiB/s, write: 145.5 MiB/s
INFO: 48% (552.0 GiB of 1.1 TiB) in 1h 44m 47s, read: 82.3 MiB/s, write: 80.4 MiB/s
INFO: 49% (563.5 GiB of 1.1 TiB) in 1h 46m 49s, read: 96.5 MiB/s, write: 96.1 MiB/s
INFO: 50% (575.0 GiB of 1.1 TiB) in 1h 48m 24s, read: 124.1 MiB/s, write: 124.0 MiB/s
INFO: 51% (586.5 GiB of 1.1 TiB) in 1h 51m 4s, read: 73.6 MiB/s, write: 73.4 MiB/s
INFO: 52% (598.0 GiB of 1.1 TiB) in 1h 54m 4s, read: 65.5 MiB/s, write: 65.4 MiB/s
INFO: 53% (609.6 GiB of 1.1 TiB) in 1h 56m 33s, read: 79.6 MiB/s, write: 79.3 MiB/s
INFO: 54% (621.1 GiB of 1.1 TiB) in 1h 58m 22s, read: 108.0 MiB/s, write: 107.6 MiB/s
INFO: 55% (632.6 GiB of 1.1 TiB) in 1h 59m 52s, read: 130.3 MiB/s, write: 129.7 MiB/s
INFO: 56% (644.0 GiB of 1.1 TiB) in 2h 2m 3s, read: 89.8 MiB/s, write: 89.7 MiB/s
INFO: 57% (655.5 GiB of 1.1 TiB) in 2h 5m 5s, read: 64.7 MiB/s, write: 64.6 MiB/s
INFO: 58% (667.1 GiB of 1.1 TiB) in 2h 6m 56s, read: 106.3 MiB/s, write: 106.2 MiB/s
INFO: 59% (678.5 GiB of 1.1 TiB) in 2h 11m 5s, read: 47.2 MiB/s, write: 47.0 MiB/s
INFO: 60% (690.0 GiB of 1.1 TiB) in 2h 13m 44s, read: 74.0 MiB/s, write: 73.2 MiB/s
INFO: 61% (701.5 GiB of 1.1 TiB) in 2h 16m 20s, read: 75.5 MiB/s, write: 75.4 MiB/s
INFO: 62% (713.1 GiB of 1.1 TiB) in 2h 18m 36s, read: 87.1 MiB/s, write: 86.6 MiB/s
INFO: 63% (724.6 GiB of 1.1 TiB) in 2h 20m 21s, read: 112.2 MiB/s, write: 112.1 MiB/s
INFO: 64% (736.1 GiB of 1.1 TiB) in 2h 22m 14s, read: 103.9 MiB/s, write: 103.9 MiB/s
INFO: 65% (747.6 GiB of 1.1 TiB) in 2h 23m 52s, read: 120.3 MiB/s, write: 119.5 MiB/s
INFO: 66% (759.0 GiB of 1.1 TiB) in 2h 26m 18s, read: 80.3 MiB/s, write: 79.9 MiB/s
INFO: 67% (770.6 GiB of 1.1 TiB) in 2h 29m 31s, read: 61.3 MiB/s, write: 61.2 MiB/s
INFO: 68% (782.0 GiB of 1.1 TiB) in 2h 31m 13s, read: 114.7 MiB/s, write: 113.7 MiB/s
INFO: 69% (793.5 GiB of 1.1 TiB) in 2h 36m 2s, read: 40.7 MiB/s, write: 40.5 MiB/s
INFO: 70% (805.0 GiB of 1.1 TiB) in 2h 40m 10s, read: 47.5 MiB/s, write: 47.4 MiB/s
INFO: 71% (816.5 GiB of 1.1 TiB) in 2h 43m 53s, read: 52.9 MiB/s, write: 52.4 MiB/s
INFO: 72% (828.1 GiB of 1.1 TiB) in 2h 48m 37s, read: 41.6 MiB/s, write: 41.2 MiB/s
INFO: 73% (839.6 GiB of 1.1 TiB) in 2h 52m 51s, read: 46.5 MiB/s, write: 46.3 MiB/s
INFO: 74% (851.0 GiB of 1.1 TiB) in 2h 58m 42s, read: 33.3 MiB/s, write: 33.2 MiB/s
INFO: 75% (862.5 GiB of 1.1 TiB) in 3h 4m 43s, read: 32.6 MiB/s, write: 31.9 MiB/s
INFO: 76% (874.0 GiB of 1.1 TiB) in 3h 9m 20s, read: 42.6 MiB/s, write: 41.7 MiB/s
INFO: 77% (885.5 GiB of 1.1 TiB) in 3h 14m 56s, read: 35.1 MiB/s, write: 35.0 MiB/s
INFO: 78% (897.2 GiB of 1.1 TiB) in 3h 15m 23s, read: 441.0 MiB/s, write: 27.5 MiB/s
INFO: 79% (908.6 GiB of 1.1 TiB) in 3h 15m 38s, read: 783.7 MiB/s, write: 5.5 MiB/s
INFO: 80% (920.2 GiB of 1.1 TiB) in 3h 15m 53s, read: 788.4 MiB/s, write: 3.4 MiB/s
INFO: 81% (931.7 GiB of 1.1 TiB) in 3h 16m 8s, read: 787.5 MiB/s, write: 0 B/s
INFO: 82% (943.4 GiB of 1.1 TiB) in 3h 16m 23s, read: 798.1 MiB/s, write: 1.5 MiB/s
INFO: 83% (954.7 GiB of 1.1 TiB) in 3h 16m 38s, read: 772.6 MiB/s, write: 0 B/s
INFO: 84% (966.0 GiB of 1.1 TiB) in 3h 16m 53s, read: 770.4 MiB/s, write: 0 B/s
INFO: 85% (978.0 GiB of 1.1 TiB) in 3h 17m 9s, read: 768.9 MiB/s, write: 0 B/s
INFO: 86% (989.3 GiB of 1.1 TiB) in 3h 17m 24s, read: 770.6 MiB/s, write: 0 B/s
INFO: 87% (1000.6 GiB of 1.1 TiB) in 3h 17m 41s, read: 679.7 MiB/s, write: 28.8 MiB/s
INFO: 88% (1012.1 GiB of 1.1 TiB) in 3h 20m 11s, read: 78.1 MiB/s, write: 76.9 MiB/s
INFO: 89% (1023.5 GiB of 1.1 TiB) in 3h 22m 58s, read: 70.4 MiB/s, write: 69.6 MiB/s
INFO: 90% (1.0 TiB of 1.1 TiB) in 3h 26m 13s, read: 60.3 MiB/s, write: 59.7 MiB/s
INFO: 91% (1.0 TiB of 1.1 TiB) in 3h 28m 54s, read: 73.1 MiB/s, write: 72.2 MiB/s
INFO: 92% (1.0 TiB of 1.1 TiB) in 3h 32m 14s, read: 59.0 MiB/s, write: 58.6 MiB/s
INFO: 93% (1.0 TiB of 1.1 TiB) in 3h 34m 39s, read: 81.6 MiB/s, write: 79.8 MiB/s
INFO: 94% (1.1 TiB of 1.1 TiB) in 3h 35m 56s, read: 153.2 MiB/s, write: 147.6 MiB/s
INFO: 95% (1.1 TiB of 1.1 TiB) in 3h 37m 50s, read: 102.5 MiB/s, write: 97.5 MiB/s
INFO: 96% (1.1 TiB of 1.1 TiB) in 3h 38m 33s, read: 276.2 MiB/s, write: 68.9 MiB/s
INFO: 97% (1.1 TiB of 1.1 TiB) in 3h 38m 52s, read: 634.9 MiB/s, write: 0 B/s
INFO: 98% (1.1 TiB of 1.1 TiB) in 3h 39m 10s, read: 645.6 MiB/s, write: 0 B/s
INFO: 99% (1.1 TiB of 1.1 TiB) in 3h 39m 29s, read: 635.9 MiB/s, write: 0 B/s
INFO: 100% (1.1 TiB of 1.1 TiB) in 3h 39m 46s, read: 660.4 MiB/s, write: 240.0 B/s
INFO: backup is sparse: 172.58 GiB (15%) total zero data
INFO: transferred 1.12 TiB in 13186 seconds (89.3 MiB/s)
INFO: kvm status changed after backup ('running') - keep VM running
INFO: archive file size: 339.69GB
INFO: removing backup 'NAS:backup/vzdump-qemu-101-2022_04_13-04_30_02.vma.zst'
no lock found trying to remove 'backup' lock
INFO: Finished Backup of VM 101 (03:41:35)
INFO: Backup finished at 2022-04-18 08:11:37
INFO: Backup job finished successfully
TASK OK

 

[dcsapak]

INFO: Backup started at 2022-04-18 04:30:02
INFO: status = stopped
INFO: backup mode: stop

it seems the vm was already stopped when the backup began.
i'd check the syslog to see when (and possibly why) the vm was stopped

 

[Marcos Trigo]

Hello dcsapak thanks for the quick response, you are correct, the machine stopped on the 17th at 22:38, follow the syslog. it seems that the system ran out of available memory but we only have two VMs and 32GB of ram, we have allocated 8GB for VM 101 and 4GB for VM 100.



* Syslog at backup time April 18



Apr 18 04:30:00 ******** systemd[1]: Started Proxmox VE replication runner.

Apr 18 04:30:01 ******** CRON[3921]: pam_unix(cron:session): session opened for user root by (uid=0)

Apr 18 04:30:01 ******** CRON[3922]: (root) CMD (vzdump 101 --compress zstd --storage NAS --mode snapshot --quiet 1 --mailnotification always --mailto marcos@********net.com.br)

Apr 18 04:30:02 ******** vzdump[3923]: <root@pam> starting task UPID:********:00000F54:030281C4:625D137A:vzdump:101:root@pam:

Apr 18 04:30:02 ******** vzdump[3924]: INFO: starting new backup job: vzdump 101 --compress zstd --quiet 1 --mode snapshot --mailto marcos@********net.com.br --mailnotification always --storage NAS

Apr 18 04:30:02 ******** vzdump[3924]: INFO: Starting Backup of VM 101 (qemu)

Apr 18 04:30:03 ******** systemd[1]: Started 101.scope.

Apr 18 04:30:03 ******** systemd-udevd[3918]: Using default interface naming scheme 'v240'.

Apr 18 04:30:03 ******** systemd-udevd[3918]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.

Apr 18 04:30:03 ******** systemd-udevd[3918]: Could not generate persistent MAC address for tap101i0: No such file or directory

Apr 18 04:30:03 ******** kernel: device tap101i0 entered promiscuous mode

Apr 18 04:30:03 ******** kernel: vmbr0: port 2(tap101i0) entered blocking state

Apr 18 04:30:03 ******** kernel: vmbr0: port 2(tap101i0) entered disabled state

Apr 18 04:30:03 ******** kernel: vmbr0: port 2(tap101i0) entered blocking state

Apr 18 04:30:03 ******** kernel: vmbr0: port 2(tap101i0) entered forwarding state

Apr 18 04:31:00 ******** systemd[1]: Starting Proxmox VE replication runner...









* Syslog Before backup April 17, system has disabled vm101



Apr 17 22:38:39 ******** kernel: zfs invoked oom-killer: gfp_mask=0x42dc0(GFP_KERNEL|__GFP_NOWARN|__GFP_COMP|__GFP_ZERO), order=2, oom_score_adj=0

Apr 17 22:38:39 ******** kernel: CPU: 11 PID: 7544 Comm: zfs Tainted: P O 5.4.157-1-pve #1

Apr 17 22:38:39 ******** kernel: Hardware name: Dell Inc. PowerEdge R440/04JN2K, BIOS 2.8.2 08/31/2020

Apr 17 22:38:39 ******** kernel: Call Trace:

Apr 17 22:38:39 ******** kernel: dump_stack+0x6d/0x8b

Apr 17 22:38:39 ******** kernel: dump_header+0x4f/0x1e1

Apr 17 22:38:39 ******** kernel: oom_kill_process.cold.33+0xb/0x10

Apr 17 22:38:39 ******** kernel: out_of_memory+0x1bf/0x4e0

Apr 17 22:38:39 ******** kernel: __alloc_pages_slowpath+0xd40/0xe30

Apr 17 22:38:39 ******** kernel: __alloc_pages_nodemask+0x2df/0x330

Apr 17 22:38:39 ******** kernel: kmalloc_large_node+0x42/0x90

Apr 17 22:38:39 ******** kernel: __kmalloc_node+0x267/0x330

Apr 17 22:38:39 ******** kernel: ? lru_cache_add_active_or_unevictable+0x39/0xb0

Apr 17 22:38:39 ******** kernel: spl_kmem_zalloc+0xd1/0x120 [spl]

Apr 17 22:38:39 ******** kernel: zfsdev_ioctl+0x2b/0xe0 [zfs]

Apr 17 22:38:39 ******** kernel: do_vfs_ioctl+0xa9/0x640

Apr 17 22:38:39 ******** kernel: ? handle_mm_fault+0xc9/0x1f0

Apr 17 22:38:39 ******** kernel: ksys_ioctl+0x67/0x90

Apr 17 22:38:39 ******** kernel: __x64_sys_ioctl+0x1a/0x20

Apr 17 22:38:39 ******** kernel: do_syscall_64+0x57/0x190

Apr 17 22:38:39 ******** kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9

Apr 17 22:38:39 ******** kernel: RIP: 0033:0x7fa554829427

Apr 17 22:38:39 ******** kernel: Code: 00 00 90 48 8b 05 69 aa 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 39 aa 0c 00 f7 d8 64 89 01 48

Apr 17 22:38:39 ******** kernel: RSP: 002b:00007ffc8da1f4b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010

Apr 17 22:38:39 ******** kernel: RAX: ffffffffffffffda RBX: 00007ffc8da1f4e0 RCX: 00007fa554829427

Apr 17 22:38:39 ******** kernel: RDX: 00007ffc8da1f4e0 RSI: 0000000000005a12 RDI: 0000000000000003

Apr 17 22:38:39 ******** kernel: RBP: 00007ffc8da1f4d0 R08: 00007fa553fd0010 R09: 0000000000000000

Apr 17 22:38:39 ******** kernel: R10: 0000000000000022 R11: 0000000000000246 R12: 00005600ed1252e0

Apr 17 22:38:39 ******** kernel: R13: 00005600ed1252e0 R14: 00005600ed1270c0 R15: 00007ffc8da22bf0

Apr 17 22:38:39 ******** kernel: Mem-Info:

Apr 17 22:38:39 ******** kernel: active_anon:1711273 inactive_anon:319330 isolated_anon:0

active_file:12377 inactive_file:9171 isolated_file:0

unevictable:6536 dirty:43 writeback:2 unstable:0

slab_reclaimable:59684 slab_unreclaimable:675691

mapped:21582 shmem:4772 pagetables:9068 bounce:0

free:561343 free_pcp:466 free_cma:0

Apr 17 22:38:39 ******** kernel: Node 0 active_anon:6845092kB inactive_anon:1277320kB active_file:49508kB inactive_file:36684kB unevictable:26144kB isolated(anon):0kB isolated(file):0kB mapped:86328kB dirty:172kB writeback:8kB shmem:19088kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 100352kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no

Apr 17 22:38:39 ******** kernel: Node 0 DMA free:15888kB min:32kB low:44kB high:56kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15904kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB

Apr 17 22:38:39 ******** kernel: lowmem_reserve[]: 0 1504 31663 31663 31663

Apr 17 22:38:39 ******** kernel: Node 0 DMA32 free:125712kB min:3208kB low:4748kB high:6288kB active_anon:116492kB inactive_anon:21576kB active_file:152kB inactive_file:16kB unevictable:0kB writepending:4kB present:1690624kB managed:1596016kB mlocked:0kB kernel_stack:48kB pagetables:148kB bounce:0kB free_pcp:8kB local_pcp:0kB free_cma:0kB

Apr 17 22:38:39 ******** kernel: lowmem_reserve[]: 0 0 30159 30159 30159

Apr 17 22:38:39 ******** kernel: Node 0 Normal free:2103772kB min:64340kB low:95220kB high:126100kB active_anon:6729020kB inactive_anon:1255716kB active_file:49356kB inactive_file:36668kB unevictable:26144kB writepending:176kB present:31457280kB managed:30882900kB mlocked:26144kB kernel_stack:18016kB pagetables:36124kB bounce:0kB free_pcp:1856kB local_pcp:0kB free_cma:0kB

Apr 17 22:38:39 ******** kernel: lowmem_reserve[]: 0 0 0 0 0

Apr 17 22:38:39 ******** kernel: Node 0 DMA: 2*4kB (U) 1*8kB (U) 0*16kB 2*32kB (U) 3*64kB (U) 2*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15888kB

Apr 17 22:38:39 ******** kernel: Node 0 DMA32: 30751*4kB (UMEH) 91*8kB (UMH) 1*16kB (H) 1*32kB (H) 1*64kB (H) 1*128kB (H) 1*256kB (H) 1*512kB (H) 1*1024kB (H) 0*2048kB 0*4096kB = 125764kB

Apr 17 22:38:39 ******** kernel: Node 0 Normal: 523459*4kB (UMEH) 1162*8kB (UMEH) 13*16kB (H) 8*32kB (H) 4*64kB (H) 5*128kB (H) 1*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2104748kB

Apr 17 22:38:39 ******** kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB

Apr 17 22:38:39 ******** kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB

Apr 17 22:38:39 ******** kernel: 32268 total pagecache pages

Apr 17 22:38:39 ******** kernel: 3359 pages in swap cache

Apr 17 22:38:39 ******** kernel: Swap cache stats: add 40058173, delete 40055768, find 22506369/39603077

Apr 17 22:38:39 ******** kernel: Free swap = 4526604kB

Apr 17 22:38:39 ******** kernel: Total swap = 8388604kB

Apr 17 22:38:39 ******** kernel: 8290975 pages RAM

Apr 17 22:38:39 ******** kernel: 0 pages HighMem/MovableOnly

Apr 17 22:38:39 ******** kernel: 167270 pages reserved

Apr 17 22:38:39 ******** kernel: 0 pages cma reserved

Apr 17 22:38:39 ******** kernel: 0 pages hwpoisoned

Apr 17 22:38:39 ******** kernel: Tasks state (memory values in pages):

Apr 17 22:38:39 ******** kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name

Apr 17 22:38:39 ******** kernel: [ 565] 0 565 15824 4512 147456 628 0 systemd-journal

Apr 17 22:38:39 ******** kernel: [ 590] 0 590 20263 5929 94208 0 -1000 dmeventd

Apr 17 22:38:39 ******** kernel: [ 594] 0 594 5745 733 73728 248 -1000 systemd-udevd

Apr 17 22:38:39 ******** kernel: [ 2377] 100 2377 23270 999 86016 178 0 systemd-timesyn

Apr 17 22:38:39 ******** kernel: [ 2380] 106 2380 1705 569 53248 58 0 rpcbind

Apr 17 22:38:39 ******** kernel: [ 2409] 104 2409 2270 847 49152 57 -900 dbus-daemon

Apr 17 22:38:39 ******** kernel: [ 2423] 0 2423 56455 702 90112 323 0 rsyslogd

Apr 17 22:38:39 ******** kernel: [ 2424] 0 2424 170352 567 131072 13 0 pve-lxc-syscall

Apr 17 22:38:39 ******** kernel: [ 2428] 0 2428 1022 201 49152 0 0 qmeventd

Apr 17 22:38:39 ******** kernel: [ 2432] 0 2432 21333 339 53248 53 0 lxcfs

Apr 17 22:38:39 ******** kernel: [ 2433] 0 2433 4879 672 73728 195 0 systemd-logind

Apr 17 22:38:39 ******** kernel: [ 2437] 0 2437 535 328 36864 0 -1000 watchdog-mux

Apr 17 22:38:39 ******** kernel: [ 2438] 0 2438 3164 488 61440 320 0 smartd

Apr 17 22:38:39 ******** kernel: [ 2442] 0 2442 41547 642 77824 111 0 zed

Apr 17 22:38:39 ******** kernel: [ 2445] 0 2445 1681 564 53248 27 0 ksmtuned

Apr 17 22:38:39 ******** kernel: [ 3156] 0 3156 1823 100 49152 84 0 lxc-monitord

Apr 17 22:38:39 ******** kernel: [ 3163] 0 3163 568 170 40960 19 0 none

Apr 17 22:38:39 ******** kernel: [ 3167] 0 3167 1123514 1968 675840 936 0 aakore

Apr 17 22:38:39 ******** kernel: [ 3187] 0 3187 1722 60 49152 0 0 iscsid

Apr 17 22:38:39 ******** kernel: [ 3188] 0 3188 1848 1254 49152 0 -17 iscsid

Apr 17 22:38:39 ******** kernel: [ 3203] 0 3203 3962 610 65536 170 -1000 sshd

Apr 17 22:38:39 ******** kernel: [ 3279] 0 3279 1402 323 49152 0 0 agetty

Apr 17 22:38:39 ******** kernel: [ 3404] 0 3404 183191 94 188416 244 0 rrdcached

Apr 17 22:38:39 ******** kernel: [ 3454] 0 3454 144468 2064 344064 1402 0 pmxcfs

Apr 17 22:38:39 ******** kernel: [ 3952] 0 3952 10867 480 81920 184 0 master

Apr 17 22:38:39 ******** kernel: [ 3959] 107 3959 10987 272 77824 190 0 qmgr

Apr 17 22:38:39 ******** kernel: [ 5166] 0 5166 2125 538 57344 31 0 cron

Apr 17 22:38:39 ******** kernel: [ 6071] 0 6071 76537 6693 307200 14860 0 pve-firewall

Apr 17 22:38:39 ******** kernel: [ 6094] 0 6094 76200 8571 303104 12979 0 pvestatd

Apr 17 22:38:39 ******** kernel: [ 6520] 0 6520 88897 547 417792 30005 0 pvedaemon

Apr 17 22:38:39 ******** kernel: [ 6526] 0 6526 91073 503 442368 30299 0 pvedaemon worke

Apr 17 22:38:39 ******** kernel: [ 6532] 0 6532 91094 894 438272 30273 0 pvedaemon worke

Apr 17 22:38:39 ******** kernel: [ 6533] 0 6533 91058 840 442368 30284 0 pvedaemon worke

Apr 17 22:38:39 ******** kernel: [ 7377] 0 7377 84550 1155 368640 22892 0 pve-ha-crm

Apr 17 22:38:39 ******** kernel: [ 7875] 0 7875 137345 148 122880 147 0 schedul2-bin

Apr 17 22:38:39 ******** kernel: [ 7884] 0 7884 153113 15303 1236992 19964 0 mms

Apr 17 22:38:39 ******** kernel: [ 7886] 33 7886 89256 113 446464 30371 0 pveproxy

Apr 17 22:38:39 ******** kernel: [ 7901] 33 7901 17622 118 172032 12125 0 spiceproxy

Apr 17 22:38:39 ******** kernel: [ 7903] 0 7903 84450 2092 389120 22113 0 pve-ha-lrm

Apr 17 22:38:39 ******** kernel: [ 8748] 0 8748 1002716 780 593920 1502 0 grpm-sync-unit

Apr 17 22:38:39 ******** kernel: [ 8750] 0 8750 572779 8129 413696 1264 0 task-manager

Apr 17 22:38:39 ******** kernel: [ 8751] 0 8751 347500 153 520192 1151 0 adp-agent

Apr 17 22:38:39 ******** kernel: [ 8925] 0 8925 2866565 1886466 17670144 227305 0 kvm

Apr 17 22:38:39 ******** kernel: [ 9846] 108 9846 14475 364 159744 12322 0 rpc.statd

Apr 17 22:38:39 ******** kernel: [ 26104] 0 26104 1518943 467695 9584640 594768 0 kvm

Apr 17 22:38:39 ******** kernel: [ 22124] 33 22124 17688 519 167936 12046 0 spiceproxy work

Apr 17 22:38:39 ******** kernel: [ 22194] 33 22194 89290 664 430080 30320 0 pveproxy worker

Apr 17 22:38:39 ******** kernel: [ 22200] 33 22200 89290 397 430080 30322 0 pveproxy worker

Apr 17 22:38:39 ******** kernel: [ 22203] 33 22203 89290 387 430080 30328 0 pveproxy worker

Apr 17 22:38:39 ******** kernel: [ 628] 0 628 21543 51 69632 0 0 pvefw-logger

Apr 17 22:38:39 ******** kernel: [ 5588] 107 5588 10956 640 81920 0 0 pickup

Apr 17 22:38:39 ******** kernel: [ 6358] 0 6358 1314 172 53248 0 0 sleep

Apr 17 22:38:39 ******** kernel: [ 7544] 0 7544 2708 580 57344 0 0 zfs

Apr 17 22:38:39 ******** kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/qemu.slice/101.scope,task=kvm,pid=8925,uid=0

Apr 17 22:38:39 ******** kernel: Out of memory: Killed process 8925 (kvm) total-vm:11466260kB, anon-rss:7542228kB, file-rss:3632kB, shmem-rss:4kB, UID:0 pgtables:17256kB oom_score_adj:0

Apr 17 22:38:39 ******** pvestatd[6094]: status update time (6.489 seconds)

Apr 17 22:38:40 ******** kernel: oom_reaper: reaped process 8925 (kvm), now anon-rss:0kB, file-rss:104kB, shmem-rss:4kB

Apr 17 22:38:41 ******** kernel: vmbr0: port 2(tap101i0) entered disabled state

Apr 17 22:38:41 ******** kernel: vmbr0: port 2(tap101i0) entered disabled state

Apr 17 22:38:41 ******** systemd[1]: 101.scope: Succeeded.

Apr 17 22:38:42 ******** qmeventd[2426]: Starting cleanup for 101

Apr 17 22:38:42 ******** qmeventd[2426]: Finished cleanup for 101





Best regards,

 

[dcsapak]

zfs uses up to 50% memory by default, so 32 - 16 (50% zfs) - 8 (vm1) - 4 (vm2) = 4GiB remaining for all other daemons & services. while that should be enough, depending on the workload and what else is running, this could be too little so the kernel has to kill some processes (most likely the one that uses the most memory..)

you can limit the zfs use though: https://pve.proxmox.com/wiki/ZFS_on_Linux#sysadmin_zfs_limit_memory_usage