[SOLVED] Windows Guests crashing during backup

[kbrault]

Hi everyone,
The last couple of weeks we are having random crashes of various windows 2016 guests when the backup runs. They do not always crash and not always the same guest.

The Windows guest log has:
The previous system shutdown at 12:10:47 PM on 1/14/204 was unexpected.

Proxmox log has:

Jan 14 12:09:52 vsvr-7865-9 pvescheduler[120630]: INFO: Finished Backup of VM 103 (00:06:09)
Jan 14 12:09:52 vsvr-7865-9 pvescheduler[120630]: INFO: Starting Backup of VM 221 (qemu)
Jan 14 12:11:14 vsvr-7865-9 kernel: corosync invoked oom-killer: gfp_mask=0x4c2cc0(GFP_KERNEL_ACCOUNT|__GFP_NOWARN|__GFP_COMP|__GFP_NOMEMALLOC), order=2, oom_score_adj=0
Jan 14 12:11:15 vsvr-7865-9 kernel: CPU: 7 PID: 2145 Comm: corosync Tainted: P           O       6.5.11-7-pve #1
Jan 14 12:11:15 vsvr-7865-9 kernel: Hardware name: Dell Inc. Precision 7865 Tower/0DMYN9, BIOS 1.1.0 06/14/2023
Jan 14 12:11:15 vsvr-7865-9 kernel: Call Trace:
Jan 14 12:11:15 vsvr-7865-9 kernel:  <TASK>
Jan 14 12:11:15 vsvr-7865-9 kernel:  dump_stack_lvl+0x48/0x70
Jan 14 12:11:15 vsvr-7865-9 kernel:  dump_stack+0x10/0x20
Jan 14 12:11:15 vsvr-7865-9 kernel:  dump_header+0x4f/0x260
Jan 14 12:11:15 vsvr-7865-9 kernel:  oom_kill_process+0x10d/0x1c0
Jan 14 12:11:15 vsvr-7865-9 kernel:  out_of_memory+0x270/0x560
Jan 14 12:11:15 vsvr-7865-9 kernel:  __alloc_pages+0x114f/0x12e0
Jan 14 12:11:15 vsvr-7865-9 kernel:  ? __alloc_skb+0x8a/0x1b0
Jan 14 12:11:15 vsvr-7865-9 kernel:  __kmalloc_large_node+0x7e/0x160
Jan 14 12:11:15 vsvr-7865-9 kernel:  ? memcg_slab_post_alloc_hook+0x1bf/0x280
Jan 14 12:11:15 vsvr-7865-9 kernel:  __kmalloc_node_track_caller.cold+0x5/0xa3
Jan 14 12:11:15 vsvr-7865-9 kernel:  kmalloc_reserve+0x67/0x100
Jan 14 12:11:15 vsvr-7865-9 kernel:  __alloc_skb+0x8a/0x1b0
Jan 14 12:11:15 vsvr-7865-9 kernel:  alloc_skb_with_frags+0x4d/0x240
Jan 14 12:11:15 vsvr-7865-9 kernel:  sock_alloc_send_pskb+0x20e/0x260
Jan 14 12:11:15 vsvr-7865-9 kernel:  ? srso_alias_return_thunk+0x5/0x7f
Jan 14 12:11:15 vsvr-7865-9 kernel:  ? srso_alias_return_thunk+0x5/0x7f
Jan 14 12:11:15 vsvr-7865-9 kernel:  ? wait_for_unix_gc+0x46/0x110
Jan 14 12:11:15 vsvr-7865-9 kernel:  unix_dgram_sendmsg+0x171/0xb80
Jan 14 12:11:15 vsvr-7865-9 kernel:  ? inet_recvmsg+0x121/0x140
Jan 14 12:11:15 vsvr-7865-9 kernel:  ? srso_alias_return_thunk+0x5/0x7f
Jan 14 12:11:15 vsvr-7865-9 kernel:  ? security_socket_recvmsg+0x47/0x80
Jan 14 12:11:15 vsvr-7865-9 kernel:  unix_seqpacket_sendmsg+0x34/0x80
Jan 14 12:11:15 vsvr-7865-9 kernel:  sock_write_iter+0x191/0x1a0
Jan 14 12:11:15 vsvr-7865-9 kernel:  do_iter_readv_writev+0xf2/0x160
Jan 14 12:11:15 vsvr-7865-9 kernel:  do_iter_write+0xa5/0x220
Jan 14 12:11:15 vsvr-7865-9 kernel:  vfs_writev+0xf8/0x1c0
Jan 14 12:11:15 vsvr-7865-9 kernel:  do_writev+0x108/0x170
Jan 14 12:11:15 vsvr-7865-9 kernel:  __x64_sys_writev+0x1c/0x30
Jan 14 12:11:15 vsvr-7865-9 kernel:  do_syscall_64+0x5b/0x90
Jan 14 12:11:15 vsvr-7865-9 kernel:  ? syscall_exit_to_user_mode+0x37/0x60
Jan 14 12:11:15 vsvr-7865-9 kernel:  ? srso_alias_return_thunk+0x5/0x7f
Jan 14 12:11:15 vsvr-7865-9 kernel:  ? do_syscall_64+0x67/0x90
Jan 14 12:11:15 vsvr-7865-9 kernel:  ? srso_alias_return_thunk+0x5/0x7f
Jan 14 12:11:15 vsvr-7865-9 kernel:  ? do_syscall_64+0x67/0x90
Jan 14 12:11:15 vsvr-7865-9 kernel:  ? do_syscall_64+0x67/0x90
Jan 14 12:11:15 vsvr-7865-9 kernel:  ? srso_alias_return_thunk+0x5/0x7f
Jan 14 12:11:15 vsvr-7865-9 kernel:  ? do_syscall_64+0x67/0x90
Jan 14 12:11:15 vsvr-7865-9 kernel:  ? do_syscall_64+0x67/0x90
Jan 14 12:11:15 vsvr-7865-9 kernel:  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
Jan 14 12:11:15 vsvr-7865-9 kernel: RIP: 0033:0x7ff40d5e0c8d
Jan 14 12:11:15 vsvr-7865-9 kernel: Code: 28 89 54 24 1c 48 89 74 24 10 89 7c 24 08 e8 3a 7a f8 ff 8b 54 24 1c 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 14 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 33 44 89 c7 48 89 44 24 08 e8 8e 7a f8 ff 48
Jan 14 12:11:15 vsvr-7865-9 kernel: RSP: 002b:00007ff40a595a60 EFLAGS: 00000293 ORIG_RAX: 0000000000000014
Jan 14 12:11:15 vsvr-7865-9 kernel: RAX: ffffffffffffffda RBX: 00007ff401f5c010 RCX: 00007ff40d5e0c8d
Jan 14 12:11:15 vsvr-7865-9 kernel: RDX: 0000000000000001 RSI: 00007ff40a595b50 RDI: 0000000000000016
Jan 14 12:11:15 vsvr-7865-9 kernel: RBP: 00007ff40a5cfe40 R08: 0000000000000000 R09: 0000000000000004
Jan 14 12:11:15 vsvr-7865-9 kernel: R10: 0000000000000575 R11: 0000000000000293 R12: 00007ff409c8e010
Jan 14 12:11:15 vsvr-7865-9 kernel: R13: 00005559bfce0d70 R14: 00007ff40a595b50 R15: 00007ff40a597e70
Jan 14 12:11:15 vsvr-7865-9 kernel:  </TASK>
Jan 14 12:11:15 vsvr-7865-9 kernel: Mem-Info:
Jan 14 12:11:15 vsvr-7865-9 kernel: active_anon:4066383 inactive_anon:1829383 isolated_anon:0
active_file:1287 inactive_file:7621 isolated_file:0
unevictable:40861 dirty:9 writeback:23
slab_reclaimable:69256 slab_unreclaimable:942755
mapped:22837 shmem:22081 pagetables:25254
sec_pagetables:10902 bounce:0
kernel_misc_reclaimable:0
free:1416123 free_pcp:62 free_cma:0
Jan 14 12:11:15 vsvr-7865-9 kernel: Node 0 active_anon:16265532kB inactive_anon:7317532kB active_file:5552kB inactive_file:30080kB unevictable:163444kB isolated(anon):0kB isolated(file):0kB mapped:91348kB dirty:36kB writeback:92kB shmem:88324kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 3778560kB writeback_tmp:0kB kernel_stack:15072kB pagetables:101016kB sec_pagetables:43608kB all_unreclaimable? no
Jan 14 12:11:15 vsvr-7865-9 kernel: Node 0 DMA free:11264kB boost:0kB min:12kB low:24kB high:36kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jan 14 12:11:15 vsvr-7865-9 kernel: lowmem_reserve[]: 0 2834 64066 64066 64066
Jan 14 12:11:15 vsvr-7865-9 kernel: Node 0 DMA32 free:402384kB boost:0kB min:2988kB low:5888kB high:8788kB reserved_highatomic:6144KB active_anon:580kB inactive_anon:248504kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3026784kB managed:2960920kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jan 14 12:11:15 vsvr-7865-9 kernel: lowmem_reserve[]: 0 0 61232 61232 61232
Jan 14 12:11:15 vsvr-7865-9 kernel: Node 0 Normal free:5250844kB boost:0kB min:64576kB low:127276kB high:189976kB reserved_highatomic:155648KB active_anon:16264840kB inactive_anon:7069140kB active_file:7224kB inactive_file:30224kB unevictable:163444kB writepending:128kB present:63948800kB managed:62709828kB mlocked:163444kB bounce:0kB free_pcp:248kB local_pcp:0kB free_cma:0kB
Jan 14 12:11:15 vsvr-7865-9 kernel: lowmem_reserve[]: 0 0 0 0 0
Jan 14 12:11:15 vsvr-7865-9 kernel: Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 2*4096kB (M) = 11264kB
Jan 14 12:11:15 vsvr-7865-9 kernel: Node 0 DMA32: 19922*4kB (UME) 40337*8kB (UE) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 402384kB
Jan 14 12:11:15 vsvr-7865-9 kernel: Node 0 Normal: 408266*4kB (UM) 452398*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5252248kB
Jan 14 12:11:15 vsvr-7865-9 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Jan 14 12:11:15 vsvr-7865-9 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jan 14 12:11:15 vsvr-7865-9 kernel: 34724 total pagecache pages
Jan 14 12:11:15 vsvr-7865-9 kernel: 0 pages in swap cache
Jan 14 12:11:15 vsvr-7865-9 kernel: Free swap  = 0kB
Jan 14 12:11:15 vsvr-7865-9 kernel: Total swap = 0kB
Jan 14 12:11:15 vsvr-7865-9 kernel: 16747894 pages RAM
Jan 14 12:11:15 vsvr-7865-9 kernel: 0 pages HighMem/MovableOnly
Jan 14 12:11:15 vsvr-7865-9 kernel: 326367 pages reserved
Jan 14 12:11:15 vsvr-7865-9 kernel: 0 pages hwpoisoned
Jan 14 12:11:15 vsvr-7865-9 kernel: Tasks state (memory values in pages):
Jan 14 12:11:15 vsvr-7865-9 kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Jan 14 12:11:15 vsvr-7865-9 kernel: [    832]     0   832    10327      624   114688        0          -250 systemd-journal
Jan 14 12:11:15 vsvr-7865-9 kernel: [    859]     0   859     6719      720    77824        0         -1000 systemd-udevd
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1585]     0  1585    19796      576    53248        0             0 pvefw-logger
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1587]   103  1587     1969      576    53248        0             0 rpcbind
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1590]   101  1590     2292      576    57344        0          -900 dbus-daemon
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1594]     0  1594    38187      336    65536        0         -1000 lxcfs
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1597]     0  1597    69539      480    81920        0             0 pve-lxc-syscall
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1600]     0  1600     2895      720    65536        0             0 smartd
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1601]     0  1601     1766      291    49152        0             0 ksmtuned
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1605]     0  1605     1327      288    49152        0             0 qmeventd
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1607]     0  1607    12489      912    86016        0             0 systemd-logind
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1613]     0  1613      583      192    36864        0         -1000 watchdog-mux
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1617]     0  1617    60164      816    90112        0             0 zed
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1633]     0  1633      618      240    49152        0             0 atopacctd
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1660]     0  1660     5212     4819    86016        0         -1000 atop
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1930]     0  1930     1256      384    49152        0             0 lxc-monitord
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1947]     0  1947     1468      432    49152        0             0 agetty
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1952]     0  1952     3853     1152    73728        0         -1000 sshd
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1972]   100  1972     4714      634    57344        0             0 chronyd
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1977]   100  1977     2632      496    57344        0             0 chronyd
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2016]     0  2016   126542      691   151552        0             0 rrdcached
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2031]     0  2031   144103    19819   471040        0             0 pmxcfs
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2123]     0  2123    10664      772    69632        0             0 master
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2125]   104  2125    10776      960    69632        0             0 qmgr
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2130]     0  2130   139855    41667   401408        0             0 corosync
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2131]     0  2131     1652      480    53248        0             0 cron
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2149]     0  2149    39286    24713   319488        0             0 pvestatd
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2150]     0  2150    39311    24587   294912        0             0 pve-firewall
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2154]     0  2154      615      240    45056        0             0 bpfilter_umh
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2177]     0  2177    58307    34274   401408        0             0 pvedaemon
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2178]     0  2178    60436    34515   425984        0             0 pvedaemon worke
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2179]     0  2179    60439    34563   425984        0             0 pvedaemon worke
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2180]     0  2180    60439    34563   425984        0             0 pvedaemon worke
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2185]     0  2185    54841    27683   360448        0             0 pve-ha-crm
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2187]    33  2187    58652    34385   421888        0             0 pveproxy
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2193]    33  2193    20199    13189   180224        0             0 spiceproxy
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2194]    33  2194    20260    13286   180224        0             0 spiceproxy work
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2195]     0  2195    54707    27688   376832        0             0 pve-ha-lrm
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2209]     0  2209  2670274  2096191 19308544        0             0 kvm
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2353]     0  2353  4741529  4214387 36196352        0             0 kvm
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2463]     0  2463     3287      919    57344        0             0 swtpm
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2471]     0  2471  2687472  2146671 19619840        0             0 kvm
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2686]     0  2686     3287      871    61440        0             0 swtpm
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2719]     0  2719  2654217  2119771 19456000        0             0 kvm
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2821]     0  2821    53610    28237   360448        0             0 pvescheduler
Jan 14 12:11:15 vsvr-7865-9 kernel: [  29066]    33 29066    60818    35057   438272        0             0 pveproxy worker
Jan 14 12:11:15 vsvr-7865-9 kernel: [  36482]    33 36482    60818    35201   438272        0             0 pveproxy worker
Jan 14 12:11:15 vsvr-7865-9 kernel: [  37138]    33 37138    60818    34961   438272        0             0 pveproxy worker
Jan 14 12:11:15 vsvr-7865-9 kernel: [  88453]     0 88453     4934     1536    81920        0           100 systemd
Jan 14 12:11:15 vsvr-7865-9 kernel: [  88454]     0 88454    42439     1344    94208        0           100 (sd-pam)
Jan 14 12:11:15 vsvr-7865-9 kernel: [  89354]     0 89354     4753     1853    77824        0             0 sshd
Jan 14 12:11:15 vsvr-7865-9 kernel: [  89410]     0 89410    53010    28845   409600        0             0 qm
Jan 14 12:11:15 vsvr-7865-9 kernel: [  97689]   104 97689    10764      960    69632        0             0 pickup
Jan 14 12:11:15 vsvr-7865-9 kernel: [ 120630]     0 120630    57986    29321   397312        0             0 task UPID:vsvr-
Jan 14 12:11:15 vsvr-7865-9 kernel: [ 125959]     0 125959     1366      336    49152        0             0 sleep
Jan 14 12:11:15 vsvr-7865-9 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=corosync.service,mems_allowed=0,global_oom,task_memcg=/qemu.slice/103.scope,task=kvm,pid=2353,uid=0
Jan 14 12:11:15 vsvr-7865-9 kernel: Out of memory: Killed process 2353 (kvm) total-vm:18966116kB, anon-rss:16853764kB, file-rss:3784kB, shmem-rss:0kB, UID:0 pgtables:35348kB oom_score_adj:0
Jan 14 12:11:15 vsvr-7865-9 systemd[1]: 103.scope: A process of this unit has been killed by the OOM killer.
Jan 14 12:11:15 vsvr-7865-9 systemd[1]: 103.scope: Failed with result 'oom-kill'.
Jan 14 12:11:15 vsvr-7865-9 systemd[1]: 103.scope: Consumed 30min 31.305s CPU time.
Jan 14 12:11:17 vsvr-7865-9 kernel: oom_reaper: reaped process 2353 (kvm), now anon-rss:0kB, file-rss:744kB, shmem-rss:0kB
Jan 14 12:11:19 vsvr-7865-9 kernel:  zd64: p1 p2
Jan 14 12:11:19 vsvr-7865-9 kernel: vmbr1: port 3(tap103i0) entered disabled state
Jan 14 12:11:19 vsvr-7865-9 kernel: tap103i0 (unregistering): left allmulticast mode
Jan 14 12:11:19 vsvr-7865-9 kernel: vmbr1: port 3(tap103i0) entered disabled state
Jan 14 12:11:19 vsvr-7865-9 kernel:  zd128: p1
Jan 14 12:11:19 vsvr-7865-9 qmeventd[126402]: Starting cleanup for 103
Jan 14 12:11:19 vsvr-7865-9 qmeventd[126402]: Finished cleanup for 103
Jan 14 12:12:37 vsvr-7865-9 pvescheduler[120630]: INFO: Finished Backup of VM 221 (00:02:45)
Jan 14 12:12:37 vsvr-7865-9 pvescheduler[120630]: INFO: Starting Backup of VM 223 (qemu)
Jan 14 12:14:37 vsvr-7865-9 pvescheduler[120630]: INFO: Finished Backup of VM 223 (00:02:00)
Jan 14 12:14:37 vsvr-7865-9 pvescheduler[120630]: INFO: Backup job finished successfully

Any ideas would be appreciated.

Thank you,

 

[Moayad]

Hi,

Jan 14 12:11:15 vsvr-7865-9 kernel: Out of memory: Killed process 2353 (kvm) total-vm:18966116kB, anon-rss:16853764kB, file-rss:3784kB, shmem-rss:0kB, UID:0 pgtables:35348kB oom_score_adj:0

In the log you've provided, it says that there is an OOM (Out of Memory). Overcommitting memory can lead to situations where the OOM Killer is triggered. So, I would give more RAM memory to your Proxmox VE.

 

[kbrault]

Thank you Moayad,
These machines have 64GB. ZFS grabs up to 32GB and the VMs are allocated a total of 40GB (no ballooning) so it looks like ZFS is not releasing RAM fast enough during a backup.
I'll get some more RAM anyway.

 

[kbrault]

I have RAM on the way and in the meantime, I am limiting ARC to 16 GB but I still have the problem. Backing up uses all the RAM and system is shutting down a VM. This just started happening in the last week. Been running fine for the previous six months.

Here is the output of "top" during a backup:


Does this look abnormal?

Thanks again for your help.

 

[Moayad]

Hi,

Could you provide us with the syslog at the time when the VM was killed or shut down? You can use the journalctl to generate a syslog with specific time as the following command:

journalctl --since "2024-01-18 01:00" --until "2024-01-18 15:00" > /tmp/$(hostname)-Syslog.txt

You may edit the date/time in the above command.

Please, also provide us with the VM config qm config <VMID> and the output of pveversion -v.

 

[flece]

same Problem on my Site. Specs are just doublet and all VM are Debian
as you can see from all specs below i have currently 15GB free MEM
that 15GB gets eaten by vmbackup and causes OOM.

also pls explain the huge difference from what htop says (51G used) to free (116G used)

to me it looks like ARC-ZFS is calculated as used from free and seen as cache from htop


Specs

PVE Version  = 8.1.3
pve-qemu-kvm = 8.1.2-4
Host RAM     = 128G
allocated all VM in MiB = 90112
ZFS-ARC      = 64 GB (not touched, installed with Proxmox 8.0.3)
options      = balloon: 1 Minimium = Memory (effectively no ballooning)

numbers from "htop"


numbers from "free -w"

           total          used           free         shared    buffers   cache     available
Mem:       131.720.600    116.216.008    16.263.208   80.176    0         275.492   15.504.592

numbers from "ps"

Allocated      Virt    Resident  Share
32.00 GiB      34.2 G   9761 M     12288   57.1  7.6  2h28:09 /usr/bin/kvm -id 113
12.00 GiB      19.6 G   7676 M     10424    9.1  6.0 17h31:11 /usr/bin/kvm -id 102
 8.00 GiB      15.9 G   3603 M      8304    9.7  2.8  8h10:58 /usr/bin/kvm -id 103      (Virt double then allowed to VM? Why?)
 8.00 GiB      9657 M   2652 M     10752    1.3  2.1  4h50:25 /usr/bin/kvm -id 133
 8.00 GiB      9633 M   2027 M      9216    3.9  1.6 12h45:30 /usr/bin/kvm -id 101
 8.00 GiB      9397 M   6374 M     11136    1.9  5.0  7h04:44 /usr/bin/kvm -id 120
 4.00 GiB      6023 M   2800 M     14208    1.9  2.2  4h08:26 /usr/bin/kvm -id 111
 4.00 GiB      5407 M   3130 M      4224    0.6  2.4  4h32:37 /usr/bin/kvm -id 107
 2.00 GiB      3443 M   1875 M      9224    1.9  1.5  5h36:21 /usr/bin/kvm -id 108
 2.00 GiB      3085 M    688 M     11136    3.9  0.5  9h26:51 /usr/bin/kvm -id 126

 

[kbrault]

Does this work?

qm config 103
agent: 1,fstrim_cloned_disks=1
balloon: 0
boot: order=scsi0
cores: 10
cpu: x86-64-v2-AES
hotplug: disk,network,usb,cpu
ide3: none,media=cdrom
machine: pc-q35-5.1
memory: 16384
name: <removed>
net0: virtio=<removed>,bridge=vmbr1
numa: 0
onboot: 1
ostype: win10
scsi0: storage:vm-103-disk-0,cache=writeback,discard=on,size=128G,ssd=1
scsi1: storage:vm-103-disk-1,cache=writeback,discard=on,size=320G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=0702219a-50ef-470b-b85b-10be52963f19
sockets: 1
startup: order=3
vmgenid: c9bd083d-913d-4647-8187-648276034ca5

pveversion -v
proxmox-ve: 8.1.0 (running kernel: 6.5.11-7-pve)
pve-manager: 8.1.4 (running version: 8.1.4/ec5affc9e41f1d79)
proxmox-kernel-helper: 8.1.0
pve-kernel-6.2: 8.0.5
proxmox-kernel-6.5: 6.5.11-7
proxmox-kernel-6.5.11-7-pve-signed: 6.5.11-7
proxmox-kernel-6.5.11-4-pve-signed: 6.5.11-4
proxmox-kernel-6.2.16-20-pve: 6.2.16-20
proxmox-kernel-6.2: 6.2.16-20
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.0.7
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.1.0
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.5
libpve-network-perl: 0.9.5
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.0.5
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve4
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.1.2-1
proxmox-backup-file-restore: 3.1.2-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.2
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.1.3
pve-cluster: 8.0.5
pve-container: 5.0.8
pve-docs: 8.1.3
pve-edk2-firmware: 4.2023.08-2
pve-firewall: 5.0.3
pve-firmware: 3.9-1
pve-ha-manager: 4.0.3
pve-i18n: 3.1.5
pve-qemu-kvm: 8.1.2-6
pve-xtermjs: 5.3.0-3
qemu-server: 8.0.10
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.2-pve1

journalctl --since "2024-1-14 12:11" --until "2024-01-14 12:13"
Jan 14 12:11:15 vsvr-7865-9 kernel: Tasks state (memory values in pages):
Jan 14 12:11:15 vsvr-7865-9 kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Jan 14 12:11:15 vsvr-7865-9 kernel: [    832]     0   832    10327      624   114688        0          -250 systemd-journal
Jan 14 12:11:15 vsvr-7865-9 kernel: [    859]     0   859     6719      720    77824        0         -1000 systemd-udevd
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1585]     0  1585    19796      576    53248        0             0 pvefw-logger
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1587]   103  1587     1969      576    53248        0             0 rpcbind
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1590]   101  1590     2292      576    57344        0          -900 dbus-daemon
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1594]     0  1594    38187      336    65536        0         -1000 lxcfs
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1597]     0  1597    69539      480    81920        0             0 pve-lxc-syscall
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1600]     0  1600     2895      720    65536        0             0 smartd
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1601]     0  1601     1766      291    49152        0             0 ksmtuned
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1605]     0  1605     1327      288    49152        0             0 qmeventd
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1607]     0  1607    12489      912    86016        0             0 systemd-logind
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1613]     0  1613      583      192    36864        0         -1000 watchdog-mux
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1617]     0  1617    60164      816    90112        0             0 zed
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1633]     0  1633      618      240    49152        0             0 atopacctd
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1660]     0  1660     5212     4819    86016        0         -1000 atop
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1930]     0  1930     1256      384    49152        0             0 lxc-monitord
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1947]     0  1947     1468      432    49152        0             0 agetty
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1952]     0  1952     3853     1152    73728        0         -1000 sshd
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1972]   100  1972     4714      634    57344        0             0 chronyd
Jan 14 12:11:15 vsvr-7865-9 kernel: [   1977]   100  1977     2632      496    57344        0             0 chronyd
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2016]     0  2016   126542      691   151552        0             0 rrdcached
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2031]     0  2031   144103    19819   471040        0             0 pmxcfs
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2123]     0  2123    10664      772    69632        0             0 master
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2125]   104  2125    10776      960    69632        0             0 qmgr
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2130]     0  2130   139855    41667   401408        0             0 corosync
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2131]     0  2131     1652      480    53248        0             0 cron
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2149]     0  2149    39286    24713   319488        0             0 pvestatd
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2150]     0  2150    39311    24587   294912        0             0 pve-firewall
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2154]     0  2154      615      240    45056        0             0 bpfilter_umh
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2177]     0  2177    58307    34274   401408        0             0 pvedaemon
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2178]     0  2178    60436    34515   425984        0             0 pvedaemon worke
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2179]     0  2179    60439    34563   425984        0             0 pvedaemon worke
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2180]     0  2180    60439    34563   425984        0             0 pvedaemon worke
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2185]     0  2185    54841    27683   360448        0             0 pve-ha-crm
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2187]    33  2187    58652    34385   421888        0             0 pveproxy
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2193]    33  2193    20199    13189   180224        0             0 spiceproxy
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2194]    33  2194    20260    13286   180224        0             0 spiceproxy work
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2195]     0  2195    54707    27688   376832        0             0 pve-ha-lrm
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2209]     0  2209  2670274  2096191 19308544        0             0 kvm
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2353]     0  2353  4741529  4214387 36196352        0             0 kvm
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2463]     0  2463     3287      919    57344        0             0 swtpm
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2471]     0  2471  2687472  2146671 19619840        0             0 kvm
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2686]     0  2686     3287      871    61440        0             0 swtpm
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2719]     0  2719  2654217  2119771 19456000        0             0 kvm
Jan 14 12:11:15 vsvr-7865-9 kernel: [   2821]     0  2821    53610    28237   360448        0             0 pvescheduler
Jan 14 12:11:15 vsvr-7865-9 kernel: [  29066]    33 29066    60818    35057   438272        0             0 pveproxy worker
Jan 14 12:11:15 vsvr-7865-9 kernel: [  36482]    33 36482    60818    35201   438272        0             0 pveproxy worker
Jan 14 12:11:15 vsvr-7865-9 kernel: [  37138]    33 37138    60818    34961   438272        0             0 pveproxy worker
Jan 14 12:11:15 vsvr-7865-9 kernel: [  88453]     0 88453     4934     1536    81920        0           100 systemd
Jan 14 12:11:15 vsvr-7865-9 kernel: [  88454]     0 88454    42439     1344    94208        0           100 (sd-pam)
Jan 14 12:11:15 vsvr-7865-9 kernel: [  89354]     0 89354     4753     1853    77824        0             0 sshd
Jan 14 12:11:15 vsvr-7865-9 kernel: [  89410]     0 89410    53010    28845   409600        0             0 qm
Jan 14 12:11:15 vsvr-7865-9 kernel: [  97689]   104 97689    10764      960    69632        0             0 pickup
Jan 14 12:11:15 vsvr-7865-9 kernel: [ 120630]     0 120630    57986    29321   397312        0             0 task UPID:vsvr-
Jan 14 12:11:15 vsvr-7865-9 kernel: [ 125959]     0 125959     1366      336    49152        0             0 sleep
Jan 14 12:11:15 vsvr-7865-9 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=corosync.service,mems_allowed=0,global_oom,task_memcg=/qemu.slice/103.scope,task=kvm,pid=2353,uid=0
Jan 14 12:11:15 vsvr-7865-9 kernel: Out of memory: Killed process 2353 (kvm) total-vm:18966116kB, anon-rss:16853764kB, file-rss:3784kB, shmem-rss:0kB, UID:0 pgtables:35348kB oom_score_adj:0
Jan 14 12:11:15 vsvr-7865-9 systemd[1]: 103.scope: A process of this unit has been killed by the OOM killer.
Jan 14 12:11:15 vsvr-7865-9 systemd[1]: 103.scope: Failed with result 'oom-kill'.
Jan 14 12:11:15 vsvr-7865-9 systemd[1]: 103.scope: Consumed 30min 31.305s CPU time.
Jan 14 12:11:17 vsvr-7865-9 kernel: oom_reaper: reaped process 2353 (kvm), now anon-rss:0kB, file-rss:744kB, shmem-rss:0kB
Jan 14 12:11:19 vsvr-7865-9 kernel:  zd64: p1 p2
Jan 14 12:11:19 vsvr-7865-9 kernel: vmbr1: port 3(tap103i0) entered disabled state
Jan 14 12:11:19 vsvr-7865-9 kernel: tap103i0 (unregistering): left allmulticast mode
Jan 14 12:11:19 vsvr-7865-9 kernel: vmbr1: port 3(tap103i0) entered disabled state
Jan 14 12:11:19 vsvr-7865-9 kernel:  zd128: p1
Jan 14 12:11:19 vsvr-7865-9 qmeventd[126402]: Starting cleanup for 103
Jan 14 12:11:19 vsvr-7865-9 qmeventd[126402]: Finished cleanup for 103
Jan 14 12:12:37 vsvr-7865-9 pvescheduler[120630]: INFO: Finished Backup of VM 221 (00:02:45)
Jan 14 12:12:37 vsvr-7865-9 pvescheduler[120630]: INFO: Starting Backup of VM 223 (qemu)

 

[kbrault]

Dropping the upper limit for ZFS cache I think did the trick.

 

[flece]

I changed "/sys/module/zfs/parameters/zfs_arc_max" from 0 to "echo "$[10 * 1024*1024*1024]" >/sys/module/zfs/parameters/zfs_arc_max"
effectively changed ARC from 64G to 10G

I conclude that memory used by ARC does not get released for other processes. I for myself consider ARC as not free and that is what you can see in the WebUI from Proxmox and "free"

for htop it is "cache" but unlike cache from ext4 this cache can not to be expected to get released.