One proxmox node
4 - VM's
1 - ZFS NFS
Every once in awhile, when I move files from the VM that I have setup to access the ZFS NFS share, the VM will crash. After it reboots, I notice the total memory available on the VM starts to dwindle until it again, starts crashing. The only fix I can find for this is to completely reboot the node and after that everything normalizes.
here's top on the VM after a few days when it's running normally (and I haven't accessed the ZFS NFS share):
Here's top on the VM when it's crashing. The KiB Mem total starts at the max of 64gig and then ram starts disappearing... in this case it's already down to 52gig total after 18 minutes. It will slowly go down to zero and crash again.
After 40 minutes, total ram on the VM is down to 38gig
Any ideas?
------
root@proxmox:~# pveversion -v
proxmox-ve: 6.1-2 (running kernel: 5.3.18-3-pve)
pve-manager: 6.1-8 (running version: 6.1-8/806edfe1)
pve-kernel-helper: 6.1-8
pve-kernel-5.3: 6.1-6
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.18-1-pve: 5.3.18-1
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libpve-access-control: 6.0-6
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.0-17
libpve-guest-common-perl: 3.0-5
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-5
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 3.2.1-1
lxcfs: 4.0.1-pve1
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-3
pve-cluster: 6.1-4
pve-container: 3.0-23
pve-docs: 6.1-6
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.0-10
pve-firmware: 3.0-7
pve-ha-manager: 3.0-9
pve-i18n: 2.0-4
pve-qemu-kvm: 4.1.1-4
pve-xtermjs: 4.3.0-1
qemu-server: 6.1-7
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1
4 - VM's
1 - ZFS NFS
Every once in awhile, when I move files from the VM that I have setup to access the ZFS NFS share, the VM will crash. After it reboots, I notice the total memory available on the VM starts to dwindle until it again, starts crashing. The only fix I can find for this is to completely reboot the node and after that everything normalizes.
here's top on the VM after a few days when it's running normally (and I haven't accessed the ZFS NFS share):
Code:
top - 11:10:23 up 3 days, 23:08, 2 users, load average: 0.70, 0.56, 0.48
Tasks: 268 total, 1 running, 267 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 1.1 sy, 4.1 ni, 89.8 id, 3.6 wa, 0.0 hi, 1.4 si, 0.0 st
KiB Mem : 65806168 total, 352844 free, 3962160 used, 61491164 buff/cache
KiB Swap: 33030140 total, 32992836 free, 37304 used. 61302144 avail Mem
Code:
[root@localhost~]# free
total used free shared buff/cache available
Mem: 65806168 3962512 411388 5560 61432268 61301984
Swap: 33030140 37304 32992836Here's top on the VM when it's crashing. The KiB Mem total starts at the max of 64gig and then ram starts disappearing... in this case it's already down to 52gig total after 18 minutes. It will slowly go down to zero and crash again.
Code:
top - 10:08:55 up 18 min, 2 users, load average: 0.04, 0.27, 0.21
Tasks: 253 total, 1 running, 252 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.1 us, 0.0 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 52477784 total, 50751324 free, 1109224 used, 617236 buff/cache
KiB Swap: 33030140 total, 33030140 free, 0 used. 50872556 avail Mem
Code:
[root@localhost ~]# free
total used free shared buff/cache available
Mem: 51453784 1107844 49712272 9184 633668 49849924
Swap: 33030140 0 33030140After 40 minutes, total ram on the VM is down to 38gig
Code:
top - 10:30:53 up 40 min, 2 users, load average: 0.05, 0.19, 0.25
Tasks: 254 total, 1 running, 253 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.1 us, 0.1 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 38960984 total, 36769848 free, 1090380 used, 1100756 buff/cache
KiB Swap: 33030140 total, 33030140 free, 0 used. 37368412 avail Mem
Code:
[root@localhost ~]# free
total used free shared buff/cache available
Mem: 38551384 1089924 36352536 9188 1108924 36959300
Swap: 33030140 0 33030140Any ideas?
------
root@proxmox:~# pveversion -v
proxmox-ve: 6.1-2 (running kernel: 5.3.18-3-pve)
pve-manager: 6.1-8 (running version: 6.1-8/806edfe1)
pve-kernel-helper: 6.1-8
pve-kernel-5.3: 6.1-6
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.18-1-pve: 5.3.18-1
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libpve-access-control: 6.0-6
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.0-17
libpve-guest-common-perl: 3.0-5
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-5
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 3.2.1-1
lxcfs: 4.0.1-pve1
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-3
pve-cluster: 6.1-4
pve-container: 3.0-23
pve-docs: 6.1-6
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.0-10
pve-firmware: 3.0-7
pve-ha-manager: 3.0-9
pve-i18n: 2.0-4
pve-qemu-kvm: 4.1.1-4
pve-xtermjs: 4.3.0-1
qemu-server: 6.1-7
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1
Last edited:
