High swap and ram usage , proxmox/ceph

[brucexx]

I have 3 nodes dedicated to storage, exact the same hardware spec, and setup in the cluster with the same hard drive modes for ceph OSDs, everything the same. The PVE and Ceph clusters works perfectly fine (a little bit of OI wait but I was told it is normal). All the VMs work with no problems.

I noticed on just one node that the swap is full and the RAM usage is going up every day (currently 81%), it turns out the process responsible for 65% of RAM usage and 99% of swap usage is ceph-mon (I checked with proc/{pidid}/status.

Any advice or anything you guys recommend checking maybe rebooting ? It is just this one node, the other nodes swap usage is non existent and ram usage is 15%.

Ver:
proxmox-ve: 4.3-71 (running kernel: 4.4.21-1-pve)
pve-manager: 4.3-9 (running version: 4.3-9/f7c6f0cd)
pve-kernel-4.4.21-1-pve: 4.4.21-71
pve-kernel-4.4.19-1-pve: 4.4.19-66
lvm2: 2.02.116-pve3
corosync-pve: 2.4.0-1
libqb0: 1.0-1
pve-cluster: 4.0-46
qemu-server: 4.0-92
pve-firmware: 1.1-10
libpve-common-perl: 4.0-79
libpve-access-control: 4.0-19
libpve-storage-perl: 4.0-68
pve-libspice-server1: 0.12.8-1
vncterm: 1.2-1
pve-docs: 4.3-12
pve-qemu-kvm: 2.7.0-4
pve-container: 1.0-80
pve-firewall: 2.0-31
pve-ha-manager: 1.0-35
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u2
lxc-pve: 2.0.5-1
lxcfs: 2.0.4-pve2
criu: 1.6.0-1
novnc-pve: 0.5-8
smartmontools: 6.5+svn4324-1~pve80
zfsutils: 0.6.5.8-pve13~bpo80
ceph: 0.94.9-1~bpo80+1

Thank you

 

[brucexx]

So it turns out I had a problem with one OSD which was down/out and nothing I could do to bringing it back up. I just removed it and successfully re-added it and it is working fine. During this process as I moved all the VMs to a different storage I rebooted the all the nodes and the ram and swap got reset. Now as I am watching the RAM usage I see that it is growing overtime while the load is not (the same number of VMs and load on them). The process that takes majority of RAM is ceph-mon. I am not sure if at some point the RAM usage will just stabilize and will stay on some level or it will be growing slowly until I will have to reboot again. The RAM usage after the reboot is growing on all nodes at least for now. How does it compare to your ceph storage when it comes to RAM usage over a long period of time ?

Thank you