ZFS Memory issues (leak?)

@sshutdownow
Note that this issue is related to the network and not ZFS. Do you have a VM running with lots of traffic/packets per second?

There are one VM and one LXC on this server. Server has 64 GB RAM, CT is restricted to use 12GB, VM to 2GB. Total network load is less than 60Mbits, I suppose, that it is not very high.20171206-pve00-kashtan-network.png

On previous versions of kernel there were a such messages in log just before reboot:

Nov 27 23:17:41 pve00-kashtan kernel: [83717.531948] Possible memory allocation deadlock: size=46288 lflags=0x1404200
Nov 27 23:17:41 pve00-kashtan kernel: [83717.531949] CPU: 3 PID: 3349 Comm: zpool Tainted: P O 4.13.4-1-pve #1
Nov 27 23:17:41 pve00-kashtan kernel: [83717.531949] Hardware name: Supermicro SYS-5019S-MR/X11SSH-F, BIOS 1.0a 12/29/2015
Nov 27 23:17:41 pve00-kashtan kernel: [83717.531949] Call Trace:
Nov 27 23:17:41 pve00-kashtan kernel: [83717.531951] dump_stack+0x63/0x8b
Nov 27 23:17:41 pve00-kashtan kernel: [83717.531953] spl_kmem_alloc_impl+0x173/0x180 [spl]
Nov 27 23:17:41 pve00-kashtan kernel: [83717.531955] spl_vmem_alloc+0x19/0x20 [spl]
Nov 27 23:17:41 pve00-kashtan kernel: [83717.531957] nv_alloc_sleep_spl+0x1f/0x30 [znvpair]
Nov 27 23:17:41 pve00-kashtan kernel: [83717.531959] nv_mem_zalloc.isra.0+0x15/0x40 [znvpair]
Nov 27 23:17:41 pve00-kashtan kernel: [83717.531962] nvlist_xpack+0xb4/0x110 [znvpair]
Nov 27 23:17:41 pve00-kashtan kernel: [83717.531964] ? nvlist_common.part.89+0x118/0x200 [znvpair]
Nov 27 23:17:41 pve00-kashtan kernel: [83717.531966] nvlist_pack+0x34/0x40 [znvpair]
Nov 27 23:17:41 pve00-kashtan kernel: [83717.531968] fnvlist_pack+0x3e/0xa0 [znvpair]
Nov 27 23:17:41 pve00-kashtan kernel: [83717.531988] put_nvlist+0x95/0x100 [zfs]
Nov 27 23:17:41 pve00-kashtan kernel: [83717.532008] zfs_ioc_pool_stats+0x50/0x90 [zfs]
Nov 27 23:17:41 pve00-kashtan kernel: [83717.532028] zfsdev_ioctl+0x5d4/0x660 [zfs]
Nov 27 23:17:41 pve00-kashtan kernel: [83717.532029] do_vfs_ioctl+0xa3/0x610
Nov 27 23:17:41 pve00-kashtan kernel: [83717.532030] ? handle_mm_fault+0xce/0x1c0
Nov 27 23:17:41 pve00-kashtan kernel: [83717.532031] ? __do_page_fault+0x266/0x4e0
Nov 27 23:17:41 pve00-kashtan kernel: [83717.532032] SyS_ioctl+0x79/0x90
Nov 27 23:17:41 pve00-kashtan kernel: [83717.532033] entry_SYSCALL_64_fastpath+0x1e/0xa9
Nov 27 23:17:41 pve00-kashtan kernel: [83717.532033] RIP: 0033:0x7fca47ea9e07
Nov 27 23:17:41 pve00-kashtan kernel: [83717.532033] RSP: 002b:00007ffdd03c1dd8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Nov 27 23:17:41 pve00-kashtan kernel: [83717.532034] RAX: ffffffffffffffda RBX: 00007fca48162b00 RCX: 00007fca47ea9e07
Nov 27 23:17:41 pve00-kashtan kernel: [83717.532035] RDX: 00007ffdd03c1e00 RSI: 0000000000005a05 RDI: 0000000000000003
Nov 27 23:17:41 pve00-kashtan kernel: [83717.532035] RBP: 000056099abe1fb0 R08: 0000000000000003 R09: 0000000000010010
Nov 27 23:17:41 pve00-kashtan kernel: [83717.532035] R10: 00007fca47eeab20 R11: 0000000000000246 R12: 0000000000010000
Nov 27 23:17:41 pve00-kashtan kernel: [83717.532036] R13: 0000000000020060 R14: 000056099abe1fa0 R15: 00007ffdd03c5450
 
Hello,
I can confirm that the issue was fixed with the last kernel update. The host is stable now and no memory leaks happened.
 
  • Like
Reactions: chrone

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!