kernel-4.15.18-56 NFS freeze with Intel Nic XL710

[Paolo_Bernardoni]

Hi, i've installed a new Intel Nic XL710 into all my cluster nodes to connect the new NFS storage based on Centos 7.

After migrate about 20 virtual servers from old storage, the migration node freeze with kernel error below (freeze only nfs client that point to a new NFS storage, other storage continue to work correctly via other NIC 10Gbit, XL710 Nic just upgrated to the latest Intel firmware version.):

Some suggestion ?
Thank you
Paolo


May 20 20:31:16 proxmox50 kernel: [162470.750755] nfs: server 10.10.12.200 not responding, still trying
May 20 20:33:58 proxmox50 kernel: [162632.534556] mv D 0 135889 121004 0x00000000
May 20 20:33:58 proxmox50 kernel: [162632.534559] Call Trace:
May 20 20:33:58 proxmox50 kernel: [162632.534568] __schedule+0x3e0/0x870
May 20 20:33:58 proxmox50 kernel: [162632.534571] schedule+0x36/0x80
May 20 20:33:58 proxmox50 kernel: [162632.534572] io_schedule+0x16/0x40
May 20 20:33:58 proxmox50 kernel: [162632.534577] wait_on_page_bit_common+0xf3/0x190
May 20 20:33:58 proxmox50 kernel: [162632.534580] ? page_cache_tree_insert+0xe0/0xe0
May 20 20:33:58 proxmox50 kernel: [162632.534583] __filemap_fdatawait_range+0xfa/0x170
May 20 20:33:58 proxmox50 kernel: [162632.534587] filemap_write_and_wait+0x4d/0x90
May 20 20:33:58 proxmox50 kernel: [162632.534603] nfs_wb_all+0x20/0x110 [nfs]
May 20 20:33:58 proxmox50 kernel: [162632.534612] nfs_setattr+0x204/0x220 [nfs]
May 20 20:33:58 proxmox50 kernel: [162632.534617] notify_change+0x2f4/0x440
May 20 20:33:58 proxmox50 kernel: [162632.534620] utimes_common+0xf9/0x1b0
May 20 20:33:58 proxmox50 kernel: [162632.534624] do_utimes+0xde/0x160
May 20 20:33:58 proxmox50 kernel: [162632.534626] SyS_utimensat+0x87/0xd0
May 20 20:33:58 proxmox50 kernel: [162632.534629] do_syscall_64+0x73/0x130
May 20 20:33:58 proxmox50 kernel: [162632.534631] ? do_syscall_64+0x73/0x130
May 20 20:33:58 proxmox50 kernel: [162632.534634] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
May 20 20:33:58 proxmox50 kernel: [162632.534636] RIP: 0033:0x7fdb47d4cc96
May 20 20:33:58 proxmox50 kernel: [162632.534637] RSP: 002b:00007ffee84b7468 EFLAGS: 00000246 ORIG_RAX: 0000000000000118
May 20 20:33:58 proxmox50 kernel: [162632.534639] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007fdb47d4cc96
May 20 20:33:58 proxmox50 kernel: [162632.534640] RDX: 00007ffee84b7480 RSI: 0000000000000000 RDI: 0000000000000004
May 20 20:33:58 proxmox50 kernel: [162632.534641] RBP: 0000563a983c8e30 R08: 000000003614525e R09: 0000000000000000
May 20 20:33:58 proxmox50 kernel: [162632.534642] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000004
May 20 20:33:58 proxmox50 kernel: [162632.534643] R13: 0000000000000000 R14: 00007ffee84b7aa0 R15: 0000000000000000
May 20 20:35:59 proxmox50 kernel: [162753.360277] mv D 0 135889 121004 0x00000000
May 20 20:35:59 proxmox50 kernel: [162753.360280] Call Trace:
May 20 20:35:59 proxmox50 kernel: [162753.360289] __schedule+0x3e0/0x870
May 20 20:35:59 proxmox50 kernel: [162753.360291] schedule+0x36/0x80
May 20 20:35:59 proxmox50 kernel: [162753.360293] io_schedule+0x16/0x40
May 20 20:35:59 proxmox50 kernel: [162753.360298] wait_on_page_bit_common+0xf3/0x190
May 20 20:35:59 proxmox50 kernel: [162753.360301] ? page_cache_tree_insert+0xe0/0xe0
May 20 20:35:59 proxmox50 kernel: [162753.360304] __filemap_fdatawait_range+0xfa/0x170
May 20 20:35:59 proxmox50 kernel: [162753.360308] filemap_write_and_wait+0x4d/0x90
May 20 20:35:59 proxmox50 kernel: [162753.360326] nfs_wb_all+0x20/0x110 [nfs]
May 20 20:35:59 proxmox50 kernel: [162753.360335] nfs_setattr+0x204/0x220 [nfs]
May 20 20:35:59 proxmox50 kernel: [162753.360339] notify_change+0x2f4/0x440
May 20 20:35:59 proxmox50 kernel: [162753.360343] utimes_common+0xf9/0x1b0
May 20 20:35:59 proxmox50 kernel: [162753.360345] do_utimes+0xde/0x160
May 20 20:35:59 proxmox50 kernel: [162753.360348] SyS_utimensat+0x87/0xd0
May 20 20:35:59 proxmox50 kernel: [162753.360351] do_syscall_64+0x73/0x130
May 20 20:35:59 proxmox50 kernel: [162753.360353] ? do_syscall_64+0x73/0x130
May 20 20:35:59 proxmox50 kernel: [162753.360356] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
May 20 20:35:59 proxmox50 kernel: [162753.360359] RIP: 0033:0x7fdb47d4cc96
May 20 20:35:59 proxmox50 kernel: [162753.360360] RSP: 002b:00007ffee84b7468 EFLAGS: 00000246 ORIG_RAX: 0000000000000118
May 20 20:35:59 proxmox50 kernel: [162753.360361] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007fdb47d4cc96
May 20 20:35:59 proxmox50 kernel: [162753.360362] RDX: 00007ffee84b7480 RSI: 0000000000000000 RDI: 0000000000000004
May 20 20:35:59 proxmox50 kernel: [162753.360363] RBP: 0000563a983c8e30 R08: 000000003614525e R09: 0000000000000000
May 20 20:35:59 proxmox50 kernel: [162753.360364] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000004
May 20 20:35:59 proxmox50 kernel: [162753.360365] R13: 0000000000000000 R14: 00007ffee84b7aa0 R15: 0000000000000000
May 20 20:38:00 proxmox50 kernel: [162874.186065] mv D 0 135889 121004 0x00000000
May 20 20:38:00 proxmox50 kernel: [162874.186073] Call Trace:
May 20 20:38:00 proxmox50 kernel: [162874.186083] __schedule+0x3e0/0x870
May 20 20:38:00 proxmox50 kernel: [162874.186086] schedule+0x36/0x80
May 20 20:38:00 proxmox50 kernel: [162874.186088] io_schedule+0x16/0x40
May 20 20:38:00 proxmox50 kernel: [162874.186093] wait_on_page_bit_common+0xf3/0x190
May 20 20:38:00 proxmox50 kernel: [162874.186096] ? page_cache_tree_insert+0xe0/0xe0
May 20 20:38:00 proxmox50 kernel: [162874.186099] __filemap_fdatawait_range+0xfa/0x170
May 20 20:38:00 proxmox50 kernel: [162874.186102] filemap_write_and_wait+0x4d/0x90
May 20 20:38:00 proxmox50 kernel: [162874.186122] nfs_wb_all+0x20/0x110 [nfs]
May 20 20:38:00 proxmox50 kernel: [162874.186134] nfs_setattr+0x204/0x220 [nfs]
May 20 20:38:00 proxmox50 kernel: [162874.186140] notify_change+0x2f4/0x440
May 20 20:38:00 proxmox50 kernel: [162874.186145] utimes_common+0xf9/0x1b0
May 20 20:38:00 proxmox50 kernel: [162874.186150] do_utimes+0xde/0x160
May 20 20:38:00 proxmox50 kernel: [162874.186153] SyS_utimensat+0x87/0xd0
May 20 20:38:00 proxmox50 kernel: [162874.186156] do_syscall_64+0x73/0x130
May 20 20:38:00 proxmox50 kernel: [162874.186158] ? do_syscall_64+0x73/0x130
May 20 20:38:00 proxmox50 kernel: [162874.186161] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
May 20 20:38:00 proxmox50 kernel: [162874.186163] RIP: 0033:0x7fdb47d4cc96
May 20 20:38:00 proxmox50 kernel: [162874.186164] RSP: 002b:00007ffee84b7468 EFLAGS: 00000246 ORIG_RAX: 0000000000000118
May 20 20:38:00 proxmox50 kernel: [162874.186166] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007fdb47d4cc96
May 20 20:38:00 proxmox50 kernel: [162874.186167] RDX: 00007ffee84b7480 RSI: 0000000000000000 RDI: 0000000000000004
May 20 20:38:00 proxmox50 kernel: [162874.186168] RBP: 0000563a983c8e30 R08: 000000003614525e R09: 0000000000000000
May 20 20:38:00 proxmox50 kernel: [162874.186169] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000004
May 20 20:38:00 proxmox50 kernel: [162874.186170] R13: 0000000000000000 R14: 00007ffee84b7aa0 R15: 0000000000000000
May 20 20:40:01 proxmox50 kernel: [162995.011852] mv D 0 135889 121004 0x00000000
May 20 20:40:01 proxmox50 kernel: [162995.011856] Call Trace:
May 20 20:40:01 proxmox50 kernel: [162995.011866] __schedule+0x3e0/0x870
May 20 20:40:01 proxmox50 kernel: [162995.011869] schedule+0x36/0x80
May 20 20:40:01 proxmox50 kernel: [162995.011871] io_schedule+0x16/0x40
May 20 20:40:01 proxmox50 kernel: [162995.011876] wait_on_page_bit_common+0xf3/0x190
May 20 20:40:01 proxmox50 kernel: [162995.011879] ? page_cache_tree_insert+0xe0/0xe0
May 20 20:40:01 proxmox50 kernel: [162995.011881] __filemap_fdatawait_range+0xfa/0x170
May 20 20:40:01 proxmox50 kernel: [162995.011885] filemap_write_and_wait+0x4d/0x90
May 20 20:40:01 proxmox50 kernel: [162995.011908] nfs_wb_all+0x20/0x110 [nfs]
May 20 20:40:01 proxmox50 kernel: [162995.011918] nfs_setattr+0x204/0x220 [nfs]
May 20 20:40:01 proxmox50 kernel: [162995.011922] notify_change+0x2f4/0x440
May 20 20:40:01 proxmox50 kernel: [162995.011926] utimes_common+0xf9/0x1b0
May 20 20:40:01 proxmox50 kernel: [162995.011928] do_utimes+0xde/0x160
May 20 20:40:01 proxmox50 kernel: [162995.011931] SyS_utimensat+0x87/0xd0
May 20 20:40:01 proxmox50 kernel: [162995.011935] do_syscall_64+0x73/0x130
May 20 20:40:01 proxmox50 kernel: [162995.011938] ? do_syscall_64+0x73/0x130
May 20 20:40:01 proxmox50 kernel: [162995.011942] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
May 20 20:40:01 proxmox50 kernel: [162995.011946] RIP: 0033:0x7fdb47d4cc96
May 20 20:40:01 proxmox50 kernel: [162995.011948] RSP: 002b:00007ffee84b7468 EFLAGS: 00000246 ORIG_RAX: 0000000000000118
May 20 20:40:01 proxmox50 kernel: [162995.011951] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007fdb47d4cc96
May 20 20:40:01 proxmox50 kernel: [162995.011952] RDX: 00007ffee84b7480 RSI: 0000000000000000 RDI: 0000000000000004
May 20 20:40:01 proxmox50 kernel: [162995.011953] RBP: 0000563a983c8e30 R08: 000000003614525e R09: 0000000000000000
May 20 20:40:01 proxmox50 kernel: [162995.011954] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000004
May 20 20:40:01 proxmox50 kernel: [162995.011955] R13: 0000000000000000 R14: 00007ffee84b7aa0 R15: 0000000000000000

 

[Paolo_Bernardoni]

Node configuration (for all nodes):
pveversion -v
proxmox-ve: 5.4-2 (running kernel: 4.15.18-28-pve)
pve-manager: 5.4-15 (running version: 5.4-15/d0ec33c6)
pve-kernel-4.15: 5.4-17
pve-kernel-4.15.18-28-pve: 4.15.18-56
pve-kernel-4.15.18-27-pve: 4.15.18-55
pve-kernel-4.15.18-25-pve: 4.15.18-53
pve-kernel-4.15.18-24-pve: 4.15.18-52
pve-kernel-4.15.18-21-pve: 4.15.18-48
pve-kernel-4.15.18-20-pve: 4.15.18-46
pve-kernel-4.15.18-12-pve: 4.15.18-36
pve-kernel-4.15.18-9-pve: 4.15.18-30
pve-kernel-4.15.18-5-pve: 4.15.18-24
pve-kernel-4.15.18-2-pve: 4.15.18-21
pve-kernel-4.15.17-3-pve: 4.15.17-14
pve-kernel-4.15.17-2-pve: 4.15.17-10
pve-kernel-4.15.17-1-pve: 4.15.17-9
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-12
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-56
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-14
libpve-storage-perl: 5.0-44
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-7
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-3
proxmox-widget-toolkit: 1.0-28
pve-cluster: 5.0-38
pve-container: 2.0-42
pve-docs: 5.4-2
pve-edk2-firmware: 1.20190312-1
pve-firewall: 3.0-22
pve-firmware: 2.0-7
pve-ha-manager: 2.0-9
pve-i18n: 1.1-4
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 3.0.1-4
pve-xtermjs: 3.12.0-1
qemu-server: 5.0-56
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.13-pve1~bpo2

 

[Paolo_Bernardoni]

The last proxmox 5.4 kernel - Linux proxmox50 4.15.18-28-pve #1 SMP PVE 4.15.18-56 (Wed, 08 Apr 2020 20:06:11 +0200) x86_64 GNU/Linux -
containti40 driver version :
root@proxmox50:~# modinfo i40e
filename: /lib/modules/4.15.18-28-pve/kernel/drivers/net/ethernet/intel/i40e/i40e.ko
version: 2.1.14-k
license: GPL
description: Intel(R) Ethernet Connection XL710 Network Driver
author: Intel Corporation, <e1000-devel@lists.sourceforge.net>
srcversion: 0CFC6F59054387D53FF23FF

In Centos 7 kernel - Linux storage1 3.10.0-1127.el7.x86_64 #1 SMP Tue Mar 31 23:36:51 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
i40e driver veris is more recent

[root@storage1 i40e]# modinfo i40e
filename: /lib/modules/3.10.0-1127.el7.x86_64/kernel/drivers/net/ethernet/intel/i40e/i40e.ko.xz
version: 2.8.20-k
license: GPL v2
description: Intel(R) Ethernet Connection XL710 Network Driver
author: Intel Corporation, <e1000-devel@lists.sourceforge.net>
retpoline: Y
rhelversion: 7.8
srcversion: DB6044FF171EDB2184638E3

Maybe a i40e driver regression in Proxmox kernel ?