LXC on NFS (sync) does not work

[Republicus]

I am mounting several NFS shares. For my LXC and QEMU images I wish to mount the NFS share as synchronous.

QEMU guests are working well on the NFS sync share.

For LXC, however, I noticed my sync writes dropped down below 10 MB/s and would hang for several minutes after writing test files.
The same NFS sync share on the host directly and in QEMU guests are reaching up to 1 GB/s writes.

I saw on the host syslog at the same time the LXC guest was hung after making writes:

Oct 06 22:48:48 node04 kernel: INFO: task sync:6262 blocked for more than 120 seconds.
Oct 06 22:48:48 node04 kernel:       Tainted: P           O      5.0.21-2-pve #1
Oct 06 22:48:48 node04 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 06 22:48:48 node04 kernel: sync            D    0  6262   6177 0x00000100
Oct 06 22:48:48 node04 kernel: Call Trace:
Oct 06 22:48:48 node04 kernel:  __schedule+0x2d4/0x870
Oct 06 22:48:48 node04 kernel:  schedule+0x2c/0x70
Oct 06 22:48:48 node04 kernel:  wb_wait_for_completion+0x64/0x90
Oct 06 22:48:48 node04 kernel:  ? wait_woken+0x80/0x80
Oct 06 22:48:48 node04 kernel:  sync_inodes_sb+0xd2/0x290
Oct 06 22:48:48 node04 kernel:  sync_inodes_one_sb+0x15/0x20
Oct 06 22:48:48 node04 kernel:  iterate_supers+0xaa/0x100
Oct 06 22:48:48 node04 kernel:  ? default_file_splice_write+0x30/0x30
Oct 06 22:48:48 node04 kernel:  ksys_sync+0x42/0xb0
Oct 06 22:48:48 node04 kernel:  __ia32_sys_sync+0xe/0x20
Oct 06 22:48:48 node04 kernel:  do_syscall_64+0x5a/0x110
Oct 06 22:48:48 node04 kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xa9
Oct 06 22:48:48 node04 kernel: RIP: 0033:0x7f4c21b1cb37
Oct 06 22:48:48 node04 kernel: Code: Bad RIP value.
Oct 06 22:48:48 node04 kernel: RSP: 002b:00007ffe437674a8 EFLAGS: 00000202 ORIG_RAX: 00000000000000a2
Oct 06 22:48:48 node04 kernel: RAX: ffffffffffffffda RBX: 00007ffe437675b8 RCX: 00007f4c21b1cb37
Oct 06 22:48:48 node04 kernel: RDX: 00007f4c21dee280 RSI: 0000000000000001 RDI: 00007f4c21baff77
Oct 06 22:48:48 node04 kernel: RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000
Oct 06 22:48:48 node04 kernel: R10: 00007ffe43767070 R11: 0000000000000202 R12: 000000000040156d
Oct 06 22:48:48 node04 kernel: R13: 00007ffe437675b0 R14: 0000000000000000 R15: 0000000000000000

I remounted the share as NFS Async and the LXC guests write comparable to the sync writes (~700 MB/s).

This error and performance issue is only happening on SYNC shares and in RAW format.
If setup as a bind mount at the host /mnt/pve/ mountpoint for the NFS sync share -- the LXC guest writes data to the bind mount as expected.

proxmox-ve: 6.0-2 (running kernel: 5.0.21-2-pve)
pve-manager: 6.0-7 (running version: 6.0-7/28984024)
pve-kernel-5.0: 6.0-8
pve-kernel-helper: 6.0-8
pve-kernel-5.0.21-2-pve: 5.0.21-6
pve-kernel-5.0.21-1-pve: 5.0.21-2
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve2 criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.12-pve1
libpve-access-control: 6.0-2
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-5
libpve-guest-common-perl: 3.0-1
libpve-http-server-perl: 3.0-2
libpve-storage-perl: 6.0-9
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.1.0-65
lxcfs: 3.0.3-pve60 novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-7
pve-cluster: 6.0-7
pve-container: 3.0-7
pve-docs: 6.0-4
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-7
pve-firmware: 3.0-2
pve-ha-manager: 3.0-2
pve-i18n: 2.0-3
pve-qemu-kvm: 4.0.0-5
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-7
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.1-pve2

 

[Republicus]

Adding that this is occurring on both lxc privileged and unprivileged

 

[Republicus]

Ive tested an LXC container whose mount point is stored on the same NFS synchronous share on a PVE 5.4-1 node -- and, it is not suffering from the synchronous write errors that I am seeing on PVE 6.0-2

Writes are hitting my SLOG/ZIL and at expected speeds.

I've also created a bug report #2409 for this issue.