Slow fsync on Proxmox over NFS

[slowfsync]

I have three machines that all have the same hardware configuration. They all serve as nodes in a Proxmox cluster.

Two of them have been running VMs over NFS for months with no issues. pveperf on these, targeting the NFS share, shows ~2000 fsync/sec. Trying to boot off of the third node however, also under the same NFS share, basically doesn't work. Takes forever. Seems to take up 100% of a single CPU (vcore) during the boot process and takes about 13 hours to finally load the OS image from NFS into RAM and boot. Running pveperf on the NFS share on _this_ machine produces these results:

CPU BOGOMIPS:      166420.16
REGEX/SECOND:      1696783
HD SIZE:           20446.67 GB (storage.family.site:/mnt/zcore/vm_storage)
FSYNCS/SECOND:     166.79
DNS EXT:           45.01 ms
DNS INT:           27.57 ms (family.site)

This is on a fresh installation of Proxmox 6.0. On a previous fresh installation of Proxmox 5.4, the FSYNC/SECOND was usually <1. Yes. Less than 1.

I'm not sure where to go from here. Being that the hardware is the same as the other two machines, and both of those other machines also perform fine on fresh installs, I expect this third machine to reach the >2000 FSYNC/SECOND that the other two consistently receive.

 

[Alwin]

It may be an hardware issue? Is there something in the journal/syslog? Can you test with iperf, if maybe the NIC has some issues?

 

[slowfsync]

It may be an hardware issue? Is there something in the journal/syslog? Can you test with iperf, if maybe the NIC has some issues?

I too was leaning on hardware issue. Wasn't sure where to go -- had already swapped out the NIC as well as the DAC. Still had the issue. "Surely both these DACs cant be bad" I thought and never even tried to run iperf till you mentioned it. Well both DACs were bad. Ordering some new ones and I expect my issue will be solved with them. I'm an idiot. Thank you Alwin.

 

[Alwin]

I hope that is it, but can you try a test with one of the cables on a functioning node? Just to rule out that there isn't a different issue.

 

[slowfsync]

CPU BOGOMIPS:      166433.60
REGEX/SECOND:      1865437
HD SIZE:           20446.67 GB (storage.family.site:/mnt/zcore/vm_storage)
FSYNCS/SECOND:     1649.24
DNS EXT:           27.47 ms
DNS INT:           20.10 ms (family.site)

Swapped cable and got these results, which are about what I expect.