Slow fsync on Proxmox over NFS

slowfsync

New Member
Oct 22, 2019
3
1
3
34
I have three machines that all have the same hardware configuration. They all serve as nodes in a Proxmox cluster.

Two of them have been running VMs over NFS for months with no issues. pveperf on these, targeting the NFS share, shows ~2000 fsync/sec. Trying to boot off of the third node however, also under the same NFS share, basically doesn't work. Takes forever. Seems to take up 100% of a single CPU (vcore) during the boot process and takes about 13 hours to finally load the OS image from NFS into RAM and boot. Running pveperf on the NFS share on _this_ machine produces these results:

Code:
CPU BOGOMIPS:      166420.16
REGEX/SECOND:      1696783
HD SIZE:           20446.67 GB (storage.family.site:/mnt/zcore/vm_storage)
FSYNCS/SECOND:     166.79
DNS EXT:           45.01 ms
DNS INT:           27.57 ms (family.site)

This is on a fresh installation of Proxmox 6.0. On a previous fresh installation of Proxmox 5.4, the FSYNC/SECOND was usually <1. Yes. Less than 1.

I'm not sure where to go from here. Being that the hardware is the same as the other two machines, and both of those other machines also perform fine on fresh installs, I expect this third machine to reach the >2000 FSYNC/SECOND that the other two consistently receive.
 
It may be an hardware issue? Is there something in the journal/syslog? Can you test with iperf, if maybe the NIC has some issues?
 
It may be an hardware issue? Is there something in the journal/syslog? Can you test with iperf, if maybe the NIC has some issues?

I too was leaning on hardware issue. Wasn't sure where to go -- had already swapped out the NIC as well as the DAC. Still had the issue. "Surely both these DACs cant be bad" I thought and never even tried to run iperf till you mentioned it. Well both DACs were bad. Ordering some new ones and I expect my issue will be solved with them. I'm an idiot. Thank you Alwin.
 
I hope that is it, but can you try a test with one of the cables on a functioning node? Just to rule out that there isn't a different issue.
 
Code:
CPU BOGOMIPS:      166433.60
REGEX/SECOND:      1865437
HD SIZE:           20446.67 GB (storage.family.site:/mnt/zcore/vm_storage)
FSYNCS/SECOND:     1649.24
DNS EXT:           27.47 ms
DNS INT:           20.10 ms (family.site)

Swapped cable and got these results, which are about what I expect.
 
  • Like
Reactions: Alwin

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!