NFS Questions with Performance

[epytir]

Hello Proxmox Users,

I migrated from Ceph to NFS with 10Gbit Interface(tested with aroud 9.3 Gbit real speed) and a Dell/EMC Unity Storage (NFS Speedtest with iozone around 430MB/s write speed).
When I now move VMs to the Storage I get around 130 MB/s (in the VM I get nearly the same speed).

I have now 2 problems:
1) Why I only get 130 MB/s with Proxmox and inside the VM but in CLI a copy is faster ?
2) When I move a VM to the NFS all VMs on the NFS Storage get IO Problems and sometimes freeze for some seconds. Is this normal or can I do something against that?

Are there any best practise options to get better results ?
I used NFS3 (also tested NFS4) both has the same speed in my tests is NFS4 better ?

I used the forum search but did not find exact answers.
I also set some sysctl configs on all hosts:
cat /etc/sysctl.d/nfs.conf

cat /etc/sysctl.d/nfs.conf

net.ipv4.tcp_max_tw_buckets = 180000
net.ipv4.tcp_mem = 3622080 4829440 7244160
net.ipv4.tcp_wmem = 40960 163840 41943040
net.ipv4.tcp_rmem = 40960 873800 41943040
net.ipv4.udp_mem = 363168 484224 726336
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
sunrpc.tcp_slot_table_entries = 128

Any help would be appreciated.

Greetings Epytir

 

[epytir]

Hello Proxmox Users,

I migrated from Ceph to NFS with 10Gbit Interface(tested with aroud 9.3 Gbit real speed) and a Dell/EMC Unity Storage (NFS Speedtest with iozone around 430MB/s write speed).
When I now move VMs to the Storage I get around 130 MB/s (in the VM I get nearly the same speed).

I have now 2 problems:
1) Why I only get 130 MB/s with Proxmox and inside the VM but in CLI a copy is faster ?
2) When I move a VM to the NFS all VMs on the NFS Storage get IO Problems and sometimes freeze for some seconds. Is this normal or can I do something against that?

Are there any best practise options to get better results ?
I used NFS3 (also tested NFS4) both has the same speed in my tests is NFS4 better ?

I used the forum search but did not find exact answers.
I also set some sysctl configs on all hosts:
cat /etc/sysctl.d/nfs.conf

cat /etc/sysctl.d/nfs.conf

net.ipv4.tcp_max_tw_buckets = 180000
net.ipv4.tcp_mem = 3622080 4829440 7244160
net.ipv4.tcp_wmem = 40960 163840 41943040
net.ipv4.tcp_rmem = 40960 873800 41943040
net.ipv4.udp_mem = 363168 484224 726336
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
sunrpc.tcp_slot_table_entries = 128

Any help would be appreciated.

Greetings Epytir

ok I tested NFS4 again but this time with VM speed test (not CLI) and the perforamnce is the same but is seams that the other VMs does not freeze during big copies anymore.

 

[vix9]

Can you confirm that the swap to NFS4 fixed your freezing issue? We have issues whenever there is significant IO on a VM that is sharing NFS storage. It's most noticable with our SQL servers. On Proxmox 6 with the same exact setup, we had none of these issues.

 

[epytir]

Its not fixed I would say because when I copy big data all vms seems to be super slow or frozen. Thats a problem when every copy get the full performance and all other stuff dies. In the past proxmox was able to split the traffix so no process got everything I think.

 

[vix9]

My issue seems to span nodes. For Example:

I have Nodes 1,2,3,4,5 and NFS Storage A and B

Machine X is on Storage A, running on Node 1. Machine Y is on Storage B, Running on Node 1. Machine Z is on Storage A, Node 5.

If Machine X is doing anything with high IO (backup is a great example), Machine Z will lock up. Machine Y will be fine.

 

[epytir]

I tested some more things now and the host that produces the IO all VMs get freeze. FOr example I started a Restore from Backup on node1 than all VMs on node 1 got performance issues or small freezes. The vms on the other hosts are fine. Are there any seetings to prevent proxmox from giving all the performance to one vm or task ?