Hello,
I have a proxmox (version 7.4) cluster of 3 hosts and there is a GlusterFS configured between these servers. Hosts are deployed on Leaseweb and there is 10GB/s network beteen hosts. People started to complain about slow work with VMs. I monitored metrics CPU/RAM/Disk usage but didn't noticed any bottlenecks despite sometimes from VM itself monitoring disk latency jumping to 50-90ms but just for a moment. But the slowness is there and I can also feel it, just can't prove it.
Started searching for an issue.
iperf server1 -> server2
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 642 MBytes 5.39 Gbits/sec 1093 1.42 MBytes
[ 5] 1.00-2.00 sec 1.05 GBytes 9.05 Gbits/sec 1335 988 KBytes
[ 5] 2.00-3.00 sec 1.04 GBytes 8.91 Gbits/sec 980 996 KBytes
[ 5] 3.00-4.00 sec 704 MBytes 5.91 Gbits/sec 375 1.46 MBytes
[ 5] 4.00-5.00 sec 751 MBytes 6.30 Gbits/sec 1068 1.06 MBytes
[ 5] 5.00-6.00 sec 1.05 GBytes 9.03 Gbits/sec 984 1.10 MBytes
[ 5] 6.00-7.00 sec 1.06 GBytes 9.14 Gbits/sec 867 1012 KBytes
[ 5] 7.00-8.00 sec 1.04 GBytes 8.90 Gbits/sec 494 1005 KBytes
[ 5] 8.00-9.00 sec 1.06 GBytes 9.13 Gbits/sec 66 1.37 MBytes
[ 5] 9.00-10.00 sec 701 MBytes 5.88 Gbits/sec 1039 235 KBytes
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 9.04 GBytes 7.76 Gbits/sec 8301 sender
[ 5] 0.00-10.05 sec 9.03 GBytes 7.72 Gbits/sec receiver
iperf server1 -> server3
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.02 GBytes 8.78 Gbits/sec 1233 1.15 MBytes
[ 5] 1.00-2.00 sec 372 MBytes 3.12 Gbits/sec 1010 964 KBytes
[ 5] 2.00-3.00 sec 841 MBytes 7.06 Gbits/sec 664 1.42 MBytes
[ 5] 3.00-4.00 sec 830 MBytes 6.96 Gbits/sec 1038 1.12 MBytes
[ 5] 4.00-5.00 sec 659 MBytes 5.53 Gbits/sec 928 1.18 MBytes
[ 5] 5.00-6.00 sec 1.04 GBytes 8.91 Gbits/sec 610 1.10 MBytes
[ 5] 6.00-7.00 sec 1.05 GBytes 9.02 Gbits/sec 353 1.39 MBytes
[ 5] 7.00-8.00 sec 1.01 GBytes 8.66 Gbits/sec 693 1.09 MBytes
[ 5] 8.00-9.00 sec 609 MBytes 5.11 Gbits/sec 791 990 KBytes
[ 5] 9.00-10.00 sec 798 MBytes 6.69 Gbits/sec 881 1.39 MBytes
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 8.13 GBytes 6.98 Gbits/sec 8201 sender
[ 5] 0.00-10.04 sec 8.13 GBytes 6.95 Gbits/sec receiver
iperf server2 -> server3
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 862 MBytes 7.23 Gbits/sec 1172 1.10 MBytes
[ 5] 1.00-2.00 sec 611 MBytes 5.13 Gbits/sec 1134 488 KBytes
[ 5] 2.00-3.00 sec 219 MBytes 1.83 Gbits/sec 1615 1.43 MBytes
[ 5] 3.00-4.00 sec 558 MBytes 4.68 Gbits/sec 1395 2.28 MBytes
[ 5] 4.00-5.00 sec 581 MBytes 4.88 Gbits/sec 1701 1020 KBytes
[ 5] 5.00-6.00 sec 994 MBytes 8.34 Gbits/sec 761 872 KBytes
[ 5] 6.00-7.00 sec 535 MBytes 4.49 Gbits/sec 936 872 KBytes
[ 5] 7.00-8.00 sec 746 MBytes 6.26 Gbits/sec 692 963 KBytes
[ 5] 8.00-9.00 sec 1002 MBytes 8.41 Gbits/sec 339 769 KBytes
[ 5] 9.00-10.00 sec 925 MBytes 7.76 Gbits/sec 436 311 KBytes
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 6.87 GBytes 5.90 Gbits/sec 10181 sender
[ 5] 0.00-10.04 sec 6.87 GBytes 5.87 Gbits/sec receiver
Network looks okay, not perfect 10gb/s but I think it's because of MTU 1500
Next running dd inside volume on which brick resides:
server1
dd if=/dev/mapper/pve-root of=test1.img bs=5G count=3 oflag=dsync
dd: warning: partial read (2147479552 bytes); suggest iflag=fullblock
0+3 records in
0+3 records out
6442438656 bytes (6.4 GB, 6.0 GiB) copied, 15.0352 s, 428 MB/s
server2
dd if=/dev/mapper/pve-root of=test1.img bs=5G count=3 oflag=dsync
dd: warning: partial read (2147479552 bytes); suggest iflag=fullblock
0+3 records in
0+3 records out
6442438656 bytes (6.4 GB, 6.0 GiB) copied, 16.0089 s, 402 MB/s
server3
dd if=/dev/mapper/pve-root of=test1.img bs=5G count=3 oflag=dsync
dd: warning: partial read (2147479552 bytes); suggest iflag=fullblock
0+3 records in
0+3 records out
6442438656 bytes (6.4 GB, 6.0 GiB) copied, 20.3058 s, 317 MB/s
dd also looks good. Next I'm trying to write file to GlusterFS mount
dd if=/dev/mapper/pve-root of=test1.img bs=5G count=3 oflag=dsync
dd: warning: partial read (2147479552 bytes); suggest iflag=fullblock
0+3 records in
0+3 records out
6442438656 bytes (6.4 GB, 6.0 GiB) copied, 174.394 s, 36.9 MB/s
Write speed drops ~10 times is it normal behavior? It doesn't look okay to me. How can I identify the problem?
gluster volume info
Volume Name: o.......e1
Type: Replicate
Volume ID: e0............a79
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: p................./brick
Brick2: p................./brick
Brick3: p................./brick
Options Reconfigured:
nfs.disable: on
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
performance.client-io-threads: off
transport.address-family: inet
storage.fips-mode-rchecksum: on
cluster.granular-entry-heal: on
This issue was noticed on GlusterFS version 10.2. Tried upgrading to GlusterFS 11.1 but the problem still persists.
Any comment or thoughts?
I have a proxmox (version 7.4) cluster of 3 hosts and there is a GlusterFS configured between these servers. Hosts are deployed on Leaseweb and there is 10GB/s network beteen hosts. People started to complain about slow work with VMs. I monitored metrics CPU/RAM/Disk usage but didn't noticed any bottlenecks despite sometimes from VM itself monitoring disk latency jumping to 50-90ms but just for a moment. But the slowness is there and I can also feel it, just can't prove it.
Started searching for an issue.
iperf server1 -> server2
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 642 MBytes 5.39 Gbits/sec 1093 1.42 MBytes
[ 5] 1.00-2.00 sec 1.05 GBytes 9.05 Gbits/sec 1335 988 KBytes
[ 5] 2.00-3.00 sec 1.04 GBytes 8.91 Gbits/sec 980 996 KBytes
[ 5] 3.00-4.00 sec 704 MBytes 5.91 Gbits/sec 375 1.46 MBytes
[ 5] 4.00-5.00 sec 751 MBytes 6.30 Gbits/sec 1068 1.06 MBytes
[ 5] 5.00-6.00 sec 1.05 GBytes 9.03 Gbits/sec 984 1.10 MBytes
[ 5] 6.00-7.00 sec 1.06 GBytes 9.14 Gbits/sec 867 1012 KBytes
[ 5] 7.00-8.00 sec 1.04 GBytes 8.90 Gbits/sec 494 1005 KBytes
[ 5] 8.00-9.00 sec 1.06 GBytes 9.13 Gbits/sec 66 1.37 MBytes
[ 5] 9.00-10.00 sec 701 MBytes 5.88 Gbits/sec 1039 235 KBytes
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 9.04 GBytes 7.76 Gbits/sec 8301 sender
[ 5] 0.00-10.05 sec 9.03 GBytes 7.72 Gbits/sec receiver
iperf server1 -> server3
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.02 GBytes 8.78 Gbits/sec 1233 1.15 MBytes
[ 5] 1.00-2.00 sec 372 MBytes 3.12 Gbits/sec 1010 964 KBytes
[ 5] 2.00-3.00 sec 841 MBytes 7.06 Gbits/sec 664 1.42 MBytes
[ 5] 3.00-4.00 sec 830 MBytes 6.96 Gbits/sec 1038 1.12 MBytes
[ 5] 4.00-5.00 sec 659 MBytes 5.53 Gbits/sec 928 1.18 MBytes
[ 5] 5.00-6.00 sec 1.04 GBytes 8.91 Gbits/sec 610 1.10 MBytes
[ 5] 6.00-7.00 sec 1.05 GBytes 9.02 Gbits/sec 353 1.39 MBytes
[ 5] 7.00-8.00 sec 1.01 GBytes 8.66 Gbits/sec 693 1.09 MBytes
[ 5] 8.00-9.00 sec 609 MBytes 5.11 Gbits/sec 791 990 KBytes
[ 5] 9.00-10.00 sec 798 MBytes 6.69 Gbits/sec 881 1.39 MBytes
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 8.13 GBytes 6.98 Gbits/sec 8201 sender
[ 5] 0.00-10.04 sec 8.13 GBytes 6.95 Gbits/sec receiver
iperf server2 -> server3
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 862 MBytes 7.23 Gbits/sec 1172 1.10 MBytes
[ 5] 1.00-2.00 sec 611 MBytes 5.13 Gbits/sec 1134 488 KBytes
[ 5] 2.00-3.00 sec 219 MBytes 1.83 Gbits/sec 1615 1.43 MBytes
[ 5] 3.00-4.00 sec 558 MBytes 4.68 Gbits/sec 1395 2.28 MBytes
[ 5] 4.00-5.00 sec 581 MBytes 4.88 Gbits/sec 1701 1020 KBytes
[ 5] 5.00-6.00 sec 994 MBytes 8.34 Gbits/sec 761 872 KBytes
[ 5] 6.00-7.00 sec 535 MBytes 4.49 Gbits/sec 936 872 KBytes
[ 5] 7.00-8.00 sec 746 MBytes 6.26 Gbits/sec 692 963 KBytes
[ 5] 8.00-9.00 sec 1002 MBytes 8.41 Gbits/sec 339 769 KBytes
[ 5] 9.00-10.00 sec 925 MBytes 7.76 Gbits/sec 436 311 KBytes
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 6.87 GBytes 5.90 Gbits/sec 10181 sender
[ 5] 0.00-10.04 sec 6.87 GBytes 5.87 Gbits/sec receiver
Network looks okay, not perfect 10gb/s but I think it's because of MTU 1500
Next running dd inside volume on which brick resides:
server1
dd if=/dev/mapper/pve-root of=test1.img bs=5G count=3 oflag=dsync
dd: warning: partial read (2147479552 bytes); suggest iflag=fullblock
0+3 records in
0+3 records out
6442438656 bytes (6.4 GB, 6.0 GiB) copied, 15.0352 s, 428 MB/s
server2
dd if=/dev/mapper/pve-root of=test1.img bs=5G count=3 oflag=dsync
dd: warning: partial read (2147479552 bytes); suggest iflag=fullblock
0+3 records in
0+3 records out
6442438656 bytes (6.4 GB, 6.0 GiB) copied, 16.0089 s, 402 MB/s
server3
dd if=/dev/mapper/pve-root of=test1.img bs=5G count=3 oflag=dsync
dd: warning: partial read (2147479552 bytes); suggest iflag=fullblock
0+3 records in
0+3 records out
6442438656 bytes (6.4 GB, 6.0 GiB) copied, 20.3058 s, 317 MB/s
dd also looks good. Next I'm trying to write file to GlusterFS mount
dd if=/dev/mapper/pve-root of=test1.img bs=5G count=3 oflag=dsync
dd: warning: partial read (2147479552 bytes); suggest iflag=fullblock
0+3 records in
0+3 records out
6442438656 bytes (6.4 GB, 6.0 GiB) copied, 174.394 s, 36.9 MB/s
Write speed drops ~10 times is it normal behavior? It doesn't look okay to me. How can I identify the problem?
gluster volume info
Volume Name: o.......e1
Type: Replicate
Volume ID: e0............a79
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: p................./brick
Brick2: p................./brick
Brick3: p................./brick
Options Reconfigured:
nfs.disable: on
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
performance.client-io-threads: off
transport.address-family: inet
storage.fips-mode-rchecksum: on
cluster.granular-entry-heal: on
This issue was noticed on GlusterFS version 10.2. Tried upgrading to GlusterFS 11.1 but the problem still persists.
Any comment or thoughts?