[SOLVED] Proxmox w/ Ceph Slow restore Troubleshooting

[cyrus104]

I have a 3 node Proxmox cluster each with a Xeon 8c/16t, 64GB ram, 512GBnvme (OS), 1TB Samsung 970 Pro NVME (OSD), 6.4TB Intel P4600 NVME (OSD), 10Gb to management / nfs nas network, 10Gb to Ceph VLAN. This is only 6 OSDs, I know this is a very light install but it's all I can fit into these cases.

I am restoring a 200GB VM from the NFS share on my Synology NAS. I was able to upload it awhile ago before a rebuild fairly quickly but after this rebuild the restore is going very slow. All of the drives in this are very fast, the Samsung drives obviously aren't enterprise but are still top of the line consumer. The Intels are fairly good drives with.

When I restore to a local lvm, the speed from the NAS is in the 300-400MB/s.

As you can see from the picture, there is a slow down somewhere but I'm not sure where or how to troubleshoot.

 

[cyrus104]

I was able to fix this using iperf to troubleshoot.

Turns out on of my SFP+ DAC cables wasn't allowing one device to transmit properly. I tried to unplug and replug it in but that didn't help. When I swapped it with an identical cable it worked and I was able to transfer at normal speeds.

After running the server on each node, I found that node 3 couldn't transmit properly to nodes 1 or 2.

 

[Stoiko Ivanov]

Thanks for sharing your solution!
(you can also use the 'SOLVED' prefix from the drop-down to indicate that the issue is gone - that way it's visually better shown that the issue is resolved).