Poor network IOPS - Troubleshooting

[shadowq]

I have two nodes set up, geographically 3-4,000 KMs away (200ms).
Node 1 (3 x 1.92GB SSD) scores great IOPS testing using FIO, around 470,000 for sequential reads, and around 2000MB/s. For random read/write, around 2500 IOPS, and 10MB/s.
Node 2 (2 x 480GB SSD) scores great IOPS testing using FIO, around 350,000 for sequential reads, and around 1500MB/s. For random read/write, around 6250 IOPS, and 25MB/s.
Both running on ZFS, with only 2 containers running on them, not super intensive loads. Both in an OVH DC (APAC and N.A)

I've been experimenting with how I'm going to increase storage space for non-critical data on Node 2, but would still like to achieve 50-100 IOPS ideally. However, even when experimenting with mounting an external drive via NFS, or SSHFS, I'm getting abysmal IOPS (1-5). Excuse my ignorance, but would IOPS be limited by actual number of roundtrips it can take in a second? Eg- a test with mounting Node 2 dir on Node 1, achieves only a IOPS of < 5, is that because each round trip is 200ms or so? Or is there something else at play here? (Regular bandwidth between the nodes are 100MB/s+)

TIA.

 

[shadowq]

Nevermind. (Was going to remove this to save embarrassment, but I guess it may help someone else down the track).
IOPS was indeed extremely poor due to network latency. I spun up a public cloud instance in the same DC as one of the nodes and read IOPS are approx 2000, with read/write being 150-200.