[SOLVED] How to get 10G speed? 2 pve to same 1G router, each pve has 2 10g NIC

[davidng]

Hi network masters,
I am new to this, and I have the following simple setup.

1 no vlan
2 all proxmox and router and vm are under 192.168.1.x
3 one wireless router with 3 NIC port, each port speed is 1G/s, this router connected to a 500M/s ISP
4 one "media proxmox" with 2 10G NIC port, one of the port is connected to the above router by ethernet cable, the other port is NOT Connected to anything.
5 one "backup proxmox" with 2 10G NIC port, one of the port is connected to the above router by ethernet cable, the other port is NOT Connected to anything.


now I am doing some huge files transfer between the pve hosts (NOT in VM) from "media proxmox" to "backup proxmox" using zfs send and ssh and zfs recv.
the file transfer speed stay at 1G/s, as limited by the router.

Question:
is there a way to connect the free 10G/s NIC port in each proxmox server,
and have 10G/s transfer speed between the 2 proxmox server under the same router network?

I try connecting two free 10G NIC with a ethernet cable, both NIC hardware light did not blink.
I guess this is because I did not setup the port in /etc/network/interfaces
but the question is, how to set it up?

 

[justinclift]

Does this help? https://forum.proxmox.com/threads/zfs-replication-via-second-lan.147665/post-667288

 

[davidng]

thank you, a group of people help me on this, and this is the solution.

no need cross cable for connect NIC2 and NIC3, as nowaday many NIC support MDI-X.

in pve1
nano /etc/network/interfaces
allow-hotplug eno2
iface eno2 inet static
address 172.16.5.1
netmask 255.255.255.0
#gateway 172.16.5.2
up ip route add default via 192.168.1.1

then I do
ifdown eno2
ifup eno2

ethtool eno2
Supported ports: [ TP MII ]
Port: Twisted Pair
MDI-X: Unknown
Link detected: yes

ethtool -s eno2 mdix auto
ethtool eno2
MDI-X: Unknown (auto)

---

in pve2
nano /etc/network/interfaces
allow-hotplug eno3
iface eno3 inet static
address 172.16.5.2
netmask 255.255.255.0
#gateway 172.16.5.1
up ip route add default via 192.168.1.1

then I do
ifdown eno3
ifup eno3

ethtool eno3
Supported ports: [ TP ]
Port: Twisted Pair
MDI-X: off (auto)
Link detected: yes

---

then on pve2
ping 172.16.5.1
it will work.