Hello all (and happy new year)
i'm asking expert after lots of time trying and searching.
My lab is
Fiber Freebox Delta bridge mode (Free / France) 10Gb/s Download / 700 MB Upload
Storage
Truenas Scale = 1x10GB SFP+
A cluster of 3 proxmox
Proxmox 1 = 2x10Gb SFP+
Proxmox 2 = 1x1Gb ethernet
Proxmox 3 = 1x2.5Gb Ethernet
On proxmox 1:
linux brige (vmbr0) link with nic0 LAN
linux brige (vmbr1) link with nic1 WAN
First VM (opnsense) with 2 NIC
NIC0 LAN (on vmbr0)
NIC1 WAN (on vmbr1)
Second VM (alpine linux) with one NIC
NIC0 LAN (on vmbr0)
On proxmox 2:
linux brige (vmbr0) link with nic0 LAN
First VM (alpine linux) with one NIC
NIC0 LAN (on vmbr0)
Truenas Scale
NIC0 = LAN
VM Opnsense = Gateway (firewall/NAT)
Everything work fine (even IPV6 with Freebox TV player IPV6) Except one "little" thing, NAT outside proxmox 1 (so, all my lan)
LAN
Trafic performance (done with iperf3)
proxmox 1 -> truenas = 9Gb/s
truenas -> proxmox 1 = 9Gb/s
proxmox 2 -> truenas = 2.5Gb/s
truenas -> proxmox 2 = 2.5Gb/s
vm(proxmox 1) -> truenas = 9Gb/s
truenas -> vm (proxmox 1) = 9Gb/s
vm(proxmox 2) -> truenas = 2.5Gb/s
truenas -> vm (proxmox 2) = 2.5Gb/s
INTERNET (NAT Thrue opnsense)
Trafic performance (done with speedtest)
Proxmox 1 -> Internet = 1.5 Gb/s : OK for me
Internet -> Proxmox 1 = 700MB/s : OK for me
VM Opensense -> Internet = 1.5 Gb/s : OK for me
Internet -> VM Opensense = 700MB/s : OK for me
vm(proxmox 1) -> Internet = 1.5 Gb/s : OK for me
Internet -> vm(proxmox 1) = 700MB/s : OK for me
Anything else not directly INSIDE proxmox 1 (where opnsense is hosted)
=============== BBR
Server: LaFibre.info BBR-IPv4 - Palaiseau (id: 45446)
Download: 1427.95 Mbps (data used: 957.4 MB)
Upload: 590.06 Mbps (data used: 1.1 GB)
Server: BOUYGUES TELECOM BBR - Meudon (id: 47318)
Download: 1762.61 Mbps (data used: 2.5 GB)
Upload: 614.28 Mbps (data used: 641.0 MB)
Server: LaFibre.info BBR - Massy (id: 2231)
Download: 1513.09 Mbps (data used: 1.4 GB)
Upload: 618.12 Mbps (data used: 586.8 MB)
=============== CUBIC
Server: LaFibre.info CUBIC - Orly (id: 45454)
Download: 34.60 Mbps (data used: 55.2 MB)
Upload: 575.48 Mbps (data used: 806.2 MB)
Server: BOUYGUES TELECOM CUBIC - Vélizy (id: 47424)
Download: 32.60 Mbps (data used: 49.4 MB)
Upload: 597.55 Mbps (data used: 856.8 MB)
Server: Hivane NetWork Cubic - Ivry-sur-Seine (id: 49781)
Download: 30.76 Mbps (data used: 46.6 MB)
Upload: 120.26 Mbps (data used: 190.6 MB)
So, any VM, computer or whatever (not directrly inside proxmox 1) downloading internet things thrue opnsense got worst download speed ever.
Speedtest is the best i can do. but when i try do download alpine iso (exemple)
im 700Mb/s from proxmox 1, (or any vm hosted by proxmox one)
im 1Mbs/s from anything not hosted inside proxmox one itself.
If someone have any idea ???
Thx a lot
i'm asking expert after lots of time trying and searching.
My lab is
Fiber Freebox Delta bridge mode (Free / France) 10Gb/s Download / 700 MB Upload
Storage
Truenas Scale = 1x10GB SFP+
A cluster of 3 proxmox
Proxmox 1 = 2x10Gb SFP+
Proxmox 2 = 1x1Gb ethernet
Proxmox 3 = 1x2.5Gb Ethernet
On proxmox 1:
linux brige (vmbr0) link with nic0 LAN
linux brige (vmbr1) link with nic1 WAN
First VM (opnsense) with 2 NIC
NIC0 LAN (on vmbr0)
NIC1 WAN (on vmbr1)
Second VM (alpine linux) with one NIC
NIC0 LAN (on vmbr0)
On proxmox 2:
linux brige (vmbr0) link with nic0 LAN
First VM (alpine linux) with one NIC
NIC0 LAN (on vmbr0)
Truenas Scale
NIC0 = LAN
VM Opnsense = Gateway (firewall/NAT)
Everything work fine (even IPV6 with Freebox TV player IPV6) Except one "little" thing, NAT outside proxmox 1 (so, all my lan)
LAN
Trafic performance (done with iperf3)
proxmox 1 -> truenas = 9Gb/s
truenas -> proxmox 1 = 9Gb/s
proxmox 2 -> truenas = 2.5Gb/s
truenas -> proxmox 2 = 2.5Gb/s
vm(proxmox 1) -> truenas = 9Gb/s
truenas -> vm (proxmox 1) = 9Gb/s
vm(proxmox 2) -> truenas = 2.5Gb/s
truenas -> vm (proxmox 2) = 2.5Gb/s
INTERNET (NAT Thrue opnsense)
Trafic performance (done with speedtest)
Proxmox 1 -> Internet = 1.5 Gb/s : OK for me
Internet -> Proxmox 1 = 700MB/s : OK for me
VM Opensense -> Internet = 1.5 Gb/s : OK for me
Internet -> VM Opensense = 700MB/s : OK for me
vm(proxmox 1) -> Internet = 1.5 Gb/s : OK for me
Internet -> vm(proxmox 1) = 700MB/s : OK for me
Anything else not directly INSIDE proxmox 1 (where opnsense is hosted)
=============== BBR
Server: LaFibre.info BBR-IPv4 - Palaiseau (id: 45446)
Download: 1427.95 Mbps (data used: 957.4 MB)
Upload: 590.06 Mbps (data used: 1.1 GB)
Server: BOUYGUES TELECOM BBR - Meudon (id: 47318)
Download: 1762.61 Mbps (data used: 2.5 GB)
Upload: 614.28 Mbps (data used: 641.0 MB)
Server: LaFibre.info BBR - Massy (id: 2231)
Download: 1513.09 Mbps (data used: 1.4 GB)
Upload: 618.12 Mbps (data used: 586.8 MB)
=============== CUBIC
Server: LaFibre.info CUBIC - Orly (id: 45454)
Download: 34.60 Mbps (data used: 55.2 MB)
Upload: 575.48 Mbps (data used: 806.2 MB)
Server: BOUYGUES TELECOM CUBIC - Vélizy (id: 47424)
Download: 32.60 Mbps (data used: 49.4 MB)
Upload: 597.55 Mbps (data used: 856.8 MB)
Server: Hivane NetWork Cubic - Ivry-sur-Seine (id: 49781)
Download: 30.76 Mbps (data used: 46.6 MB)
Upload: 120.26 Mbps (data used: 190.6 MB)
So, any VM, computer or whatever (not directrly inside proxmox 1) downloading internet things thrue opnsense got worst download speed ever.
Speedtest is the best i can do. but when i try do download alpine iso (exemple)
im 700Mb/s from proxmox 1, (or any vm hosted by proxmox one)
im 1Mbs/s from anything not hosted inside proxmox one itself.
If someone have any idea ???
Thx a lot