Hello - this didnt work. But I noticed that the proxmox hostname is set to the current bridge ip.( 172.20.2.3). Does this change it?I can try again.perhaps something I missed.
Changing proxmox management ip
- Thread starter juju01
- Start date
-
- Tags
- bridge management vlan
I think I certainly have something misconfigured. Instead of trying to troublehsoot it, lets blow it all up and start fresh? The goal is to put the proxmox management gui on the pfsense management vlan ( is that best practice? ), then allow for all lxc containers to be vlan aware - so I can put containers anywhere I want. I dont mind starting from scratch with a clean install if I can save my containers and vms and restore them?
Last edited:
You don't need a clean install, it looks for me all good. If you reinstall, it won't help you.
Seems for me simply that your switch isn't configured right, or your 10.0/24 network isn't on vlan 99. Or it is on 99, but you on the whole way between the switches, one port is missing vlan 99 etc... Something like that.
But it could be either that i forgot something in the network/interfaces config or it doesn't work as i want....
That's why the best way is simply to put a vm into vlan 99... Your vmbr0 is vlan aware, so you don't need anything to configure.
If then the vm still can't reach pfsense, well it's definitively the way in between then... The switchports etc...
Seems for me simply that your switch isn't configured right, or your 10.0/24 network isn't on vlan 99. Or it is on 99, but you on the whole way between the switches, one port is missing vlan 99 etc... Something like that.
But it could be either that i forgot something in the network/interfaces config or it doesn't work as i want....
That's why the best way is simply to put a vm into vlan 99... Your vmbr0 is vlan aware, so you don't need anything to configure.
If then the vm still can't reach pfsense, well it's definitively the way in between then... The switchports etc...
@Ramalama - So I have changed this around abit to simplify things and potential correct any configurations.
I now have :
I now have :
- proxmox management/gui on 10.10.100.20 with gateway @10.10.100.1. This is the same vlan as my network mamagement vlan ( 1100). At least I can log in etc.
- I now want to make this bridge vlan aware so that any other LXC container I setup, say in vlan 1040, will go to 10.10.100.40 etc...
Code:
auto lo
iface lo inet loopback
iface eno1 inet manual
auto eno5
iface eno5 inet manual
auto eno6
iface eno6 inet manual
auto eno7
iface eno7 inet manual
auto eno3
iface eno3 inet manual
auto eno4
iface eno4 inet manual
auto eno2
iface eno2 inet manual
auto eno8
iface eno8 inet manual
auto vmbr0
iface vmbr0 inet static
address 10.10.100.20/24
gateway 10.10.100.1
bridge-ports eno1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
Last edited:
With this config you can put your containers or vm's, in any vlan you want. Right
not really - still not working. I tried the code block above alone then also added the block below before the vmbro block in a second attempt. Still didn't work.
Code:
auto eno1.1100
iface eno1.100 inet static
address 10.10.100.20/24
gateway 10.10.100.1
vlan-raw-device eno1You have in one eno1.1100 in the other eno1.100
However, this:
Code:
auto vmbr0
iface vmbr0 inet static
address 10.10.100.20/24
gateway 10.10.100.1
bridge-ports eno1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094Is the most basic and easy config.
It means:
10.10.100.20 = mgmt ip
10.10.100.0/24 is untagged pvid (vlan xxxx) on your switch.
Vlan aware 2-4094 = you can assign in your vms any vlan tag you want. If you don't add a tag = 10.10.100.0/24 network
If that doesn't work, well sorry, but you have somewhere on the switches a misconfiguration.
Cheers
@Ramalama
I managed to get it to work with the setup below. Its been working great for awhile now. I am now trying to change things around again.
I'd like to setup an LACP aggregation using eno7 and eno8 ( 2 sftp+ ports ). Then, move the vlan setup from eno5 to the new bonded LACP ports. Will appreciate if you can point me in the right direction. Should I use a Linux Bond or OVS bond ?
I managed to get it to work with the setup below. Its been working great for awhile now. I am now trying to change things around again.
I'd like to setup an LACP aggregation using eno7 and eno8 ( 2 sftp+ ports ). Then, move the vlan setup from eno5 to the new bonded LACP ports. Will appreciate if you can point me in the right direction. Should I use a Linux Bond or OVS bond ?
Code:
auto lo
iface lo inet loopback
iface eno1 inet manual
auto eno5
iface eno5 inet manual
auto eno6
iface eno6 inet manual
auto eno7
iface eno7 inet manual
auto eno3
iface eno3 inet manual
auto eno4
iface eno4 inet manual
auto eno2
iface eno2 inet manual
auto eno8
iface eno8 inet manual
auto vmbr0
iface vmbr0 inet static
address 10.0.0.21/24
gateway 10.0.0.1
bridge-ports eno1
bridge-stp off
bridge-fd 0
auto vmbr1020
iface vmbr1020 inet manual
bridge-ports eno5.1020
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#lab vlan
auto vmbr1030
iface vmbr1030 inet manual
bridge-ports eno5.1030
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#iot vlan
auto vmbr1010
iface vmbr1010 inet manual
bridge-ports eno5.1010
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#home vlan
auto vmbr99
iface vmbr99 inet manual
bridge-ports eno5.99
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#dmz vlanSmall background: eno=phy & vmbr=virtual
You are using in your bridges physical vlan ports (eno5.99 for example), that means for me that that bridge doesn't need to be vlan aware... cause you bind already to phy port vlan.
Maybe im wrong and it works in your config like pvid 99 and vlan aware for tagging, but i doubt it. it looks wrong somehow...
however, my idea was to make this all much easier, like:
Code:
auto lo
iface lo inet loopback
auto eno1
iface eno1 inet static
address 10.0.0.21/24
gateway 10.0.0.1
dns-nameservers 10.0.0.1 8.8.8.8
auto eno2
iface eno2 inet manual
auto eno3
iface eno3 inet manual
auto eno4
iface eno4 inet manual
auto eno5
iface eno5 inet manual
auto eno6
iface eno6 inet manual
auto eno7
iface eno7 inet manual
bond-master bond0
auto eno8
iface eno8 inet manual
bond-master bond0
auto bond0
iface bond0 inet manual
bond-mode 802.3ad
bond-miimon 100
bond-downdelay 200
bond-updelay 200
bond-lacp-rate 1
bond-min-links 1
bond-slaves none
auto vmbr0
iface vmbr0 inet manual
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094Take the portnames as example, i don't know which ports you want to use.
So basically, proxmox itself has an ip on eno1, the 802.3ad bond doesn't have any ip and vmbr0 sits on bond0 with all vlan's.
So you can connect all your VM's to vmbr0 and assign a tag in the proxmox gui for every vm-networkadapter.
For the opensense, you need to give only one networkadapter (without a tag in the gui) and you create vlans in opnsense itself. (Just don't forget to disable the offloading in opnsense)
LRO-> Is incompatible with packet forwarding
TX/RX -> Is buggy with vmxnet on freebsd
Vlan Hardware Offloading -> May work, leave it default.
Some more background, maybe it helps you a bit, since i am an slow replyer xD
- you need either ifenslave+ifupdown or only ifupdown2 (ifenslaves is integrated there)
- you don't need any other packages or modules... Proxmox comes with everything.
- you can either use inside enoX "bond-master bond0" or inside bond0 "bond-slaves enoX enoX enoX...", but not both together.
- bond-mode: https://wiki.pratznschutz.com/index.php/Network_Bonding (directly at top) (802.3ad is a synonyme for 4)
- bond-miimon: checks all 100ms the linkstatus of the slave interfaces...
- bond-up/downdelay: 200ms till an offline/online interface gets added/removed
- bond-lacp-rate 1 (0/1 slow/fast lacp control packet transmission)
- iface XXXX inet manual (basically for interfaces without ip) / static (For interfaces with an ip)
Bridge:
- bridge-vids 20 30 40-1200 (vlans that you want)
- bridge-pvid 20 (Primary trunk vlan)
- bridge-access 20 (Access Vlan)
- More: https://manpages.debian.org/testing/ifupdown2/ifupdown-addons-interfaces.5.en.html
- you don't need bridge-pvid if you want to configure a pvid on your switch. But you can use both and even multiple bridges bound to bond0 with different pvid's.
Im only not sure if you need then vlanaware on a bridge with only one vlan. but you can test it and play with the options.
I hope you understand it a bit better now.
Cheers
EDIT:
Ah and about linux-bridge vs ovs... Forget ovs, basically the linux bridge can everything and is probably even more efficient.
Last edited:
@Ramalama - Thanks for the detailed write up. it cleared a lot of things for me. I got it all working with your new code, with one exception - a VM on the native vlan 1 is not able to get an ip address from the new bond and vmbr0 bridge. I have 3 other vlans, and I am able to assign them to other vms without a problem. I have tried setting the vlan tag to 1 and not setting it at all - the vm boots and gets stuck at where it needs to pull an ip address. here is what it looks like now:
i wrote somewhere, that vlan1 (tagged) is problematic.
best would be, if you make vlan1 untagged pvid on your switch and your vmbr has then vlan1 as default with the rest tagged.
is your bond up? you could cat the status of /sys/class/net/bond0 if i remember it right xD
best would be, if you make vlan1 untagged pvid on your switch and your vmbr has then vlan1 as default with the rest tagged.
is your bond up? you could cat the status of /sys/class/net/bond0 if i remember it right xD
Last edited:
what about the rest not working now after the reboot? They all look like the network device screen above but with different vlan tags and mac addresses. Seems the bond or bridge is not working.
exactly.
i think it's the bond. you need to check first if the bond is up. if it is, probably some sort of lacp misconfiguration between switch and bond.
There are tons of options, maybe you need a hash policy etc.
What would help in first place is, if you simply add an address to the bond 0 itself, without gateway or anything. then check from there if the lacp works....
you need to google a bit and try yourself some options. and you don't need to reboot always, simply bring interfaces up and down with ifupdown2.
the syntax above is for ifupdown2, so check if you have that installed, instead of ifupdown+ifenslaves.
or post me simply your config and as much details you can get... ifconfig, cat of the bond and network interfaces config.
i will look today, but a bit later, since i have just a phone now and im not at home xD
Last edited:
and in the case that it works, if you bring the interface up with ifupdown2, but it doesn't work on boot.
this looks more like an issue with systemd-networkd...
systemd uses another network configuration inside /etc/systemd/network...
probably just:
systemctl disable systemd-networkd.socket systemd-networkd.service
systemctl mask systemd-networkd.socket systemd-networkd.service
systemctl unmask networking
systemctl enable networking
i need to check later at home, if systemd conflicts with ifupdown2. And what proxmox modifies.
You can get away without disabling systemd, just deleting the network configuration for your interface in the systemd folder. If there is one.
Or you can use systemd-networkd for your complete network configuration, including bonding and bridges. It works the same, just another syntax.
Then you won't need /etc/network/interfaces and ifupdown2.
Only the configuration inside /etc/systemd/network
You see there are many ways, to realize the same thing, and cause there are many ways, there are many possible conflicts if something doesn't work, that makes that shit harder to find xD
this looks more like an issue with systemd-networkd...
systemd uses another network configuration inside /etc/systemd/network...
probably just:
systemctl disable systemd-networkd.socket systemd-networkd.service
systemctl mask systemd-networkd.socket systemd-networkd.service
systemctl unmask networking
systemctl enable networking
i need to check later at home, if systemd conflicts with ifupdown2. And what proxmox modifies.
You can get away without disabling systemd, just deleting the network configuration for your interface in the systemd folder. If there is one.
Or you can use systemd-networkd for your complete network configuration, including bonding and bridges. It works the same, just another syntax.
Then you won't need /etc/network/interfaces and ifupdown2.
Only the configuration inside /etc/systemd/network
You see there are many ways, to realize the same thing, and cause there are many ways, there are many possible conflicts if something doesn't work, that makes that shit harder to find xD
Code:
auto lo
iface lo inet loopback
auto eno1
iface eno1 inet static
address 10.0.0.21/24
gateway 10.0.0.1
dns-nameservers 10.0.0.1 8.8.8.8
auto eno2
iface eno2 inet manual
auto eno3
iface eno3 inet manual
auto eno4
iface eno4 inet manual
auto eno5
iface eno5 inet manual
auto eno6
iface eno6 inet manual
iface eno7 inet manual
iface eno8 inet manual
auto bond0
iface bond0 inet manual
bond-mode 802.3ad
bond-miimon 100
bond-downdelay 200
bond-updelay 200
bond-lacp-rate 1
bond-min-links 1
bond-slaves eno7 eno8
bond-xmit-hash-policy layer2+3
auto vmbr0
iface vmbr0 inet manual
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 1-4094
Code:
apt install ifupdown2
systemctl disable systemd-networkd.socket systemd-networkd.service
systemctl mask systemd-networkd.socket systemd-networkd.service
systemctl unmask networking
systemctl enable networking
systemctl restart networkingDo this, i added there "bond-xmit-hash-policy layer2+3", probably it helps.
And "bridge-vids 1-4094" 1 should work tagged there. So it's probably how you want. But i would prefer to make on the switch vlan1 pvid untagged. However, both ways will work, just tagged vlan 1 can sometimes cause issues or not work.
And if you want to test/play with all the options inside /etc/network/interfaces, you can always simply "systemctl restart networking".
That will restart the network interfaces / apply configs, through ifupdown2, the easy way.
Look here for more details: https://pve.proxmox.com/pve-docs/pve-network-plain.html
Last edited:
@Ramalama
Managed to get it working with this .
Managed to get it working with this .
Code:
auto lo
iface lo inet loopback
iface eno1 inet manual
auto eno5
iface eno5 inet manual
auto eno6
iface eno6 inet manual
auto eno7
iface eno7 inet manual
auto eno3
iface eno3 inet manual
auto eno4
iface eno4 inet manual
auto eno2
iface eno2 inet manual
auto eno8
iface eno8 inet manual
auto bond0
iface bond0 inet manual
bond-slaves eno7 eno8
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3
auto vmbr0
iface vmbr0 inet static
address 10.0.0.21/24
gateway 10.0.0.1
bridge-ports eno1
bridge-stp off
bridge-fd 0
auto vmbr1
iface vmbr1 inet manual
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094This looks very good. So yes, you needed the hash policy xD
You could try on your vmbr1 to set bridge-vids to 1-4094, then you can use vlan 1. Even if i don't recommend it, i did it myself and it worked.
and about vmbr0, im wondering that it didn't worked on the interface directly? So you had to make the workaround over the vmbr0 bridge?
congratulations anyway, good config xD