Using bond, vmbr and vlan at the same time - confusion

[subjectx]

Greetings,

I have went through alot of instructions on how to configure networking side of proxmox, but still cannot manage to do next organization:
having one public IP, inside proxmox local IP pool with two or three separate vlans. All VMs should be accessible from outside.

Server has two NIC, connected to separate switches, bonded together with LACP(802.3ad), added vmbr0 with vlan awareness:

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual

auto eno2
iface eno2 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves eno1 eno2
        bond-miimon 100
        bond-mode 802.3ad
        bond-downdelay 200
        bond-updelay 200

auto vmbr0
iface vmbr0 inet static
        address 21X.XXX.XXX.28/29
        gateway 21X.XXX.XXX.25
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

I would like to have 10.9.8.7 IP on one of my VMs (runing ubuntu if that matters), how do I manage that?
I tried entering my public IP on bond0 and then private IP range on vmbr0 but it nags that I cannot have IP on a if that I'm bonding to.

Bonus question:
How do I route traffic via some domain (domain is set to point onto public IP of proxmox server) to this VM? HAProxy running on barebone server?

To not look like I'm that guy that installs something and then starts shouting this doesnt work, I did research on few pages:
- https://wiki.debian.org/NetworkConfiguration#Bringing_up_an_interface_without_an_IP_address
- https://pve.proxmox.com/wiki/Networ...ith_tt_span_class_monospaced_iptables_span_tt -> this here has alot of info, but looking at Linux Bond, first ifconfig shown doesnt work for me, proxmox says I cannot have address on bond0 (iface bond0 - ip address can't be set on interface if bridged in vmbr0 (500)), second one is like like mine current, but with public IP instead of private IP..
- https://forum.proxmox.com/threads/ve_host-web-interface-setup-for-dhcp.27481/
- many others from this forum..

At first I thought to set up public IP on eno1 and eno2, bond them together into bond0 without IP and then bridge vmbr0 with bond0 and set local IP range, but that doesnt work, cannot access server anymore..

 

[wolfgang]

Hi,

first of all, is the bond working?

ip -c link show 
cat /proc/net/bonding/bond0

I miss the hashing policy at the bond level.

 

[subjectx]

Greetings,

thank you for reply.

Yes, bond0 was working. I could apt update OS, I could ping in and out. Even removing cable from one NIC is detected and there is no link downtime.

I did have to break bond apart, because I didnt know what to do with OP and how to solve the problem, so I'm back on one link (without redundancy) to actually test how proxmox works at all and at later time will implement redundancy back.

Currently I have next ifconfig:

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet static
        address 21*.***.***.28/29
        gateway 21*.***.***.25
        pointopoint 21*.***.***.25

iface enp67s0f0 inet manual
iface enp67s0f1 inet manual
iface eno2 inet manual

auto vmbr0
iface vmbr0 inet static
        address 10.10.10.1/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0

        post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE

I still have one public IP directly on NIC and then vmbr0 for VMs to use. From host I can ping gateway and internet.

I tried installing multiple different VMs: pfsense, proxmox backup, Ubuntu, and none have internet access, cant even ping vmbr0.
For example, in Ubuntu 20.04 I have config shown in attachment. netplan apply returns NO error. No difference if I comment out that gateway4.
I also attached ip a from Ubuntu VM.

Pinging anything (domain, ip) from VM returns Destination Host Unreachable.

What am I missing?