Hello there,
I've been working on a proof of concept on a testserver of mine for Networking without any bridges, just with tap interfaces.
Qemu does allow you to run vms without any bridge, only creating a tap interface on the system but it is not an option in proxmox yet.
I think espacially if you have vms with public IP addresses it makes sence to be without a bridge, especally if there is no need for any broadcast traffic or services that rely on being in the same L2 domain. It gives the ability to assign an IP address to the tap interface and use that as gateway from the vm.
The IP adress for that can be the same on every tap Interface for every vm and that even on every pve node, thus allowing migration easely.
To get it to work i changed the logic in qemuserver.pm and in network.qm to include a option to set the parameter taprouted on a vm network interface and added helper scripts that add the ip address on the tap and a route to the guest address.
I use it with setting a /32 address on the guest and a onlink gateway, which works just fine under linux and windows guests.
Configuration of the tap interface is done in the following manner:
Please note that for that to work the tap interface can't be attached to any bridge and for traffic to be forwarded ip forwarding needs to be enabled on the host.
This configuration then allows to announce the host routes via bgp or other protocols to your routers for exemple with a frr configuration like this:
I see many possible advantages in a configuration like this:
- Pure L3 design without L2 dependencies Eliminates the need for bridges, broadcast domains, ARP flooding, and other L2-related complexity when it is not required.
This aligns well with modern routed datacenter designs.
- Integrates cleanly into environments using OSPF or IS-IS internally with BGP on top, allowing VM reachability to be treated as regular host routes rather than virtual L2 segments.
- Each VM can be announced as a host route from the Proxmox node it is currently running on, making routing behavior explicit and deterministic.
- allowing to easely archiving redundandent paths without using legacy ways like lacp
- allowing to utulize ecmp over an igp
- in contrast to evpn/vxlan always routing via the right proxmox node inbound from border gateways without the added complexety
I did open a pull request regarding these changes and it works in my test system but since I'm not a developer I really don't know if i did it right.
https://github.com/proxmox/qemu-ser...80607d4c5b14a2c01d1bc4cd86843c143aefad6fbc2ca
I would welcome feedback from the community on this approach and whether it is considered useful or viable by others.
I've been working on a proof of concept on a testserver of mine for Networking without any bridges, just with tap interfaces.
Qemu does allow you to run vms without any bridge, only creating a tap interface on the system but it is not an option in proxmox yet.
I think espacially if you have vms with public IP addresses it makes sence to be without a bridge, especally if there is no need for any broadcast traffic or services that rely on being in the same L2 domain. It gives the ability to assign an IP address to the tap interface and use that as gateway from the vm.
The IP adress for that can be the same on every tap Interface for every vm and that even on every pve node, thus allowing migration easely.
To get it to work i changed the logic in qemuserver.pm and in network.qm to include a option to set the parameter taprouted on a vm network interface and added helper scripts that add the ip address on the tap and a route to the guest address.
I use it with setting a /32 address on the guest and a onlink gateway, which works just fine under linux and windows guests.
Configuration of the tap interface is done in the following manner:
ip addr add 10.10.10.254/32 dev tap100i0ip route add 192.168.1.50/32 dev tap100i0Please note that for that to work the tap interface can't be attached to any bridge and for traffic to be forwarded ip forwarding needs to be enabled on the host.
This configuration then allows to announce the host routes via bgp or other protocols to your routers for exemple with a frr configuration like this:
Code:
frr version 10.3.1
frr defaults datacenter
hostname proxmox
log syslog informational
service integrated-vtysh-config
!
!
router bgp 65050
bgp router-id 10.97.50.1
no bgp default ipv4-unicast
coalesce-time 1000
neighbor BGP peer-group
neighbor BGP remote-as 65050
neighbor BGP bfd
neighbor 10.97.50.254 peer-group BGP
!
address-family ipv4 unicast
redistribute kernel route-map ROUTED-TAP
neighbor BGP activate
neighbor BGP soft-reconfiguration inbound
exit-address-family
exit
!
route-map ROUTED-TAP permit 10
match ip address prefix-len 32
!
line vty
!I see many possible advantages in a configuration like this:
- Pure L3 design without L2 dependencies Eliminates the need for bridges, broadcast domains, ARP flooding, and other L2-related complexity when it is not required.
This aligns well with modern routed datacenter designs.
- Integrates cleanly into environments using OSPF or IS-IS internally with BGP on top, allowing VM reachability to be treated as regular host routes rather than virtual L2 segments.
- Each VM can be announced as a host route from the Proxmox node it is currently running on, making routing behavior explicit and deterministic.
- allowing to easely archiving redundandent paths without using legacy ways like lacp
- allowing to utulize ecmp over an igp
- in contrast to evpn/vxlan always routing via the right proxmox node inbound from border gateways without the added complexety
I did open a pull request regarding these changes and it works in my test system but since I'm not a developer I really don't know if i did it right.
https://github.com/proxmox/qemu-ser...80607d4c5b14a2c01d1bc4cd86843c143aefad6fbc2ca
I would welcome feedback from the community on this approach and whether it is considered useful or viable by others.