So I'm working on building out a Proxmox cluster to mimic our production environment. This mimicry is requiring a unusual network setup, that while I have it working, has me wanting a cleaner solution.
What makes it unusual is that in our production environment, all hosts are fully L2 isolated, and connected via L3 routing. Meaning every host is in a /31 subnet, with just itself, and the router. Now we could just bridge the VMs directly to the physical network, using ebtables to ensure isolation between them, and continue like we have been doing. However this makes it difficult to handle things like moving VMs between hosts in the cluster, or allowing users to launch VMs without needing to get the network team involved to provision on the router.
So what I've done is to setup a local dhcp with a pool of /31 subnets that have been allocated to the proxmox cluster. When a host requests a DHCP address, after allocating it, the dhcpd calls a script which uses the client's MAC address to find the VM ID & interface number, and then creates a udev rule which watches for that TUN interface, which then adds the IP of the VM's gateway to the bridge on the Proxmox host (and removes it when the TUN interface disappears). I then have BIRD watching for these subnets, and then advertising them upstream via BGP.
It's not too terrible. However I feel like this would be more cleanly implemented via a SDN plugin. I do see proxmox has SDN support, and that it's labeled as experimental. But I couldn't find any documentation around creation of custom SDN plugins. Is 3rd party SDN plugins a goal of the project? Is there or will there be any documentation around such? Would this be something that can be implemented via SDN, or would this custom solution be more appropriate?
What makes it unusual is that in our production environment, all hosts are fully L2 isolated, and connected via L3 routing. Meaning every host is in a /31 subnet, with just itself, and the router. Now we could just bridge the VMs directly to the physical network, using ebtables to ensure isolation between them, and continue like we have been doing. However this makes it difficult to handle things like moving VMs between hosts in the cluster, or allowing users to launch VMs without needing to get the network team involved to provision on the router.
So what I've done is to setup a local dhcp with a pool of /31 subnets that have been allocated to the proxmox cluster. When a host requests a DHCP address, after allocating it, the dhcpd calls a script which uses the client's MAC address to find the VM ID & interface number, and then creates a udev rule which watches for that TUN interface, which then adds the IP of the VM's gateway to the bridge on the Proxmox host (and removes it when the TUN interface disappears). I then have BIRD watching for these subnets, and then advertising them upstream via BGP.
It's not too terrible. However I feel like this would be more cleanly implemented via a SDN plugin. I do see proxmox has SDN support, and that it's labeled as experimental. But I couldn't find any documentation around creation of custom SDN plugins. Is 3rd party SDN plugins a goal of the project? Is there or will there be any documentation around such? Would this be something that can be implemented via SDN, or would this custom solution be more appropriate?
