Hello all,
This is my first post on this forum, so please excuse me if I'm asking this in the wrong place or doing something else wrong. Please let me know, then I can correct it.
What I'm trying to do is: start using SDN within my PVE cluster, backed by my Netbox instance. So: if I configure or start a VM, it should register the IP it is going to use in Netbox.
My PVE setup is pretty basic: I have 3 physical hosts. On each node I have 3 bond interfaces:
All this I have working using traditional networking, so on the Datacenter -> pveXX -> System -> Network page I have created a bunch of bond, vlan and bridge interfaces. then I can hook up a VM to a specific bridge and things start working. So far, so good!
Then I decided I wanted to move to the modern world and switch over to SDN. This saves me from configuring all the vlan and bridge interfaces on all PVEs (of which there will be many more in the future, I hope!) and ensures that the network configuration is correct.
So, I removed most of the bridges and the vlans. I only have 1 bridge left (vmbr0) which is for bond0. Not for a specific vlan, just for the entire bond. This is configured as a vlan aware bridge, so I can use VLANs in that way (but I don't want to anymore). I also have 1 vlan interface still left. This is the infra vlan on bond2. This is the actual interface for the proxmox webinterface on each PVE. I might move that to SDN as well later, but I do want to have SDN working correctly fully first. This is a pretty crucial IP afterall.
I started with this migration by adding the 'netbox' IPAM to my configuration. In Netbox I created a normal user, which is activated. I gave this user the permissions to View, Add, Change and Delete things in the Object Types "IPAM | aggregate", "IPAM | IP address" and "IPAM | prefix". I'm not sure if these permissions are correct, but I assume they are
. When I make this user an admin user the behaviour of Proxmox doesn't change, so my issue isn't a permission issue, I think.
I've created an API key in Netbox as well, which I used in Proxmox.
When I check the logging on my Netbox VM I do see HTTP request from the Proxmox management address to Netbox, which are answered by HTTP 200 answers. So, that is good!
I have created 1 zone, which I called "frontend". I have configured vmbr0 as bridge interface. This zone is of the type 'vlan'. When I read https://pve.proxmox.com/wiki/Software-Defined_Network#pvesdn_config_zone this is the type that matches my requirements most closely, imho. I want to have this zone available on all 3 of my PVEs and a VM on pve01 should be able to talk to a VM on pve02 which has a network interfaces configured in the same zone and vnet.
So, a 'simple' zone doesn't qualify because that is only known on 1 PVE. A 'QinQ' zone doesn't qualify because I'm not gonna nest VLANs. A 'VXLAN' doesn't qualify because I do traditional VLANs, not VXLAN. A 'EVPN' doesn't qualify because I'm not routing the inter-vm communication, I'm still switching. Or at least: this is my train of thought. Please correct me if I'm wrong.
I did configure the 'netbox' IPAM for this zone as well.
Then I created the VNets for my networks. One for each VLAN that I use. I entered the correct vlan ID in the "Tag" field, and I unchecked the "vlan aware" checkbox.
And to top it all off I created 1 or more subnets in each VNet. I have 2 public iP address ranges (1 /27, 1 /28), so I need to configure 2. For each subnet I have configured a gateway and a DNS zone prefix. I have the SNAT box unchecked. I also configured an DHCP range for each subnet. The gateway IP addresses (which are present, but live outside of my PVE cluster on physical hardware) are left outside these ranges. These gateway IP addresses did get registered in Netbox by Proxmox. So at least I did something right!
Now I have to get a VM to use these subnets, find a free IP address in Netbox, or the correct IP address for this VM, and start using that on the interface inside of the VM. I'm running Debian 12.8 as VMs to configure this with. And I don't know how to achive this.
I have configured IP addresses in Netbox for the VM to use. I configured the MAC-address of the interface next to the IP address, so I think Netbox should be able to find the correct address. I have installed dnsmasq on all my PVEs, and disabled the service (like it should, accourding to https://pve.proxmox.com/wiki/Software-Defined_Network#pvesdn_installation). When I check my PVEs I don't see dnsmasq running:
When I create a simple SDN Zone with the DHCP option enabled, I do see dnsmasq getting started, but I don't need a simple zone, I need a vlan zone .
If I configure the VM to ask an IP address via DHCP it doesn't get a reply (which makes sense, since there aren't any DHCP servers running ).
When I check the logging of Netbox, I don't see any traffic from my PVEs towards Netbox when I start a VM or start the DHCP client proces on it.
When looking through these forums for hints I did find the following thread from earlier this year: https://forum.proxmox.com/threads/sdn-problems-with-netbox-as-ipam.147395/. Here the TS states that the IP address does get registered in Netbox, so I have good hope that it should be possible. But the TS is using the simple zone-type in SDN, and not the vlan-type, like me.
How do I get a VM to actually use Netbox as a source for which IP address it should be using? I hope somebody can point me into the direction of what I'm doing wrong.
Sorry for the long post, but I wanted to give you as much information as possible, so you know what my setup is, and what I tried, so that if anybody is willing to help me, he or she has all the information required.
Kind regards,
Cybertinus
Edit:
I forgot to mention the versions of Proxmox and Netbox that I use:
Proxmox 8.2.7
Netbox 4.1.6
Both I fully patched and rebooted yesterday, so as far as I know I should be fully up-to-date.
This is my first post on this forum, so please excuse me if I'm asking this in the wrong place or doing something else wrong. Please let me know, then I can correct it.
What I'm trying to do is: start using SDN within my PVE cluster, backed by my Netbox instance. So: if I configure or start a VM, it should register the IP it is going to use in Netbox.
My PVE setup is pretty basic: I have 3 physical hosts. On each node I have 3 bond interfaces:
- bond0 is what I call "frontend". So, the VMs connect to the internet via this bond, I do VM migrations over this bond, I make VM backups over this bond, the network for my own internal VMs which shouldn't be reachable from the internet is running over this bond and I will add a management interface to each internet facing VM for stuff like SSH via a VPN. Each of these functions run in a seperate VLAN on this bond.
- bond1 I call "storage". All my Ceph communication is running over this bond.
- bond2 is for "management". There are 2 vlans here: 1 for infrastructure, 1 for cluster communication. The PVE ip address is configured in the infrastructure vlan.
All this I have working using traditional networking, so on the Datacenter -> pveXX -> System -> Network page I have created a bunch of bond, vlan and bridge interfaces. then I can hook up a VM to a specific bridge and things start working. So far, so good!
Then I decided I wanted to move to the modern world and switch over to SDN. This saves me from configuring all the vlan and bridge interfaces on all PVEs (of which there will be many more in the future, I hope!) and ensures that the network configuration is correct.
So, I removed most of the bridges and the vlans. I only have 1 bridge left (vmbr0) which is for bond0. Not for a specific vlan, just for the entire bond. This is configured as a vlan aware bridge, so I can use VLANs in that way (but I don't want to anymore). I also have 1 vlan interface still left. This is the infra vlan on bond2. This is the actual interface for the proxmox webinterface on each PVE. I might move that to SDN as well later, but I do want to have SDN working correctly fully first. This is a pretty crucial IP afterall.
I started with this migration by adding the 'netbox' IPAM to my configuration. In Netbox I created a normal user, which is activated. I gave this user the permissions to View, Add, Change and Delete things in the Object Types "IPAM | aggregate", "IPAM | IP address" and "IPAM | prefix". I'm not sure if these permissions are correct, but I assume they are
. When I make this user an admin user the behaviour of Proxmox doesn't change, so my issue isn't a permission issue, I think.I've created an API key in Netbox as well, which I used in Proxmox.
When I check the logging on my Netbox VM I do see HTTP request from the Proxmox management address to Netbox, which are answered by HTTP 200 answers. So, that is good!
I have created 1 zone, which I called "frontend". I have configured vmbr0 as bridge interface. This zone is of the type 'vlan'. When I read https://pve.proxmox.com/wiki/Software-Defined_Network#pvesdn_config_zone this is the type that matches my requirements most closely, imho. I want to have this zone available on all 3 of my PVEs and a VM on pve01 should be able to talk to a VM on pve02 which has a network interfaces configured in the same zone and vnet.
So, a 'simple' zone doesn't qualify because that is only known on 1 PVE. A 'QinQ' zone doesn't qualify because I'm not gonna nest VLANs. A 'VXLAN' doesn't qualify because I do traditional VLANs, not VXLAN. A 'EVPN' doesn't qualify because I'm not routing the inter-vm communication, I'm still switching. Or at least: this is my train of thought. Please correct me if I'm wrong.
I did configure the 'netbox' IPAM for this zone as well.
Then I created the VNets for my networks. One for each VLAN that I use. I entered the correct vlan ID in the "Tag" field, and I unchecked the "vlan aware" checkbox.
And to top it all off I created 1 or more subnets in each VNet. I have 2 public iP address ranges (1 /27, 1 /28), so I need to configure 2. For each subnet I have configured a gateway and a DNS zone prefix. I have the SNAT box unchecked. I also configured an DHCP range for each subnet. The gateway IP addresses (which are present, but live outside of my PVE cluster on physical hardware) are left outside these ranges. These gateway IP addresses did get registered in Netbox by Proxmox. So at least I did something right!
Now I have to get a VM to use these subnets, find a free IP address in Netbox, or the correct IP address for this VM, and start using that on the interface inside of the VM. I'm running Debian 12.8 as VMs to configure this with. And I don't know how to achive this.
I have configured IP addresses in Netbox for the VM to use. I configured the MAC-address of the interface next to the IP address, so I think Netbox should be able to find the correct address. I have installed dnsmasq on all my PVEs, and disabled the service (like it should, accourding to https://pve.proxmox.com/wiki/Software-Defined_Network#pvesdn_installation). When I check my PVEs I don't see dnsmasq running:
Code:
root@pve01:~# ps aux | grep dnsmasq | grep -vc grep
0If I configure the VM to ask an IP address via DHCP it doesn't get a reply (which makes sense, since there aren't any DHCP servers running
).When I check the logging of Netbox, I don't see any traffic from my PVEs towards Netbox when I start a VM or start the DHCP client proces on it.
When looking through these forums for hints I did find the following thread from earlier this year: https://forum.proxmox.com/threads/sdn-problems-with-netbox-as-ipam.147395/. Here the TS states that the IP address does get registered in Netbox, so I have good hope that it should be possible. But the TS is using the simple zone-type in SDN, and not the vlan-type, like me.
How do I get a VM to actually use Netbox as a source for which IP address it should be using? I hope somebody can point me into the direction of what I'm doing wrong.
Sorry for the long post, but I wanted to give you as much information as possible, so you know what my setup is, and what I tried, so that if anybody is willing to help me, he or she has all the information required
.Kind regards,
Cybertinus
Edit:
I forgot to mention the versions of Proxmox and Netbox that I use:
Proxmox 8.2.7
Netbox 4.1.6
Both I fully patched and rebooted yesterday, so as far as I know I should be fully up-to-date.
Last edited: