New to Proxmox, some guidance needed for network config

[lpallard]

Hello !

First of all, let me congratulate the Proxmox dev team for the wonder that Proxmox is!! Unbelievable!

After several weeks (months) searching for the ideal hypervisor for my home/small office server, I have decided to opt for Proxmox. So far things have been fairly smooth and easy (installation was actually a piece of cake).

I am now wondering about network config. The goal here is to:

1. Eliminate the physical pfsense router and virtualize it
2. Have the physical machines on my LAN obtain an IP from the pfsense VM (just like it was previously done with the physical router)
3. Have the other proxmox VM's get an ip from the pfsense vm
4. Proxmox node shall be accessible only from LAN (from one of the physical machines)

I have prepared a simple diagram to show what I want to do. Is it feasible?

Thanks!!

 

[pirateghost]

Sure it is. I do this very thing in my home configuration.

you have a couple of choices with your VMs network config.
A. Configure a vmbr with no eth device attached and add that to your pfsense VM as another nic, and utilize your PFsense install to control it as a DMZ or separate LAN altogether
B. use the same VMBR that your LAN will use if you want them to be on the same subnet as the rest of your LAN devices.

 

[lpallard]

Thanks for replying!

I am not fully understanding your reply. So how would bridges work for the pfsense VM?

I am not sure how bridges work. So why do proxmox creates a bridge (vmbr0) using eth0 right after the install? Can I use eth devices directly and assign these to VM's or do I have to create bridges instead?

 

[pirateghost]

Thanks for replying!

I am not fully understanding your reply. So how would bridges work for the pfsense VM?

I am not sure how bridges work. So why do proxmox creates a bridge (vmbr0) using eth0 right after the install? Can I use eth devices directly and assign these to VM's or do I have to create bridges instead?

No. Do not use ETH devices directly.
VMBR0 should be your INTERNAL management interface. dont mess with it
Create a VMBR<X> (any number will suffice) and assign it to the eth device you want to act as your WAN port. do not allow any other VM to use that VMBR at all.
Create a VMBR<X> (any number will suffice) and assign it to the eth device you want to act as your LAN port. You can allow other VMs to use this same VMBR if you want them to be on the same Internal network as your other devices.
Create a VMBR<X> (any number will suffice) and assign it to the eth device you want to act as your DMZ port. Anything you want on the DMZ can use this VMBR

In this instance, you will give your pfsense VM all 3 VMBR devices, and configure them as you would a normal hardware pfsense install. Remember your DMZ should be on a different subnet than your LAN or you need to get crafty with the subnets and break your LAN down into smaller segments.

Alternatively, if you only have a need for VMs to be on the DMZ, you can create a VMBR<X> and not assign it any eth device and it will function the same as your DMZ bridge you created. Add it as an interface to your PFSense and you have a VM only DMZ, any VM that needs to be on a DMZ of its own can be assigned to that VMBR and it will pass through your PFsense just like plugging in a hardware nic.

*Note: If you only have the 3 eth devices in the host box, you can use your VMBR0 as your internal LAN segment on PFsense

 

[lpallard]

Hey pirateghost!

I have 6 eth devices (supermicro board has eth0 & eth1), and a Intel PRO/1000 Quad NIC has eth2 to eth5

I want to use eth0 for the WAN (incoming from cable modem), eth1 as LAN (going to my switch and then computers), and eth2 for WAN.

Can I safely reassign eth0 to the bridge I will create (lets say vmbr1) for the WAN port, and then reassign eth1 to vmbr0?

Sorry for such baby steps, I feel stupid but I am clueless in these technologies...

 

[pirateghost]

Hey pirateghost!

I have 6 eth devices (supermicro board has eth0 & eth1), and a Intel PRO/1000 Quad NIC has eth2 to eth5

I want to use eth0 for the WAN (incoming from cable modem), eth1 as LAN (going to my switch and then computers), and eth2 for WAN.

Can I safely reassign eth0 to the bridge I will create (lets say vmbr1) for the WAN port, and then reassign eth1 to vmbr0?

Sorry for such baby steps, I feel stupid but I am clueless in these technologies...

you could do that, although I do not like changing the way that proxmox expects the management network.

you should be able to do this in the GUI and reboot for the change to take effect.

 

[lpallard]

OK some progress! First of all, I reconfigured the node as this: vmbr0 (eth0) is set to 192.168.0.100 and is physically connected to my network switch so I can access Proxmox's web interface from my LAN as desired vmbr1 (eth1) doesnt have an IP and is physically connected to my cable modem (WAN) vmbr2 (eth2) is set to 192.168.1.100 and is physically connected to my network switch so my computers can get an IP from the pfsense VM vmbr3 (eth3) is set to 192.168.2.100 and my VOIP phone is connected to it I assigned vmbr1, 2 & 3 to the pfsense VM and went to install pfsense. Questions: Is what I've done OK? How will I make sure the VM's (and not only the physical machines on my lan) are getting an IP from the pfsense VM? Do I add the vmbr2 bridge to the VM's? First when I boot the install, I get a kernel panic and a "with fatal trap 9". I am running a dual socket AMD server. Looking at https://forum.pfsense.org/index.php?topic=30593.0 it seems that setting "hw.mca.enabled=0" enables to properly boot. I can confirm this works. However the recommendation on the pfsense forum is to echo the command to /boot/loader.conf which doesnt work. I will post on the pfsense forum as well but I thought posting this here in case some of you are aware of this issue...

 

[pirateghost]

You don't need ip addresses on any interface but the one you manage from.

Sent from my Nexus 5

 

[lpallard]

You don't need ip addresses on any interface but the one you manage from. Sent from my Nexus 5

OK progressing.... Slowly but surely So I got the trick to install pfsense without messing around with boot options. Turned out, there is a bug in the kvm driver for freebsd hosts when running AMD CPU's (physical ones of course). See http://forum.proxmox.com/threads/14094-Proxmox-3-on-AMD-E450-pfSense-2-1-NAS4Free-PiaF-Debian YOu have to use qemu64 instead of kvm64 for the CPU virtualization... Now to my network issues: I have successfully installed pfsense, assigned its WAN to the proper bridge, and LAN to the other bridge.. From my desktop computer, I can request an IP from the pfsense virtual machine and access pfsense's webinterface!! but when I do so, I no longer can access proxmox's webinterface. I believe it has to do with subnets. Proxmox web interface (vmbr0) is set to static IP 192.168.0.100 in proxmox's config. pfsense's WAN (vmbr1 = eth1) is set to DHCP (so it can get an IP from my ISP) in pfsense's config. At proxmox's level, it has NO IP pfsense's LAN (vmbr2 = eth2) is set to static IP 192.168.1.100 (only in pfsense's config. At proxmox level is has NO IP). DO you think subnets are to blame?

 

[lpallard]

I dont know why my posts are so screwed up... My line returns are not being parsed by the forum and every line gets appended to the previous one making a mess. Sorry for this....

 

[lpallard]

I got almost everything up & running except that for some reasons, pfsense cant reach the WAN (the outside)..

In pfsense, WAN has an IP of 0.0.0.0...... Also I can get an IP from the pfsense VM but I cant reach the outside. I believe something is wrong in proxmox's config.. Can someone help?

My /etc/network/interfaces:

# network interface settings
auto lo
iface lo inet loopback

auto eth1
iface eth1 inet manual

iface eth3 inet manual

iface eth5 inet manual

auto eth0
iface eth0 inet manual

auto eth2
iface eth2 inet manual

iface eth4 inet manual

auto vmbr0
iface vmbr0 inet static
address 192.168.0.100
netmask 255.255.255.0
bridge_ports eth0
bridge_stp on
bridge_fd 0

auto vmbr1
iface vmbr1 inet manual
bridge_ports eth1
bridge_stp off
bridge_fd 0

auto vmbr2
iface vmbr2 inet manual
bridge_ports eth2
bridge_stp off
bridge_fd 0

auto vmbr3
iface vmbr3 inet manual
bridge_ports eth3
bridge_stp off
bridge_fd 0

 

[lpallard]

OK this time I really hit a wall...

everything works, except proxmox vmbr1 (connected to my modem = WAN) is not getting a public IP from my ISP... THerefore pfsense WAN has no valid IP (0.0.0.0). I have searched on the proxmox wiki but its rather incomplete so I could not get past this issue... Too may potential options and arguments in etc/network/interfaces for me to get it right away....

My /etc/network/interfaces as of now:

# network interface settings

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual

auto eth1
iface eth1 inet manual

auto eth2
iface eth2 inet manual

auto eth3
iface eth3 inet manual

auto eth4
iface eth4 inet manual

auto eth5
iface eth5 inet manual

auto vmbr0
iface vmbr0 inet static
address 192.168.0.1
netmask 255.255.255.0
bridge_ports eth0
bridge_stp on
bridge_fd 0

# WAN
auto vmbr1
iface vmbr1 inet dhcp
bridge_ports eth1
bridge_stp off
bridge_fd 0
post-up echo 1 > /proc/sys/net/ipv4/conf/vmbr1/proxy_arp

# LAN
auto vmbr2
iface vmbr2 inet manual
bridge_ports eth2
bridge_stp off
bridge_fd 0

# DMZ
auto vmbr3
iface vmbr3 inet manual
bridge_ports eth3
bridge_stp off
bridge_fd 0

Whats wrong with my config? I must not be far off...

 

[pirateghost]

Vmbr1 should not have a DHCP directive. Give me a few minutes to get to my computer and I can share my complete interfaces config

Sent from my Nexus 5

 

[lpallard]

Vmbr1 should not have a DHCP directive. Give me a few minutes to get to my computer and I can share my complete interfaces config

Sent from my Nexus 5

pirateghost, you already helped quite a lot, please take your time, I will patiently wait!

 

[pirateghost]

Here is my complete config. I use bonding and VLANs so you can ignore those parts. VMBR0 is my management interface, VMBR1 is my 'LAN' interface, and VMBR2 is my 'WAN' interface. I use VMBR4095 as my VLAN trunk to the router (all VLANs go through here). Then I have VMBR(VLAN-TAG) so I can have containers on VLANs. I use VMBR100 to create a direct crossover to my UTM/Web Filter, to ensure that everything has to route through it to the router. So my router (VyOS) is configured with VMBR2 as the WAN port, VMBR100 as it's LAN port, and VMBR4095 as the VLAN trunk. Then my web filter/UTM uses VMBR100 as its 'WAN' port, VMBR1 as its LAN port, and also VMBR4095 is here too. The UTM is set up as a transparent bridge, so all traffic flows through it and to the router. Essentially I am using 2 devices, where you will be using 1 (PFSense). Make sure that when you configure your PFSense VM, that you reboot your modem in order for it to get an IP. Some ISPs actually require you to call and have them clear the ARP cache in order for you to connect it to a new 'router'.

# network interface settingsauto vlan10
iface vlan10 inet manual
        vlan_raw_device bond0


auto vlan13
iface vlan13 inet manual
        vlan_raw_device bond0


auto vlan192
iface vlan192 inet manual
        vlan_raw_device bond0
        vlan_raw_device bond0


auto vlan666
iface vlan666 inet manual


auto lo
iface lo inet loopback


iface eth0 inet manual


iface eth1 inet manual


iface eth2 inet manual


iface eth3 inet manual


iface eth4 inet manual


auto bond0
iface bond0 inet manual
        slaves eth3 eth4
        bond_miimon 100
        bond_mode 802.3ad


auto vmbr0
iface vmbr0 inet static
        address  1.1.1.232
        netmask  255.255.255.0
        gateway  1.1.1.1
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0


auto vmbr1
iface vmbr1 inet manual
        bridge_ports eth1
        bridge_stp off
        bridge_fd 0


auto vmbr2
iface vmbr2 inet manual
        bridge_ports eth2
        bridge_stp off
        bridge_fd 0


auto vmbr100
iface vmbr100 inet manual
        bridge_ports none
        bridge_stp off
        bridge_fd 0


auto vmbr666
iface vmbr666 inet manual
        bridge_ports vlan666
        bridge_stp off
        bridge_fd 0


auto vmbr4095
iface vmbr4095 inet manual
        bridge_ports bond0
        bridge_stp off
        bridge_fd 0


auto vmbr10
iface vmbr10 inet manual
        bridge_ports vlan10
        bridge_stp off
        bridge_fd 0


auto vmbr13
iface vmbr13 inet manual
        bridge_ports vlan13
        bridge_stp off
        bridge_fd 0


auto vmbr11
iface vmbr11 inet manual
        bridge_ports none
        bridge_stp off
        bridge_fd 0


auto vmbr999
iface vmbr999 inet manual
        bridge_ports none
        bridge_stp off
        bridge_fd 0


auto vmbr192
iface vmbr192 inet manual
        bridge_ports vlan192
        bridge_stp off
        bridge_fd 0

 

[lpallard]

Working now!!!!

And its faster than with my old dedicated pfsense box!



I will keep an eye on the server to see if its stable enough to use for production but so far its rock solid (touching wood..)

Is there any security implications that I should be aware of? I mean now that my UTM/router/filter is virtual, do I have to do anything else to prevent intruders from taking control of my server or one of its VM or pfsense?

Thanks pirateghost for your precious help!!

BTW my config file now looks like this, if you spot anything dangerous or missing, please let me know!

root@proxmox:~# cat /etc/network/interfaces
# network interface settings

# Physical devices

# Loopback NIC
auto lo
iface lo inet loopback
# Supermicro NIC1
auto eth0
iface eth0 inet manual
# Supermicro NIC2
auto eth1
iface eth1 inet manual
# Intel PRO/1000 VT Quad NIC1
auto eth2
iface eth2 inet manual
# Intel PRO/1000 VT Quad NIC2
auto eth3
iface eth3 inet manual
# Intel PRO/1000 VT Quad NIC3
auto eth4
iface eth4 inet manual
# Intel PRO/1000 VT Quad NIC4
auto eth5
iface eth5 inet manual

# Virtual devices

# Proxmox management NIC
auto vmbr0
iface vmbr0 inet static
address 192.168.0.2
netmask 255.255.255.0
bridge_ports eth0
bridge_stp off
bridge_fd 0
# WAN for pfSense VM
auto vmbr1
iface vmbr1 inet manual
bridge_ports eth1
bridge_stp off
bridge_fd 0
# LAN for pfSense VM
auto vmbr2
iface vmbr2 inet manual
bridge_ports eth2
bridge_stp off
bridge_fd 0
# DMZ for pfSense VM
auto vmbr3
iface vmbr3 inet manual
bridge_ports eth3
bridge_stp off
bridge_fd 0

 

[lpallard]

My excitement was short lived....

Ive been playing around with proxmox for the better part of the day now, and Ive come to numerous issues. You may recognize some of them, if so, please share how you overcame them..

1. When all is fine (VM can get an IP from the pfsense VM, physical hosts can also get an IP from pfsense VM), proxmox node cannot access the internet. So I cannot see updates or download ISO's from the proxmox node. Proxmox should go through the pfsense VM same as the other guests or physical hosts on my LAN. How to do that? I have tried adding the pfsense LAN IP (in reality the vmbr2 interface in proxmox) in the definition of vmbr0 (the management interface) but it created more mess than good.

2. I have found it quite annoyting to use java for the console viewer. I get at least two warnings, and often, my browser freezes for minutes and java crashes....
Fixed. I cleared Java's cache, rebooted my desktop machine. Working for now... SInce they came up with the recent Java platforms and started adding all these security features, things became hellish.... Oh well.

3. There are several messages in dmesg that worries me

kvm: 4051: cpu0 unhandled rdmsr: 0xc001100d
kvm: 4051: cpu0 unhandled rdmsr: 0xc0010112
kvm: 4051: cpu0 unhandled rdmsr: 0xc0010001
kvm: 4051: cpu1 unhandled rdmsr: 0xc001100d
kvm: 4051: cpu2 unhandled rdmsr: 0xc001100d
kvm: 4051: cpu3 unhandled rdmsr: 0xc001100d
kvm: 4051: cpu4 unhandled rdmsr: 0xc001100d
kvm: 4051: cpu5 unhandled rdmsr: 0xc001100d
kvm: 4051: cpu6 unhandled rdmsr: 0xc001100d
kvm: 4051: cpu7 unhandled rdmsr: 0xc001100d

EDIT: THreads on this forum seems to indicate this is due to KVM CPU virtualization (maybe a bug?) but also seems to be harmless. You guys agree with this?

4. After I had modified the vmbr0 to add the gateway IP, I rebooted the proxmox node, and then started the pfsense VM. It wouldnt start. Looking in the syslog, I saw these errors:

bridge 'vmbr2' does not exist
/var/lib/qemu-server/pve-bridge: could not launch network script
kvm: -netdev type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge: Device 'tap' could not be initialized
TASK ERROR: start failed: command '/usr/bin/kvm -id 100 -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/100.vnc,x509,password -pidfile /var/run/qemu-server/100.pid -daemonize -name pfSense -smp 'sockets=2,cores=1' -nodefaults -boot 'menu=on' -vga cirrus -cpu qemu64,+x2apic -k en-us -m 4096 -cpuunits 1000 -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -drive 'file=/var/lib/vz/images/100/vm-100-disk-1.qcow2,if=none,id=drive-ide0,format=qcow2,aio=native,cache=none' -device 'ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge' -device 'e1000,romfile=,mac=12:05:96D:21:8B,netdev=net0,bus=pci.0,addr=0x12,id=net0' -netdev 'type=tap,id=net1,ifname=tap100i1,script=/var/lib/qemu-server/pve-bridge' -device 'e1000,romfile=,mac=AA:3F:05:3C:AC:CE,netdev=net1,bus=pci.0,addr=0x13,id=net1' -netdev 'type=tap,id=net2,ifname=tap100i2,script=/var/lib/qemu-server/pve-bridge' -device 'e1000,romfile=,mac=EE:43:82:B1:F3:3E,netdev=net2,bus=pci.0,addr=0x14,id=net2'' failed: exit code 1

Looks like my bridges were gone. I opened the "/etc/network/interfaces" and saw that all eth devices were no longer auto starting. I added the "auto ethX" and rebooted. All was fine again.

Right now, Im progressing, but not in the right direction

 

[pirateghost]

You have something configured wrong if your node cannot reach the internet. Is your management VMBR in the same subnet as the rest of your devices? Does it have a gateway pointing it to the PFSense LAN IP? Do you have DNS servers configured so that the node can actually resolve DNS?

Your node should only have ONE IP address, and that address is the management node. You should not be modifying this in any way. A machine can only have ONE gateway, and your node gateway should be pointing to your PFSense LAN IP. Ignore the fact that it is a virtual router and think of it as physical hardware.

 

[lpallard]

You will soon realize that I have very little understanding and experience with networks...

Right now:

vmbr0 (eth0) is set to static 192.168.0.2 with subnet 255.255.255.0 in proxmox and gateway pointing to pfsense's LAN IP (192.168.0.100)
vmbr1 (eth1) is set to no IP or subnet in proxmox, and set to WAN in pfsense with DHCP.
vmbr2 (eth2) is set to no IP or subnet in proxmox, and set to LAN in pfsense with static 192.168.0.100 subnet 24 bit count

Is your management VMBR in the same subnet as the rest of your devices?

According to my description above, I would say YES

Does it have a gateway pointing it to the PFSense LAN IP?

Now Yes.

Do you have DNS servers configured so that the node can actually resolve DNS?

Yes I do. I have:
DNS server 1: Set to my pfsense LAN IP (192.168.0.100)
DNS server 2: 8.8.8.8 (Google's DNS server)
DNS server 3: 8.8.4.4 (Another Google DNS server)

 

[lpallard]

I just tried implementing the modifications discussed in the previous post (adding the pfsense LAN iP to the vmbr0 as gateway) and same problem:

lines "auto ethX" are being removed from the interface config file by proxmox which results in bridges not auto starting, and VM's not starting

bridge 'vmbr2' does not exist
/var/lib/qemu-server/pve-bridge: could not launch network script
kvm: -netdev type=tap,id=net0,ifname=tap101i0,script=/var/lib/qemu-server/pve-bridge: Device 'tap' could not be initialized
TASK ERROR: start failed: command '/usr/bin/kvm -id 101 -chardev 'socket,id=qmp,path=/var/run/qemu-server/101.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/101.vnc,x509,password -pidfile /var/run/qemu-server/101.pid -daemonize -name CentOS -smp 'sockets=2,cores=4' -nodefaults -boot 'menu=on' -vga cirrus -cpu kvm64,+lahf_lm,+x2apic,+sep -k en-us -m 60416 -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -device 'ahci,id=ahci0,multifunction=on,bus=pci.0,addr=0x7' -drive 'file=/dev/sdf,if=none,id=drive-sata4,aio=native,cache=none' -device 'ide-drive,bus=ahci0.4,drive=drive-sata4,id=sata4' -drive 'file=/dev/sdb,if=none,id=drive-sata0,aio=native,cache=none' -device 'ide-drive,bus=ahci0.0,drive=drive-sata0,id=sata0' -drive 'file=/dev/sde,if=none,id=drive-sata3,aio=native,cache=none' -device 'ide-drive,bus=ahci0.3,drive=drive-sata3,id=sata3' -drive 'file=/dev/sdd,if=none,id=drive-sata2,aio=native,cache=none' -device 'ide-drive,bus=ahci0.2,drive=drive-sata2,id=sata2' -drive 'file=/dev/sdc,if=none,id=drive-sata1,aio=native,cache=none' -device 'ide-drive,bus=ahci0.1,drive=drive-sata1,id=sata1' -drive 'file=/var/lib/vz/images/101/vm-101-disk-2.raw,if=none,id=drive-virtio0,format=raw,aio=native,cache=none' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=105' -drive 'file=/dev/sdg,if=none,id=drive-sata5,aio=native,cache=none' -device 'ide-drive,bus=ahci0.5,drive=drive-sata5,id=sata5' -netdev 'type=tap,id=net0,ifname=tap101i0,script=/var/lib/qemu-server/pve-bridge' -device 'e1000,romfile=,mac=66:94:7A3:4C:98,netdev=net0,bus=pci.0,addr=0x12,id=net0'' failed: exit code 1

If I remove the gateway IP of vmbr0 and manually readd the auto ethX lines in interface config file, then reboot the node, all is well... I swear, theres a bug somewhere or I really dont get it!