Hi Guys,
first of all: Thanks for your great work on Proxmox, it's really impressive. I'm testing a Cluster of 3 Nodes running Proxmox 3.0 and a SAN consisting of 2 Nodes running Open-E DSS7. A few days ago I ran into a problem regarding the Networkinterfaces, but soon came to a working solution. Now that I want to use some of those IP's differently, everything comes crashing down. But first let me give you a quick introduction:
Proxmox-Nodes 1-3 have 4 NICs:
eth0: IP: 192.168.0.1-3 -> configuration network
eth1: IP: 192.168.1.1-3 -> DMZ
eth2: IP: 192.168.100.1-3 -> SAN (10GBit/s)
eth3: IP: 192.168.101.1-3 -> SAN (10GBit/s)
Storage Nodes have 8 Nics:
eth0: IP: 192.168.0.100-101
eth1:
eth2: bond0
eth3: bond0
eth4: bond1
eth5: bond1
eth6: bond2
eth7: bond2
bond0: 192.168.100.100-101 -> SAN, Heartbeat...
bond1: 192.168.101.100-101 -> SAN, heartbeat...
bond2: 192.168.5.100-101 -> Direct connected, only for replication
virtual IPs for ISCSI
bond0: 192.168.21.100 -> Target_A
bond1: 192.168.31.100 -> Target_B
That's the complete networksetup till now. iSCSI targets are on the virtual-IP Setup which works fine on the storageserver side, but not on Proxmox. I have to set up the Proxmox Nodes each with
- eth2 binding on 192.168.100.1-3 and 192.168.21.1-3 and
- eth3 on 192.168.101.1-3 and 192.168.31.1-3
I've read a lot, but nothing seems to fit. My conclusions until now:
1. One can not bind simply two or more vmbr's on one NIC with every vmbr being in a different Subnet.
2. One could bind two or more vmbrs to one Nic using VLANs (like bridge_port eth0.5) But this won't work for the host itself. Tried that, both networks are unreachable
3 One can set eth2:0 to 192.168.1001 and eth2:1 to 192.168.21.1, but this will not work, if both interfaces are configured like this:
iface eth2 inet manual
auto eth2:0
iface eth2:0 inet static
address 192.168.100.1
iface eth2:1 inet static
address 192.168.21.1
In my tests, only one address was working. I tried to configure only one, than acivate the other one manually: worked like a charm. But only, if no entry for the second interface was in the /etc/network/interfaces. So i tried the following:
iface eth2 inet manual
auto eth2:0
iface eth2:0 inet static
address 192.168.100.1
postup ifconfig eth2:1 address 192.168.21.1 netmask 255.255.255.0 up
And thats the solution, that worked. The downside is, that I can't connect my Monitoring VM to my SAN-Network, where I have SNMP enabled and can do the monitoring of the cluster. I now, that, if I want to use a Subnet from the host and the vm, my only chance is a VMBR on the Host with an IP-Adress of the subnet and a NIC in my VM bridged to this vmbr. But how to do this? How can I have two different vmbrs on one physical NIC with two IP-Adresses? I thought of the following network config..
[snip]
iface eth2 inet manual
auto vmbr1
iface vmbr1 inet static
address 192.168.100.1
netmask 255.255.255.0
bridge_ports eth3.100
bridge_stp off
bridge_fd 0
auto vmbr2
iface vmbr2 inet static
address 192.168.21.1
netmask 255.255.255.0
bridge_ports eth3.21
bridge_stp off
bridge_fd 0
[snap]
Sure enough, that won't work.. Destination Host Unreachable. Do I have to set the vlans on the other nodes or switches too? Or should it work out of the box with every client on the two subnets, without any other config?
I hope you can help me. If I forgot something, please let me now. Any Help is appreciated.
Thanks!
first of all: Thanks for your great work on Proxmox, it's really impressive. I'm testing a Cluster of 3 Nodes running Proxmox 3.0 and a SAN consisting of 2 Nodes running Open-E DSS7. A few days ago I ran into a problem regarding the Networkinterfaces, but soon came to a working solution. Now that I want to use some of those IP's differently, everything comes crashing down. But first let me give you a quick introduction:
Proxmox-Nodes 1-3 have 4 NICs:
eth0: IP: 192.168.0.1-3 -> configuration network
eth1: IP: 192.168.1.1-3 -> DMZ
eth2: IP: 192.168.100.1-3 -> SAN (10GBit/s)
eth3: IP: 192.168.101.1-3 -> SAN (10GBit/s)
Storage Nodes have 8 Nics:
eth0: IP: 192.168.0.100-101
eth1:
eth2: bond0
eth3: bond0
eth4: bond1
eth5: bond1
eth6: bond2
eth7: bond2
bond0: 192.168.100.100-101 -> SAN, Heartbeat...
bond1: 192.168.101.100-101 -> SAN, heartbeat...
bond2: 192.168.5.100-101 -> Direct connected, only for replication
virtual IPs for ISCSI
bond0: 192.168.21.100 -> Target_A
bond1: 192.168.31.100 -> Target_B
That's the complete networksetup till now. iSCSI targets are on the virtual-IP Setup which works fine on the storageserver side, but not on Proxmox. I have to set up the Proxmox Nodes each with
- eth2 binding on 192.168.100.1-3 and 192.168.21.1-3 and
- eth3 on 192.168.101.1-3 and 192.168.31.1-3
I've read a lot, but nothing seems to fit. My conclusions until now:
1. One can not bind simply two or more vmbr's on one NIC with every vmbr being in a different Subnet.
2. One could bind two or more vmbrs to one Nic using VLANs (like bridge_port eth0.5) But this won't work for the host itself. Tried that, both networks are unreachable
3 One can set eth2:0 to 192.168.1001 and eth2:1 to 192.168.21.1, but this will not work, if both interfaces are configured like this:
iface eth2 inet manual
auto eth2:0
iface eth2:0 inet static
address 192.168.100.1
iface eth2:1 inet static
address 192.168.21.1
In my tests, only one address was working. I tried to configure only one, than acivate the other one manually: worked like a charm. But only, if no entry for the second interface was in the /etc/network/interfaces. So i tried the following:
iface eth2 inet manual
auto eth2:0
iface eth2:0 inet static
address 192.168.100.1
postup ifconfig eth2:1 address 192.168.21.1 netmask 255.255.255.0 up
And thats the solution, that worked. The downside is, that I can't connect my Monitoring VM to my SAN-Network, where I have SNMP enabled and can do the monitoring of the cluster. I now, that, if I want to use a Subnet from the host and the vm, my only chance is a VMBR on the Host with an IP-Adress of the subnet and a NIC in my VM bridged to this vmbr. But how to do this? How can I have two different vmbrs on one physical NIC with two IP-Adresses? I thought of the following network config..
[snip]
iface eth2 inet manual
auto vmbr1
iface vmbr1 inet static
address 192.168.100.1
netmask 255.255.255.0
bridge_ports eth3.100
bridge_stp off
bridge_fd 0
auto vmbr2
iface vmbr2 inet static
address 192.168.21.1
netmask 255.255.255.0
bridge_ports eth3.21
bridge_stp off
bridge_fd 0
[snap]
Sure enough, that won't work.. Destination Host Unreachable. Do I have to set the vlans on the other nodes or switches too? Or should it work out of the box with every client on the two subnets, without any other config?
I hope you can help me. If I forgot something, please let me now. Any Help is appreciated.
Thanks!
Last edited: