Proper Routing and Setup - Passing Back out the Router!

Danielc1234

Active Member
Jan 10, 2010
82
0
26
Hi All.

I hope I can explain this the right way, cause it is a little confusing for me as well.
We have our IPS, which is comcast, that has a modem/router in which we have our main server (with our proxmox VMs on) directly connected to their device.
Right now on the proxmox machine we have two webservers with public ip's and local ips and two database servers using local ip's like 10.1.10.158 and 10.1.10.128.
For some reason it seems as if the local ip's of the databases are being passing back out to the modem/router and causing a lot of traffic. I spoke with comcast and they ran some tests. That is how I found out of this issue.
I was told that there was extensive traffic coming from the 10.1.10.158 (db server). Hence killing a lot of my bandwidth for internet connection.

So my question is, how would I want to resolve this issue? Can I do it via internal configurations or will I need to go out and buy a router to put behind the modem?

I'm not the best at networking, so any help would greatly be appreciated!!!

Daniel
 
Hi,
is it right that all your VMs (with two different networks) use one bridge?
If you use the network connection of the official-IPs only for this traffic there should be all fine, or?
You can simply add an dummy-bridge for inter-VM-traffic only (very fast) and use two nics (virtual) for all VMs which need connected to both networks (or use an virtual firewall/router for this).

Udo
 
Yes, I am thinking that I should be using the dummy-bridge for the inter-VM-Traffic. Is there any documents on how to properly set this up? The dbservers would only need to communicate with the webservers internally. This would be the perfect setup for me because of the fact of how fast it would make the db and web servers between each other.
Any help and insight would be greatly appreciated.
 
I found this proxmox wiki about networking and am having a time thinking which way I need to set this up.
http://pve.proxmox.com/wiki/Network_Model

So my question is, am I going to be able to do what I would like to do with the way I have it setup right now? And if so, what? Then if I can't set it up with what I have, what would I have to do to get it working?

This is on a live sites, so I have to make sure I don't screw anything up! Which is very easy for me to do. :)

And I have already created the five VMs and OpenVZ. And are set up as follows:

Code:
KVM 101 - webserver 1
    Model    Bridge         MAC address
    virtio    vmbr0        42:22:CB:76:18:08
    virtio    vmbr0        D2:02:E1:C7:98:FE
KVM 102 - db server 1 connects to webserver 1
    Model    Bridge         MAC address
    virtio    vmbr0        FE:AA:6B:80:24:A2
KVM 103 - webserver 2
    Model    Bridge         MAC address
    virtio    vmbr0        2A:D8:16:17:A7:63
KVM 104 - mail server
    Model    Bridge         MAC address
    virtio    vmbr0        52:6C:39:7A:23:8F
OpenVZ 105 - dbserver 2 connects to webserver 2
    IP Address: 10.1.10.128

And this is my current interfaces:

Code:
# network interface settings
auto lo
iface lo inet loopback


iface eth0 inet manual


iface eth1 inet static
    address  10.1.10.180
    netmask  255.255.255.0


auto vmbr0
iface vmbr0 inet static
    address  10.1.10.198
    netmask  255.255.255.0
    gateway  10.1.10.1
    bridge_ports eth0
    bridge_stp off
    bridge_fd 0
 
Here is my interface configuration

InterfaceActivePorts/SlavesAutostartIP AddressSubnet MaskGateway
tarrdown.png
eth1no10.1.10.180255.255.255.0
tarrdown.png
vmbr0yeseth010.1.10.198255.255.255.010.1.10.1
 
Here is my interface configuration

InterfaceActivePorts/SlavesAutostartIP AddressSubnet MaskGateway
tarrdown.png
eth1no10.1.10.180255.255.255.0
tarrdown.png
vmbr0yeseth010.1.10.198255.255.255.010.1.10.1
Hi,
you can do following.
create a script like this
( mkdir /usr/local/scripts )
/usr/local/scripts/generate_vmbr10.sh
Code:
modprobe -o dummy0 dummy
ifconfig dummy0 up
brctl addbr vmbr10
brctl addif vmbr10 dummy0
ifconfig vmbr10 up
chmod +x /usr/local/scripts/generate_vmbr10.sh

And add to your vmbr0-section in /etc/network/interfaces
Code:
        post-up /usr/local/scripts/generate_vmbr10.sh

Udo
 
Thanks for the response Udo!

Please excuse my ignorance, but I am pretty new to networking and wanted to gain some knowledge here.
So the above will accomplish my webserver and dbserver to communicate within the proxmox environment/machine? And do I really use the term 'dummy' in the code?

Also, would you mind just taking a moment to explain exactly what the above is actually doing? I want to learn as much as I can. I'm trying to follow the logic of how this is going to work on our system.

Thanks for your help!
 
Thanks for the response Udo!

Please excuse my ignorance, but I am pretty new to networking and wanted to gain some knowledge here.
So the above will accomplish my webserver and dbserver to communicate within the proxmox environment/machine? And do I really use the term 'dummy' in the code?

Also, would you mind just taking a moment to explain exactly what the above is actually doing? I want to learn as much as I can. I'm trying to follow the logic of how this is going to work on our system.

Thanks for your help!
Hi,
yes you need the dummy-module for the networkdevice. The first line create an networkdevice called dummy0 (you can add more dummy1, dummy2...)
Then a bridge (vmbr10) will created with this dummy-device.

If you now add an nic to your webserver on vmbr10 and change the nic of the db-server from vmbr0 to vmbr10 this both server can comunicate directly via this very fast networkconnection (you need of course the right ip-addresses on all nics).


Udo
 
And these configurations could be accomplished in the proxmox admin or would I have to set each configuration on each server's network?

And just to be clear, since I have webserver1 (10.1.10.168) communicating with dbserver1 (10.1.10.158) and webserver2 (10.1.10.178) communicating with dbserver2 (10.1.10.128), would I have to make another dummy for the web2 & db2 servers?

I apologize, for my lack of understanding, but I really want to make sure I fully understand the logic before I go in there and start making changes and screw something up?

Maybe you could give me an example using the ip's listed above. Cause I think I know what you are saying to do, but I want to make sure.

And if I had to make another dummy, would I do this?


Code:
[COLOR=#333333]create a script like this[/COLOR]
[COLOR=#333333]( mkdir /usr/local/scripts )[/COLOR]
[COLOR=#333333]/usr/local/scripts/generate_vmbr20.sh

[/COLOR]modprobe -o dummy0 dummy1
ifconfig dummy0 up
brctl addbr vmbr20
brctl addif vmbr20 dummy0
ifconfig vmbr20 up
[COLOR=#333333]
[/COLOR][COLOR=#333333]chmod +x /usr/local/scripts/generate_vmbr20.sh

[/COLOR][COLOR=#333333]And add to your vmbr0-section in /etc/network/interfaces

[/COLOR]post-up /usr/local/scripts/generate_vmbr20.sh
 
And these configurations could be accomplished in the proxmox admin or would I have to set each configuration on each server's network?

And just to be clear, since I have webserver1 (10.1.10.168) communicating with dbserver1 (10.1.10.158) and webserver2 (10.1.10.178) communicating with dbserver2 (10.1.10.128), would I have to make another dummy for the web2 & db2 servers?

I apologize, for my lack of understanding, but I really want to make sure I fully understand the logic before I go in there and start making changes and screw something up?

Maybe you could give me an example using the ip's listed above. Cause I think I know what you are saying to do, but I want to make sure.

And if I had to make another dummy, would I do this?


Code:
[COLOR=#333333]create a script like this[/COLOR]
[COLOR=#333333]( mkdir /usr/local/scripts )[/COLOR]
[COLOR=#333333]/usr/local/scripts/generate_vmbr20.sh

[/COLOR]modprobe -o dummy0 dummy1
ifconfig dummy0 up
brctl addbr vmbr20
brctl addif vmbr20 dummy0
ifconfig vmbr20 up
[COLOR=#333333]
[/COLOR][COLOR=#333333]chmod +x /usr/local/scripts/generate_vmbr20.sh

[/COLOR][COLOR=#333333]And add to your vmbr0-section in /etc/network/interfaces

[/COLOR]post-up /usr/local/scripts/generate_vmbr20.sh
Hi,
a second "host-only" bridge you only need if you want to separate the traffic (VMs of customer a should not be able to see traffic from customer b).
In your case all ips are in 10.1.10.0/24 so you can simply use vmbr10 for all traffic of this net (but limited to host-inside traffic).

For a second bridge without real nic you must something like this:
Code:
modprobe -o dummy1 dummy
ifconfig dummy1 up
brctl addbr vmbr20
brctl addif vmbr20 dummy1
ifconfig vmbr20 up
Udo
 
Okay, thanks for all your patience on this one!

You did not answer the question, if this can all be done within the proxmox control panel/admin? I watched the videos on this topic, but does not cover exactly what I am trying to do.
 
Okay, thanks for all your patience on this one!

You did not answer the question, if this can all be done within the proxmox control panel/admin? I watched the videos on this topic, but does not cover exactly what I am trying to do.
Hi,
in case of network-config i'm a little bit old-school: i configure all directly, so i don't know the gui very well, but you can't create dummy-nics with the gui. (bonding and vlan-tagging should work).

Udo
 
I made the changes in #6 and restarted network with /etc/init.d/networking restart, however it killed my sites. Do I need to restart the VMs as well? I had to comment out the command line you had me put in my interface.
However, now the new vmbr10 is now showing in the proxmox admin.

Can you please take a look at the images attached of my system and VM hardware and see if I can make the changes here?
 

Attachments

  • systemconfig.jpg
    systemconfig.jpg
    68.9 KB · Views: 9
  • VMhardware.jpg
    VMhardware.jpg
    89.7 KB · Views: 7
This is now what my interface looks like. Is this correct? This is the one that killed the sites, but had not restarted the VMs at this point


Code:
# network interface settings
auto lo
iface lo inet loopback


iface eth0 inet manual


iface eth1 inet static
    address  10.1.10.180
    netmask  255.255.255.0


auto vmbr0
iface vmbr0 inet static
    address  10.1.10.198
    netmask  255.255.255.0
    gateway  10.1.10.1
    bridge_ports eth0
    bridge_stp off
    bridge_fd 0
    post-up /usr/local/scripts/generate_vmbr10.sh
 
I tried what you suggested and it didn't work, but now my whole server and down and I cannot bring it back up. It will start, but it is like the whole configuration is wrong.

CAN YOU PLEASE HELP ME???
 
I tried what you suggested and it didn't work, but now my whole server and down and I cannot bring it back up. It will start, but it is like the whole configuration is wrong.

CAN YOU PLEASE HELP ME???
Hi,
your config is the same like before except that the script will be executed after bringing vmbr0 up.
BTW. your eth1 is in the same network than vmbr0?? strange!

An network-restart with running VMs are not the best idea i think so. For changes on the fly is an execution of the script enough.
On the other case an restart (but i prefer before to comment out your eth1 entry) of the whole server but not nessessary to activate an bridge for internal traffic.

What do you mean with whole server down? An VM or the host?
If you have troubel with an VM - post the config of them like "cat /etc/qemu-server/101.conf".

Udo
 
Here is the VMs from 1-4 but the 105 is a OpenVZ, so where would I find it? The OpenVZ is the second db server.

I made the changes last night and restarted. It didnt seem to work, so I had to re-add the virto vmbr0 and then it seemed as if it changed all the VMs configurations and IPs. It was a mess.
I got it functioning again, but I am almost 100% positive it is not setup the optimal way or possible the best way.

Any thoughts?

VM101-webserver1
Code:
name: Zicom-Webs
ide2: none,media=cdrom
vlan0: virtio=42:22:CB:76:18:08
ostype: l26
ide0: local:101/vm-101-disk-1.raw
memory: 5120
sockets: 1
boot: dc
freeze: 0
cpuunits: 1000
acpi: 1
kvm: 1
onboot: 1
cores: 3
vga: cirrus
bootdisk: ide0
vlan2: virtio=46:C7:4D:09:55:D3

VM102-dbserver1
Code:
name: Zicom-Databases
ide2: cdrom,media=cdrom
bootdisk: ide0
ostype: l26
ide0: local:102/vm-102-disk-1.raw
memory: 5120
sockets: 1
onboot: 1
cores: 3
boot: cd
freeze: 0
cpuunits: 1000
acpi: 1
kvm: 1
vlan0: virtio=52:50:32:C6:34:3C
vlan2: virtio=CA:A1:F7:8E:D1:5A

VM103
Code:
ostype: other
memory: 2048
sockets: 3
onboot: 1
name: webserver2
ide2: local:iso/ubuntu-10.10-server-amd64.iso,media=cdrom
vlan0: virtio=2A:D8:16:17:A7:63
bootdisk: scsi0
scsi0: local:103/vm-103-disk-1.raw
cores: 1
boot: cd
freeze: 0
cpuunits: 1000
acpi: 1
kvm: 1
vlan2: virtio=F6:56:4F:C2:59:C1

VM104
Code:
ostype: other
memory: 3072
sockets: 1
onboot: 1
name: mail
ide2: local:iso/ubuntu-10.04.1-server-amd64.iso,media=cdrom
vlan0: virtio=52:6C:39:7A:23:8F
bootdisk: scsi0
scsi0: local:104/vm-104-disk-1.raw
 
Here is the VMs from 1-4 but the 105 is a OpenVZ, so where would I find it? The OpenVZ is the second db server.

I made the changes last night and restarted. It didnt seem to work, so I had to re-add the virto vmbr0 and then it seemed as if it changed all the VMs configurations and IPs. It was a mess.
I got it functioning again, but I am almost 100% positive it is not setup the optimal way or possible the best way.

Any thoughts?

VM101-webserver1
Code:
name: Zicom-Webs
ide2: none,media=cdrom
vlan0: virtio=42:22:CB:76:18:08
ostype: l26
ide0: local:101/vm-101-disk-1.raw
memory: 5120
sockets: 1
boot: dc
freeze: 0
cpuunits: 1000
acpi: 1
kvm: 1
onboot: 1
cores: 3
vga: cirrus
bootdisk: ide0
vlan2: virtio=46:C7:4D:09:55:D3

VM102-dbserver1
Code:
name: Zicom-Databases
ide2: cdrom,media=cdrom
bootdisk: ide0
ostype: l26
ide0: local:102/vm-102-disk-1.raw
memory: 5120
sockets: 1
onboot: 1
cores: 3
boot: cd
freeze: 0
cpuunits: 1000
acpi: 1
kvm: 1
vlan0: virtio=52:50:32:C6:34:3C
vlan2: virtio=CA:A1:F7:8E:D1:5A
...
Hi,
why "vlan2: virtio=CA:A1:F7:8E:D1:5A"?? In your config is vlan0 (vmbr0) and due the script vlan10 (vmbr10). Where is vmbr2??

The config for openvz is below /etc/vz/conf/
To use two nics in openvz you can do something like this: (don't know if you need this)
Code:
NETIF="ifname=eth0,bridge=vmbr0,mac=CA:E3:43:25:79:66,host_ifname=veth104.0,host_mac=00:18:51:C5:C9:33;ifname=eth1,bridge=vmbr10,mac=CA:E3:43:25:79:55,host_ifname=veth104.1,host_mac=00:18:51:C5:C9:22"
Udo
 
A buddy of mine that works on our Servers had to come in and set it up a little differently. He had to set it up the way he knew how to accomplish what I was trying to do. It seems to be working fine, I just know with Proxmox certain things can be tricky and wanted to make sure that we are doing the setup the best way possible that will make proxmox the most optimized, etc.

Would you need to see any other config files on the proxmox to know a little more?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!