Setup Proxmox VE 3.2-4 with 3 NIC's and 3 public IPV4's

M

mrbrown997

New Member
Jul 16, 2013
11
0
1
Hey there,

i was wondering if someone could help on a question that chases me now for about a week.
I need to setup a server with proxmox. The server has got 3 NIC`s and 3 public IPV4-Addresses.
I do need to achieve the following setup:

Code: 
INTERNET ---NIC1(IPV4_1)---|                         |----VM1(Linux) [IPV4_1]
                           |                         |----VM2(Linux) [IPV4_2:80]
INTERNET ---NIC2(IPV4_2)------PROXMOX[IPV4_2:8006]---|
                           |                         |----VM3(Linux) [IPV4_2:8080]
INTERNET ---NIC2(IPV4_3)---|                         |----VM4(Windows) [IPV4_3]

I hope my intention is clear with this sheme.
How should my network.conf look like for this setup?
This is what it actually looks but somehow with this configuration I loose my complete connectivity to the internet.
Do I have to route at this point, when i'm using more than one NIC?

Code: 
auto lo
iface lo inet loopback

#Nic1
auto eth1
allow-hotplug eth0
iface eth0 inet static
 address 141.xxx.xxx.49
 netmask 255.255.254.0
 gateway 141.xxx.xxx.1
 dns-nameservers 141.xx.xx.3 141.xx.xx.4

#Nic2
auto eth1
iface eth0 inet static
 address 141.xxx.xxx.50
 netmask 255.255.254.0
 gateway 141.xxx.xxx.1
 dns-nameservers 141.xx.xx.3 141.xx.xx.4#Nic3

#Nic3
auto eth2
iface eth2 inet static  
address 141.xxx.xxx.51  
netmask 255.255.254.0  
gateway 141.xxx.xxx.1  
dns-nameservers 141.xx.xx.3 141.xx.xx.4

I would appreciate any help.
MrBrown
 

Attachments

  • network-model1.PNG
    network-model1.PNG
    6.7 KB · Views: 7
Hi,
for the single VMs with IPs on Nic1+3 (3? you named it also 2) simply use an bridge without IP and define the IP inside the VM.
You have also auto eth1 and iface eth0 twice!
like
Code: 
auto eth0
iface eth0 inet manual

auto eth1
iface eth1 inet manual

auto eth2
iface eth2 inet manual

auto vmbr0
iface vmbr0 inet static
        address  141.xxx.xxx.49
        netmask  255.255.254.0
        bridge_ports eth1
        bridge_stp off
        bridge_fd 0

auto vmbr50
iface vmbr50 inet static
        address  0.0.0.0
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0
        netmask  0.0.0.0

auto vmbr51
iface vmbr51 inet static
        address  0.0.0.0
        netmask  0.0.0.0
        bridge_ports eth2
        bridge_stp off
        bridge_fd 0
for the port-forwarding 80+8080 of your proxmox-ip, you need an reverseproxy like pound or nginx (with VMs in an internal network).

Udo
 
Hey Udo,
thanks for the moment for your reply!
Reading your name, i suppose your german.
If so, do you have skype and some time to answer some questions?
 
Hey,
thank for the help by now.
I managed now to setup proxmox and for now, proxmox works perfectly with one public IP on NIC 2.
I changed my Nic-Setup as following:

Code: 
INTERNET ---NIC1(IPV4_1)--- USED FOR KVM             |----VM1(Linux) [IPV4_2] ID100
                                                     |----VM2(Linux) [IPV4_2:80] ID101
INTERNET ---NIC2 eth1(IPV4_2)------PROXMOX[IPV4_2:8006]---|
                           |                         |----VM3(Linux) [IPV4_3:8080] ID102
INTERNET ---NIC2 eth2(IPV4_3)---|                    |----VM4(Windows) [IPV4_3] ID103

But this is not relevant for now.
My Interfaces-File looks als followed:

Code: 
# network interface settings
auto lo
iface lo inet loopback

auto eth1
iface eth1 inet static
        address  141.55.55.55
        netmask  255.255.254.0
        gateway  141.55.55.1
        broadcast  141.55.55.255
#       network 141.55.55.0
        up ip route add 141.55.55.0/24 dev eth1 src 141.55.55.55 table rt1
        up ip route add default via 141.55.55.1 dev eth1 table rt1
        up ip rule add from 141.55.55.55 table rt1
        #up ip route add default via 141.55.55.1 dev eth1
        up ip route add 141.55.55.0/24 dev eth1 src 141.55.55.55
        dns-nameservers 141.55.8.3 141.55.8.4

auto eth2
iface eth2 inet static
        address  141.55.55.56
        netmask  255.255.254.0
#       broadcast  141.55.55.255
#       network 141.55.55.0
        up ip route add 141.55.55.0/24 dev eth2 src 141.55.55.56 table rt2
        up ip route add default via 141.55.55.1 dev eth2 table rt2
        up ip rule add from 141.55.55.56 table rt2
        up ip route add default via 141.55.55.1 dev eth2
        up ip route add 141.55.55.0/24 dev eth2 src 141.55.55.56



iface eth0 inet manual

auto vmbr0
iface vmbr0 inet static
        address  10.0.0.1
        netmask  255.255.255.0
        broadcast  10.255.255.255
        bridge_ports none
        bridge_stp off
        bridge_fd 0

auto vmbr1
iface vmbr1 inet static
        address  192.168.0.1
        netmask  255.255.255.0
        broadcast  192.168.0.255
        bridge_ports none
        bridge_stp off
        bridge_fd 0

What I tried to achieve is a routing for the Gateway. As i use two nics with two public IP`s, in my opinion I should also have two gateways right?
The output of route -n:

Code: 
Ziel            Router          Genmask         Flags Metric Ref    Use Iface
10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0 vmbr0
141.55.55.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1
192.168.0.0     0.0.0.0         255.255.254.0   U     0      0        0 vmbr1
141.55.55.0     0.0.0.0         255.255.254.0   U     0      0        0 eth1
141.55.55.0     0.0.0.0         255.255.254.0   U     0      0        0 eth2
0.0.0.0         141.55.55.1     0.0.0.0         UG    0      0        0 eth1

For routing the IP's to the CT`s and VM`s i use shorewall, since it was the first one i found a good guide.
Here the files look als followed:

interfaces
Code: 
#ZONE   INTERFACE       BROADCAST       OPTIONS
net     eth1            detect          blacklist,nosmurfs
net     eth2            detect          blacklist,nosmurfs
dmz     venet0          detect          routeback
dmz     vmbr0           detect          routeback,bridge
dmz     vmbr1           detect          routeback,bridge
dmz     venet1          detect          routeback

masq
Code: 
#INTERFACE      SOURCE          ADDRESS         PROTO           PORT(S) IPSEC   MARK
eth1            10.0.0.0/24     141.55.55.55
eth2            192.168.0.0/24  141.55.55.56


#externe IP auf interne direkt binden
#+eth1          10.0.0.10       141.55.55.55

# LAST LINE -- DO NOT REMOVE

params
Code: 
# Shorewall version 4 - Params File
#
# /etc/shorewall/params
#
#       Assign any variables that you need here.
#
#       It is suggested that variable names begin with an upper case letter
#       to distinguish them from variables used internally within the
#       Shorewall programs
#
#       Example:
#
#               NET_IF=eth0
#               NET_BCAST=130.252.100.255
#               NET_OPTIONS=routefilter,norfc1918
#
#       Example (/etc/shorewall/interfaces record):
#
#               net     $NET_IF         $NET_BCAST      $NET_OPTIONS
#
#       The result will be the same as if the record had been written
#
#               net     eth0            130.252.100.255 routefilter,norfc1918
#
###############################################################################

#LAST LINE -- DO NOT REMOVE

policy
Code: 
#SOURCE DEST    POLICY          LOG     LIMIT:          CONNLIMIT:
#                               LEVEL   BURST           MASK

# From Firewall Policy
fw      fw      ACCEPT
fw      net     ACCEPT
fw      dmz     ACCEPT

# From DMZ Policy

dmz     dmz     ACCEPT
dmz     net     ACCEPT
dmz     fw      DROP            info

# From Net Policy
net     fw      DROP            info
net     dmz     DROP            info

#THE FOLLOWING POLICY MUST BE LAST
#
all     all     REJECT          info

rules
Code: 
#ACTION          SOURCE     DEST       PROTO   DEST        SOURCE     ORIGINAL    RATE

# Permit access to SSH
SSH/ACCEPT       net        fw         -       -            -          -          6/min:5

# Permit access to Proxmox Manager and Console
ACCEPT           net        fw         tcp      22,137:139,445,443,3389,8005,8006,8000,8002,8443,12001,10001,5900:5999
ACCEPT           all        fw         tcp      3128,3389,61000:61010
ACCEPT           all        dmz        tcp      3128,3389,61000:61010


# PING Rules
Ping/ACCEPT      all        all

#DNAT Rules

#CT ID100
#DNAT           net     dmz:10.0.0.100:21       tcp     10021
#DNAT           net     dmz:10.0.0.100:80       tcp     80
#DNAT           net     dmz:10.0.0.100:10443    tcp     443
DNAT            net     dmz:10.0.0.100:22       tcp     10022
DNAT            net     dmz:10.0.0.100:22       udp     10022
#DNAT           net     dmz:10.0.0.100:8443     tcp     8443
DNAT            net     dmz:10.0.0.100:445      tcp     445
DNAT            net     dmz:10.0.0.100:137      tcp     137
DNAT            net     dmz:10.0.0.100:138      tcp     138
DNAT            net     dmz:10.0.0.100:139      tcp     139
#DNAT           net     dmz:10.0.0.100:8082     tcp     8082
#DNAT           net     dmz:10.0.0.100:8000     tcp     8000

#VM ID103
#DNAT           net     dmz:10.0.0.103:22       tcp     10322
#DNAT           net     dmz:10.0.0.103:22       udp     10322
DNAT            net     dmz:10.0.0.103:3389     tcp     3389
DNAT            net     dmz:10.0.0.103:3389     udp     3389
DNAT            net     dmz:10.0.0.103:80       tcp     80
DNAT            net     dmz:10.0.0.103:20000    tcp     20000
DNAT            net     dmz:10.0.0.103:30000    tcp     30000
DNAT            net     dmz:10.0.0.103:40000    tcp     40000


#RULE FUER ZWEITE PUBLIC IP
#ACCEPT         net     dmz:141.55.55.55        tcp     21,22,25,53,80,110,143,443,995  -       141.55.55.55

# CT ID102
DNAT            net     dmz:192.168.0.102:22    tcp     10222
DNAT            net     dmz:192.168.0.102:22    udp     10222
DNAT            net     dmz:192.168.0.102:25    tcp     25
DNAT            net     dmz:192.168.0.102:80    tcp     10280
DNAT            net     dmz:192.168.0.102:110   tcp     110
DNAT            net     dmz:192.168.0.102:143   tcp     143
DNAT            net     dmz:192.168.0.102:443   tcp     443
DNAT            net     dmz:192.168.0.102:465   tcp     465
DNAT            net     dmz:192.168.0.102:587   tcp     587
DNAT            net     dmz:192.168.0.102:993   tcp     993
DNAT            net     dmz:192.168.0.102:995   tcp     995
DNAT            net     dmz:192.168.0.102:9071  tcp     9071
#DNAT           net     dmz:192.168.0.102:7071  tcp     7071

# LAST LINE -- DO NOT REMOVE

zones
Code: 
#ZONE   TYPE            OPTIONS         IN                      OUT
#                                       OPTIONS                 OPTIONS
fw      firewall
net     ipv4
dmz     ipv4

In the shorewall.conf file i just changed the value of DISABLE_IPV6 = NO and IP_FORWADING = ON.

My Problem now is, that all the CT`s running with vmbr1 are working and do have internet-access whereas all the CT`s and VM`s running with vmbr2 dont.
For example the interfaces-file of a ct running on vmbr1 looks like this:

Code: 
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
        address 10.0.0.100
        netmask 255.255.255.0
        gateway 10.0.0.1
        broadcast 10.0.0.255

The interfaces-file of a ct running on vmbr2 looks like this:

Code: 
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
        address 192.168.0.102
        netmask 255.255.255.0
        gateway 192.168.0.1
        broadcast 192.168.0.255


In Proxmox i assigned the vmbr as mentioned. So I really dont know where the problem is. Maybe its a route-problem or perhabs a shorewall-problem. Can anyone help me on that?
My intension was, that the IPV4 on eth1 is responsible for vmbr0 whereas the IPV4 on eth2 is responsible for vmbr1.
(All my public IP`s are edited so they are not the one I actually use)

Thanks for any help!
 
Last edited:
Hey Udo,
Thanks for the fast reply. Do you think the Routing regarding the gateway is wrong or is it actually a shorewall Problem?
I actually posted my route -n output. Could you have a look at it if its wrong?
 
mrbrown997 said:
Hey Udo,
Thanks for the fast reply. Do you think the Routing regarding the gateway is wrong or is it actually a shorewall Problem?
I actually posted my route -n output. Could you have a look at it if its wrong?
Click to expand...
Hi,
don't see the issue in a short time.

Try to find it with "tcpdump -i vmbrX host ip.ad.re.ss" and "tcpdump -i ethX host ip.ad.re.ss".
can only be an routing on wrong interface, or missing/wrong NAT.

Udo
 
okay,
manged this now by creating a bond interface with eth1 and eth2 as slaves. Additionally, to not loose the second public ip, i configured a alais called bond0:0.
This setup works now for two different vmbr (vmbr0, vmbr1) where the bond0 ist bound to vmbr0 and bond0:1 is bound to vmbr1 using masq-file of shorewall.

Thanks udo for your inputs.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back