Setting up a network between 3 servers

[qx13]

Hello everyone, I need help setting up a network between three proxmox servers. I'm new to this.
I have three servers. Each of them has two network cards, the first looks at the Internet, and the second connects the other servers via a switch.
I need the vm and ct to be visible to each other from the second network interface where only the proxmox servers are connected

 

[Chris]

Hi,
you can create a linux bridge on each of the Proxmox VE hosts and connect the NIC intended for communication between the hosts to that bridge. By attaching the VMs/CTs to that bridge they are then connected to the same network. Please have a look at the docs for an example of a bridged network configuration [0] (the example is of course without your internet facing additional network interface).

[0] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_default_configuration_using_a_bridge

 

[qx13]

Hi,
you can create a linux bridge on each of the Proxmox VE hosts and connect the NIC intended for communication between the hosts to that bridge. By attaching the VMs/CTs to that bridge they are then connected to the same network. Please have a look at the docs for an example of a bridged network configuration [0] (the example is of course without your internet facing additional network interface).

[0] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_default_configuration_using_a_bridge

I so so understood in this example there has to be a DHCP ? But I have a switch, not a router

 

[Chris]

I so so understood in this example there has to be a DHCP ? But I have a switch, not a router

No, you don't neccessarily need a DHCP server. You can manually assign static IP addresses from the same subnet, and the VMs/CTs will be able to communicate. You can of course also setup a VM which provides DHCP leases if you don't want to manually assign the addresses.

Same is true for the gateway, which you don't neccessarily need if you don't want the VMs/CTs to reach other networks or the internet. If the guests have no route to send the packets, they will drop them.

 

[qx13]

No, you don't neccessarily need a DHCP server. You can manually assign static IP addresses from the same subnet, and the VMs/CTs will be able to communicate. You can of course also setup a VM which provides DHCP leases if you don't want to manually assign the addresses.

Same is true for the gateway, which you don't neccessarily need if you don't want the VMs/CTs to reach other networks or the internet. If the guests have no route to send the packets, they will drop them.

it still doesn't work

auto lo
iface lo inet loopback

iface lo inet6 loopback

auto enp8s0
iface enp8s0 inet static
address XX.XX.XX.XX
gateway XX.XX.XX.XX
up route add -net XX.XX.XX.XX netmask XX.XX.XX.XX gw XX.XX.XX.XX dev enp8s0

auto enp1s0
iface enp1s0 inet manual

auto vmbr0
iface vmbr0 inet static
address 192.168.100.1/24
bridge-ports none
bridge-stp off
bridge-fd 0

auto vmbr1
iface vmbr1 inet static
address 192.168.10.2/24
bridge-ports enp1s0
bridge-stp off
bridge-fd 0

post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
post-up   iptables -t nat -A POSTROUTING -s '192.168.100.0/24' -o enp8s0 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '192.168.100.0/24' -o enp8s0 -j MASQUERADE

 

[Chris]

bridge-ports eno1

Do you actually have an interface named eno1? Check the output of ip link.

You must adapt the example in the docs to your setup and reload the interfaces with ifreload -a after performing changes.

 

[qx13]

Do you actually have an interface named eno1? Check the output of ip link.

You must adapt the example in the docs to your setup and reload the interfaces with ifreload -a after performing changes.

I fixed it, it still doesn't work

bridge-ports enp1s0

 

[Chris]

I fixed it, it still doesn't work

bridge-ports enp1s0

You will have to be more specific for me to follow: What does not work, what are you testing exactly and what is the result?

 

[qx13]

You will have to be more specific for me to follow: What does not work, what are you testing exactly and what is the result?

I manually assign ip addresses to the vm of the first machine on the first server 192.168.10.5, on the second machine on the vm I assign 192.168.10.105 and they do not see each other

 

[qx13]

I can ping bridge but there is no vm

 

[Chris]

Do you have possible firewall rules interfering with the traffic? You can check with the output of iptables-save. Also, ip neigh should give you some more information about your neighbours in the arp table.

 

[qx13]

Do you have possible firewall rules interfering with the traffic? You can check with the output of iptables-save. Also, ip neigh should give you some more information about your neighbours in the arp table.

it turns out that all ct see each other, vm see all ct, but vm do not see each other and all ct do not see vm. All vm is windows server

 

[Chris]

it turns out that all ct see each other, vm see all ct, but vm do not see each other and all ct do not see vm. All vm is windows server

So maybe the Windows Firewall inside the VMs is blocking traffic? You still did not specify how you are testing connectivity.

 

[qx13]

iptables-save

# Generated by iptables-save v1.8.9 on Tue Aug 29 14:14:14 2023
*raw
:PREROUTING ACCEPT [1544734:231095980]
:OUTPUT ACCEPT [1078084:344718122]
COMMIT
# Completed on Tue Aug 29 14:14:14 2023
# Generated by iptables-save v1.8.9 on Tue Aug 29 14:14:14 2023
*filter
:INPUT ACCEPT [1438137:184385499]
:FORWARD ACCEPT [106303:46690660]
:OUTPUT ACCEPT [1078084:344718122]
COMMIT
# Completed on Tue Aug 29 14:14:14 2023
# Generated by iptables-save v1.8.9 on Tue Aug 29 14:14:14 2023
*nat
:PREROUTING ACCEPT [242682:12492852]
:INPUT ACCEPT [239071:12260128]
:OUTPUT ACCEPT [1276:77569]
:POSTROUTING ACCEPT [1287:78133]
-A PREROUTING -i enp8s0 -p tcp -m tcp --dport 13000 -j DNAT --to-destination 192.168.100.2:3389
-A POSTROUTING -s 192.168.100.0/24 -o enp8s0 -j MASQUERADE
-A POSTROUTING -s 192.168.100.0/24 -o enp8s0 -j MASQUERADE
-A POSTROUTING -s 192.168.100.0/24 -o enp8s0 -j MASQUERADE
-A POSTROUTING -s 192.168.100.0/24 -o enp8s0 -j MASQUERADE
COMMIT
# Completed on Tue Aug 29 14:14:14 2023

ip neigh

xxx.xxx.xxx.xxx dev enp8s0 lladdr xx:xx:xx:xx:xx:xx REACHABLE
192.168.1.4 dev vmbr1 lladdr 8e:13:be:a3:3d:13 STALE
192.168.1.106 dev vmbr1 lladdr a2:97:e4:73:c2:d6 STALE
192.168.100.2 dev vmbr0 lladdr 6e:47:8a:c5:66:ac STALE
192.168.1.102 dev vmbr1 lladdr b2:29:7a:5c:b5:e8 STALE
192.168.1.6 dev vmbr1 lladdr fa:0f:85:55:2a:90 STALE
192.168.1.105 dev vmbr1 lladdr b2:29:7a:5c:b5:e8 STALE
192.168.1.100 dev vmbr1 lladdr 68:05:ca:10:63:15 STALE
fe80::1 dev enp8s0 lladdr xx:xx:xx:xx:xx:xx router STALE
fe80::a097:e4ff:fe73:c2d6 dev vmbr1 lladdr a2:97:e4:73:c2:d6 STALE
fe80::788a:14ff:fec6:37f3 dev vmbr0 lladdr 7a:8a:14:c6:37:f3 STALE
fe80::8d26:7f49:d64:8a1c dev vmbr1 lladdr 8e:13:be:a3:3d:13 STALE
fe80::f80f:85ff:fe55:2a90 dev vmbr1 lladdr fa:0f:85:55:2a:90 STALE

 

[qx13]

So maybe the Windows Firewall inside the VMs is blocking traffic? You still did not specify how you are testing connectivity.

windows is installed from the original image, the settings were not changed after installation. only the rdp port access is open