VM inaccessible to internet via routed config

[pri07]

Hi, I am new to PROXMOX and definitely learning through issues. The new issue I encountered is that my VM (Win10) cant access the internet. My PROXMOX host is able to reach the internet, but not the VMs. I've followed instructions per https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysadmin_network_routed.

Below is my proxmox config -
cat /etc/network/interface

auto lo
iface lo inet loopback
auto enp31s0
iface enp31s0 inet static
        address 10.101.0.4/24
        gateway 10.101.0.1

iface enp41s0f0 inet manual

iface enp41s0f1 inet manual

iface wlp30s0 inet manual

iface enp42s0f0 inet manual

iface enp42s0f1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 10.0.0.1/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o enp31s0 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/24' -o enp31s0 -j MASQUERADE

auto vmbr1
iface vmbr1 inet static
        address 10.101.1.4/24
        bridge-ports enp41s0f0
        bridge-stp off
        bridge-fd 0

auto vmbr2
iface vmbr2 inet static
        address 10.0.0.1/24
        bridge-ports enp41s0f1
        bridge-stp off
        bridge-fd 0

source /etc/network/interfaces.d/*

Windows ip config - using 'vmbr0'

Ethernet Adapter ...: (Simplified)
 Description: Red Hat VirtualIO Ethernet Adapter
 IPV4 address: 10.0.0.2
 Subnet Mask: 255.0.0.0
 Default Gateway: 10.0.0.1
 DNS: 8.8.8.8, 8.8.4.4

On the Win10 box (VM) I have changed the gateway to 10.101.0.4 and still didn't work. My host is able to access the internet, but not the Win10 (VM). Win 10 is able to ping the 10.101.0.4, but can't ping the gateway.

 

[Pifouney]

I think you've tried too hard configuration so fastly.

1) Why use a interface not bridged for your PVE?

2) WHY WHY another interface, breidged, on the SAME network ? What's your goal with this ?

3) Begin by place good your post-up rules: ADD them only on your vmbr2

4) Review yours network primary concepts: You cannot have same network and/or IP on two hadware stuff like your definition. This can be done, but only with BOND

 

[pri07]

I think you've tried too hard configuration so fastly.

1) Why use a interface not bridged for your PVE?

2) WHY WHY another interface, breidged, on the SAME network ? What's your goal with this ?

3) Begin by place good your post-up rules: ADD them only on your vmbr2

4) Review yours network primary concepts: You cannot have same network and/or IP on two hadware stuff like your definition. This can be done, but only with BOND

Thanks for your reply

1. I wanted to isolate the VMs by creating a VM network in PVE. Just like here - https://pve.proxmox.com/pve-docs/images/default-network-setup-routed.svg (BTW, I am able to get the bridge VMBR0 working but the VMs are on the same network. Eventually I want to install OPENSense to be the router, vmbr0 is right now a lab environment and get the routed config working on it.

2. Good Catch, I have changed vmbr2 to 10.0.2.1/24 - there is nothing connected to VMBR2 yet.

 

[Pifouney]

Thanks for your reply

1. I wanted to isolate the VMs by creating a VM network in PVE. Just like here - https://pve.proxmox.com/pve-docs/images/default-network-setup-routed.svg (BTW, I am able to get the bridge VMBR0 working but the VMs are on the same network. Eventually I want to install OPENSense to be the router, vmbr0 is right now a lab environment and get the routed config working on it.

2. Good Catch, I have changed vmbr2 to 10.0.2.1/24 - there is nothing connected to VMBR2 yet.

Don't forguet to update yours masquerading rules for the updated network name.

For your routing problem:
1 Deactivate your enp31s0 gateway. Make it on your vmbr1 declaration.
2 Apply configuration
3 Check routing table (ip -c r)
4 modify your iptables rule for masquerade to vmbr1 interface.

What is going on in this configuration ?

 

[pri07]

Don't forguet to update yours masquerading rules for the updated network name.

For your routing problem:
1 Deactivate your enp31s0 gateway. Make it on your vmbr1 declaration.
2 Apply configuration
3 Check routing table (ip -c r)
4 modify your iptables rule for masquerade to vmbr1 interface.

What is going on in this configuration ?

to simplify things this is what I have now (see below) and still not able to reach the internet from Win10(VM). I had removed the Masquerade line and added it back in since it didn't work. I am following every step from the Routed Config. I don't know what I am doing wrong. I also deactivated the gateway on enp31s0 and moved it to vmbr0 like you suggested in #1 above and still couldn't ping so I reverted my to my original config and disabled vmbr1 and vmbr2. Every time I make a change to the interface I run systemctl restart networking on pve host and reboot my Win10(VM)

My Win 10 (VM) is statically configured to 10.0.0.2/8 GTWY 10.101.0.1 (I also switched the GTWY to 10.0.0.1 and still didn't work) and DNS = 8.8.8.8.

Code:
auto lo
iface lo inet loopback
auto enp31s0
iface enp31s0 inet static
        address 10.101.0.4/24
        gateway 10.101.0.1

iface enp41s0f0 inet manual

iface enp41s0f1 inet manual

iface wlp30s0 inet manual

iface enp42s0f0 inet manual

iface enp42s0f1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 10.0.0.1/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o enp31s0 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/24' -o enp31s0 -j MASQUERADE

  source /etc/network/interfaces.d/*

 

[pri07]

I was able to fix the issue (VM can get to the internet on routed config) - not sure if this is best practice but below is my config. However, I am not able to access the VMs from 10.101.0.1/24 network. Is there a firewall rule that can be set to allow access from 10.101.0.1/24 address space?

The issue was I didn't have this statement on my vmbr0 post-up echo 1 > /proc/sys/net/ipv4/conf/vmbr0/proxy_arp

Win10(VM) - 10.0.0.2/24 GTWY 10.0.0.1

PVE

auto lo
iface lo inet loopback

auto enp31s0
iface enp31s0 inet static
        address 10.101.0.4/24
        gateway 10.101.0.1

iface enp41s0f0 inet manual

iface enp41s0f1 inet manual

iface wlp30s0 inet manual

iface enp42s0f0 inet manual

iface enp42s0f1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 10.0.0.1/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o enp31s0 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/24' -o enp31s0 -j MASQUERADE
        post-up echo 1 > /proc/sys/net/ipv4/conf/vmbr0/proxy_arp

auto vmbr1
iface vmbr1 inet static
        address 10.101.1.4/24
        bridge-ports enp41s0f0
        bridge-stp off
        bridge-fd 0


source /etc/network/interfaces.d/*

 

[Pifouney]

"
post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o enp31s0 -j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/24' -o enp31s0 -j MASQUERADE "

Try with converting yours iptables rules into a nftables rules. Proxmox doesn't use iptables natively. They can break each other (iptabls and nftables)
if you installed iptables, remove it, convert to nftables yours rules (more easyly to do a nfttables.conf in /etc/nftables) then, reboot your host for clear all caches and unwanted loaded modules.