Homelab: can't ping VMs behind Router VM

Z

zulh2022

New Member
Dec 27, 2022
3
0
1
This is 4th try after getting no responses from Stack Exchange sites [1, 2, 3] and I've been fighting with this issue for ~2 weeks. I really hope someone can help me with this issue.

---

I have a setup like this (this diagram is also available here):
01-network.png

I'm running a single PVE host in my homelab with single NIC port. Inside that PVE host, I configured a Ubuntu VM to be a virtual router (I'll this as 'Router VM') for all my VMs. My home network is 192.168.1.0/24 and my VMs network is 10.0.0.0/24. The Router VM is running isc-dhcp-server so it can provide dynamic IP leases to my VMs.

Note: I'm aware of free router software like pfSense, OpenWRT & etc. But I want to take this opportunity to build the router from scratch so I can learn how everything works especially on routing/bridging/iptables side of things because that's my objective with this homelab - to learn deeper about virtualization and networking.

Network configurations in my Proxmox GUI:

Screenshot 2022-12-27 at 11.27.11 AM.png

Due to physical space issue, I have to place my PVE host in different place (instead of next to my home router) and connects it to a WiFi Repeater via a network cable. This WiFi Repeater is then connected to my home WiFi AP. When I'm connected to my WiFi AP and WiFi Repeater, I'm able to ping the PVE host and Router VM just fine.

Problem: when I ping from my laptop to my VMs while I'm connected to WiFi AP, the ping timed out. But it worked fine when I'm connected to the WiFi Repeater. I have tried two repeaters from different brands but the problem persist. I really wonder what's wrong here. I started to think if this is a misconfiguration issue on my Router VM/PVE host and not repeaters issue.

I tried adding static route/next hop in my home router and also in my laptop by following this guide. Unfortunately, that didn't help.

05-mwre-vs-zulh.png

Both PVE host and Router VM has IP forwarding enabled:

Code: 
$ cat /proc/sys/net/ipv4/ip_forward
1

And below are their configurations:

1. /etc/network/interfaces in PVE host

Code: 
auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
    address 192.168.1.5/24
    gateway 192.168.1.1
    bridge-ports eno1
    bridge-stp off
    bridge-fd 0

auto vmbr1
iface vmbr1 inet manual
    bridge-ports none
    bridge-stp off
    bridge-fd 0

2. ip a in PVE host

Code: 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000
    link/ether fc:4d:d4:f3:fe:d3 brd ff:ff:ff:ff:ff:ff
    altname enp0s25
3: wlp2s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether ba:65:34:a3:09:60 brd ff:ff:ff:ff:ff:ff permaddr 28:c2:dd:a5:46:53
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fc:4d:d4:f3:fe:d3 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.5/24 scope global vmbr0
       valid_lft forever preferred_lft forever
5: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 8e:7e:21:7e:f9:e1 brd ff:ff:ff:ff:ff:ff
6: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc pfifo_fast state UNKNOWN group default qlen 500
    link/none
    inet 100.111.63.28/32 scope global tailscale0
       valid_lft forever preferred_lft forever
7: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether 62:27:fd:32:26:0c brd ff:ff:ff:ff:ff:ff
8: tap100i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UNKNOWN group default qlen 1000
    link/ether 62:c8:a9:96:dc:7e brd ff:ff:ff:ff:ff:ff
9: tap101i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UNKNOWN group default qlen 1000
    link/ether fa:35:bb:a9:7d:b8 brd ff:ff:ff:ff:ff:ff
10: tap102i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UNKNOWN group default qlen 1000
    link/ether 46:9c:a4:fd:81:27 brd ff:ff:ff:ff:ff:ff
11: tap103i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UNKNOWN group default qlen 1000
    link/ether 12:90:ee:36:27:7a brd ff:ff:ff:ff:ff:ff


3. iptables -S in PVE host

Code: 
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N ts-forward
-N ts-input
-A INPUT -j ts-input
-A FORWARD -j ts-forward
-A ts-forward -i tailscale0 -j MARK --set-xmark 0x40000/0xff0000
-A ts-forward -m mark --mark 0x40000/0xff0000 -j ACCEPT
-A ts-forward -s 100.64.0.0/10 -o tailscale0 -j DROP
-A ts-forward -o tailscale0 -j ACCEPT
-A ts-input -s 100.111.63.28/32 -i lo -j ACCEPT
-A ts-input -s 100.115.92.0/23 ! -i tailscale0 -j RETURN
-A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j DROP

4. iptables -S -t nat in PVE host

Code: 
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N ts-postrouting
-A POSTROUTING -j ts-postrouting
-A ts-postrouting -m mark --mark 0x40000/0xff0000 -j MASQUERADE

Note: I didn't have any custom iptables rules for Router VM in this PVE host. All of these were added by Tailscale when I configured it for PVE host remote access. I believe Tailscale isn't the cause/issue here because my problem was already there before I installed Tailscale.

5. ip a in Router VM

Code: 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 2a:06:12:a9:18:98 brd ff:ff:ff:ff:ff:ff
    altname enp0s18
    inet 192.168.1.6/24 brd 192.168.1.255 scope global ens18
       valid_lft forever preferred_lft forever
    inet6 fe80::2806:12ff:fea9:1898/64 scope link
       valid_lft forever preferred_lft forever
3: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000
    link/ether ca:bc:10:17:e0:2f brd ff:ff:ff:ff:ff:ff
    altname enp0s19
4: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether da:56:23:09:09:72 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/24 brd 10.0.0.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::d856:23ff:fe09:972/64 scope link
       valid_lft forever preferred_lft forever

6. /etc/netplan/00-installer-config.yaml in Router VM

Code: 
network:
  version: 2
  renderer: networkd
  ethernets:
    ens18:
     dhcp4: no
     addresses: [192.168.1.6/24]
     gateway4: 192.168.1.1
     nameservers:
       addresses: [8.8.8.8,8.8.4.4]

7. /etc/systemd/network/uplink.network in Router VM

Code: 
[Match]
Name=ens19

[Network]
Bridge=br0

8. /etc/systemd/network/br0.netdev in Router VM

Code: 
[NetDev]
Name=br0
Kind=bridge

9. /etc/systemd/network/br0.network in Router VM

Code: 
[Match]
Name=br0

[Network]
DNS=8.8.8.8
Address=10.0.0.1/24
Gateway=10.0.0.1

10. iptables -S in Router VM

Code: 
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -s 127.0.0.0/8 -d 127.0.0.0/8 -i lo -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 0 -j ACCEPT
-A INPUT -m state --state ESTABLISHED -j ACCEPT
-A INPUT -p udp -m udp --dport 33434:33523 -j REJECT --reject-with icmp-port-unreachable
-A INPUT -i br0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i br0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i br0 -p udp -m udp --dport 67:68 -j ACCEPT
-A INPUT -i ens18 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -j DROP
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 10.0.0.0/24 -d 192.168.1.0/24 -i br0 -o ens18 -j DROP
-A FORWARD -i br0 -o ens18 -j ACCEPT
-A FORWARD -d 10.0.0.0/24 -p icmp -j ACCEPT
-A FORWARD -d 10.0.0.0/24 -p tcp -m tcp --dport 80 -j ACCEPT
-A FORWARD -d 10.0.0.0/24 -p tcp -m tcp --dport 22 -j ACCEPT
-A FORWARD -j DROP

11. iptables -S -t nat in Router VM

Code: 
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A POSTROUTING -o ens18 -j MASQUERADE

12. /etc/dhcp/dhcpd.conf in Router VM

Code: 
option domain-name "example.org";
option domain-name-servers ns1.example.org, ns2.example.org;
default-lease-time 600;
max-lease-time 7200;
ddns-update-style none;
subnet 10.0.0.0 netmask 255.255.255.0 {
    range 10.0.0.35 10.0.0.39;
    option routers 10.0.0.1;
    option domain-name-servers 8.8.8.8;
    option broadcast-address 10.0.0.255;
        host ubuntu-desktop {
          hardware ethernet 5A:5B:74:01:69:34;
          fixed-address 10.0.0.3;
        }
}

What should I add/update so I can ping the VMs when I'm connected to my WiFi AP (not WiFi Repeater)? Is this iptables issue or repeaters issue in general?
 
You read this wiki article about wifi, where VMs can't reach any hosts outside of the PVE host, as wifi APs usually will drop all packets that doesn't come from a single specific MAC, so PVE needs to be configured to use NAT, so the wifi AP thinks all packets come from the PVE host itself and not from one of the VMs with different MACs?: https://pve.proxmox.com/wiki/WLAN

No sure if this is the same problem when using your Wifi repeater, or if your Wifi Repeater already is doing the NAT.
 
Dunuin said:
You read this wiki article about wifi, where VMs can't reach any hosts outside of the PVE host, as wifi APs usually will drop all packets that doesn't come from a single specific MAC, so PVE needs to be configured to use NAT, so the wifi AP thinks all packets come from the PVE host itself and not from one of the VMs with different MACs?: https://pve.proxmox.com/wiki/WLAN

No sure if this is the same problem when using your Wifi repeater, or if your Wifi Repeater already is doing the NAT.
Click to expand...
My VMs can reach internet. They also can reach my devices in 192.168.1.0/24 network when I remove -A FORWARD -s 10.0.0.0/24 -d 192.168.1.0/24 -i br0 -o ens18 -j DROP from the Router VM. I believe NAT is working in VMs network, right?

When I'm connected to the WiFi Repeater, I can ping the PVE host and Router just fine. I can also ping/reach internet. So, NAT is working here too, right?
 
Just to add before someone says "just use VLANs". My networking devices (router, WiFi AP, WiFi Repeater) don't run on VLANs.
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back