[TUTORIAL] ZeroTier + Proxmox PVE

KatyComputer · Oct 11, 2020

ZeroTier + Proxmox

We have been using ZeroTier over a year, when it came time to use it with Proxmox, I wasted hours due to my desire to over-complicate things. This four step process will let you access your containers and VMs remotely via the ZeroTier D-WAN / VPN

My goal is to configure several containers and vms on 10.101.101.0/24 on vmbr2

To do so:
1. Login to your ZeroTier account, add a network - we'll use 10.255.0.0/24
2. Join your remote workstation & PVE node to the network, assign the PVE node's IP address, we'll use 10.255.0.110
3. In my.zerotier.com add 10.101.101.0/24 as a managed route
4. Add route to 10.101.101.0/24 on vmbr2
5. When building / modifying containers, use vmbr2 10.101.101.0/24
6. For system updates, you will want to enable masquerading

/etc/network/interfaces

Code:

auto lo
iface lo inet loopback

auto eno5
iface eno5 inet manual

auto eno6
iface eno6 inet manual

iface eno7 inet manual

iface eno8 inet manual

auto enp193s0f4u4
iface enp193s0f4u4 inet dhcp

auto vmbr0
iface vmbr0 inet static
        address 2.6.170.42/30
        gateway 2.6.170.41
        bridge-ports eno5
        bridge-stp off
        bridge-fd 0
        bridge-maxwait 0
        post-up ip route add 3.3.199.160/29 dev vmbr0

auto vmbr1
iface vmbr1 inet static
        address 192.168.101.32/24
        bridge-ports eno6
        bridge-stp off
        bridge-fd 0
        bridge-maxwait 3

auto vmbr2
iface vmbr2 inet static
        address 10.101.101.1/24
        bridge-ports ztzlgi8qn1
        bridge-stp off
        bridge-fd 0
        bridge-maxwait 3
        post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '10.101.101.0/24' -o vmbr0 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.101.101.0/24' -o vmbr0 -j MASQUERADE

grep -ris "vmbr2" /etc/pve:

Code:

/etc/pve/nodes/vm101-01/qemu-server/103.conf:net0: e1000=F2:4E:F9:60:9A:40,bridge=vmbr2,firewall=1
/etc/pve/nodes/vm101-01/lxc/102.conf:net0: name=eth0,bridge=vmbr2,firewall=1,gw=10.101.101.1,hwaddr=86:11:17:5B:76:07,ip=10.101.101.10/24,type=veth
/etc/pve/nodes/vm101-01/lxc/100.conf:net1: name=eth1,bridge=vmbr2,firewall=1,gw=192.168.101.32,hwaddr=AE:79:8E:AD:E7:9D,ip=10.101.101.11/24,type=veth

my.zerotier.com:

KatyComputer · Oct 19, 2020

It seems my tutorial is flawed.

lxc-102 cannot ping 8.8.8.8, so I tried 10.101.101.1, no joy.

Yet vm101 can ping 10.101.101.10 and I can ping 10.101.101.10 remotely using ZeroTier.

Anyone know what I may have missed?

guletz · Oct 19, 2020

Hi,

I would check the default gw on lxc-102 !

KatyComputer · Oct 19, 2020

Thanks. I have default gw on lxc-102 set to 10.101.101.1 - I think that's correct, but what's odd is that from within lxc-102, I cannot ping 10.101.101.1, I suspect if I resolve this issue things will work correctly.

I know it's something silly stupid, but not sure what it is - I suppose it will be the last thing I check

guletz · Oct 19, 2020

Hi,

Check in lxc-102:

Code:

arp -an

: if you can see any ARP for 10.101.101.1
check if you have any firewall rule(including on PMX node) that could block your icmp
try to use traceroute 10.101.101.1 instead of ping

Good luck / Bafta!

kozaks · Nov 18, 2020

KatyComputer said:
Thanks. I have default gw on lxc-102 set to 10.101.101.1 - I think that's correct, but what's odd is that from within lxc-102, I cannot ping 10.101.101.1, I suspect if I resolve this issue things will work correctly.

I know it's something silly stupid, but not sure what it is - I suppose it will be the last thing I check

Did you fix your problem?

KatyComputer · Nov 18, 2020

I gave up.

jamest65 · May 1, 2021

KatyComputer said:
I gave up.

I have built a recent proxmox lab and got zerotier working through a NAT setup inside my VM's using the standard NAT config as per the instructions. I just have to work out a rule to allow the ports to get out from the VM. I can ssh from the outside into my VM from my laptop which is really awesome. I have posted this to see if anyone is interested to explore this any further.

crazy_otto · Jul 7, 2021

jamest65 said:
I have built a recent proxmox lab and got zerotier working through a NAT setup inside my VM's using the standard NAT config as per the instructions. I just have to work out a rule to allow the ports to get out from the VM. I can ssh from the outside into my VM from my laptop which is really awesome. I have posted this to see if anyone is interested to explore this any further.

Yes, please!

jamest65 · Jul 9, 2021

crazy_otto said:
Yes, please!

I have set up my own controller to allow both Proxmox server and VM/LXC to be part of the network. This allows me to deny all keep it strictly offline. Let me know what you would like to discuss.

bhughes55 · Sep 2, 2022

If anybody is interested I think I have it worked out or at least it works for my proxmox server----if you need help get back to me

KatyComputer · Sep 2, 2022

Tell us more

Prochor · Sep 2, 2022

Install ZeroTier bridge host PVE
https://www.prochor.ru/content/nashi-stati/proxmox-ve/content11/proxmox-ve-zerotier-one.html

oldercoder · Sep 8, 2022

bhughes55 said:
If anybody is interested I think I have it worked out or at least it works for my proxmox server----if you need help get back to me

Hi, here I am getting back to you. I need to run zero-tier in docker container residing on vm

struland · Nov 30, 2022

Hello,
tried very hard to get my setup working. But so far I did not manage to get it working.
In Zerotier I set up the routing:
192.168.131.0/24 via 10.147.20.131

From my PC (KINGLOUIE) I was able to ping 192.168.131.204. That is the IP Adress of the proxmox installation and on which I installed Zerotier.

What I did not manage so far is to ping the other things in the 192.168.131.0/24 network.
How do I check what is wrong?

UdoB · Nov 30, 2022

struland said:
What I did not manage so far is to ping the other things

While I won't dig into the details of your topology: in my personal experience I struggle most often by forgetting to configure the route back from the destination to the source. The relevant routers need to know which route to forward packages to - from any point of the topology map. So... check your routing tables for packages coming from those destination systems back to you (and possibly fail).

Good luck

crycheck · Sep 12, 2024

Below the bash script for auto-setup bridge which is based on zerotier.

1. You have to set ZT_NETWORK, BR_IF and static IP of zerotier interface.

Bash:

#/bin/bash

ZT_NETWORK="zerotier-network-id-16-digits"
BR_IF="vmbr1"
ZT_IP="172.30.0.2/16"

apt update
apt -y install curl pgp

# Installing zerotier-one
curl -s 'https://raw.githubusercontent.com/zerotier/ZeroTierOne/main/doc/contact%40zerotier.com.gpg' | gpg --import && \
if z=$(curl -s 'https://install.zerotier.com/' | gpg); then echo "$z" | bash; fi

# Setting zerotier interface name
echo $ZT_NETWORK"=enp1z0" > /var/lib/zerotier-one/devicemap

zerotier-cli join $ZT_NETWORK
zerotier-cli set $ZT_NETWORK allowManaged=0

echo "" >> /etc/network/interfaces
echo "auto vmbr1" >> /etc/network/interfaces
echo "iface vmbr1 inet static" >> /etc/network/interfaces
echo "    address" $ZT_IP >> /etc/network/interfaces
echo "    bridge-ports enp1z0" >> /etc/network/interfaces
echo "    bridge-stp off" >> /etc/network/interfaces
echo "    bridge-fd 0" >> /etc/network/interfaces
echo "    bridge-maxwait 3" >> /etc/network/interfaces

# ========= /sbin/zerotier-bridge.sh =============
cat << EOF | tee /sbin/zerotier-bridge.sh
#!/bin/sh
VB=vmbr1
EthZT=enp1z0
zerotier-cli status | if grep ONLINE >/dev/null
then
echo "ZT OK";
else
echo "ZT NOT OK";
systemctl start zerotier-one.service
sleep 10
fi

brctl show | if grep \$EthZT >/dev/null
then
echo "brctl OK";
else
echo "brctl NOT OK";
brctl addif \$VB \$EthZT
ip link set \$EthZT up
fi
EOF
# =================================================
chmod +x /sbin/zerotier-bridge.sh


# ========= /usr/lib/systemd/system/zerotier-bridge.service =============
cat << EOF | tee /usr/lib/systemd/system/zerotier-bridge.service
[Unit]
Description=Run script at startup zerotier-bridge.sh

[Service]
Type=oneshot
ExecStart=/sbin/zerotier-bridge.sh
TimeoutStartSec=0
EOF
# =================================================


# ========= /usr/lib/systemd/system/zerotier-bridge.timer =============
cat << EOF | tee /usr/lib/systemd/system/zerotier-bridge.timer
[Unit]
Description=Run script add ZT to Brige Proxmox

[Timer]
OnBootSec=1min

[Install]
WantedBy=default.target
EOF
# =================================================

systemctl disable zerotier-bridge.service
systemctl enable zerotier-bridge.timer

2. Set "Allow Ethernet Bridging" checkbox at the Advanced tab for connection in my.zerotier.com

3. Reboot. Keep in mind, that the zerotier interface wiill up after 1 min after boot finished or you can change it by OnBootSec=1min in zerotier-bridge.timer file.

You should get something like this:

Code:

root@pve:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000
    link/ether bc:24:11:00:00:01 brd ff:ff:ff:ff:ff:ff
    altname enp0s18
3: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether bc:24:11:00:00:01 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.234/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::be24:11ff:fe30:cbd8/64 scope link
       valid_lft forever preferred_lft forever
4: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2800 qdisc noqueue state UP group default qlen 1000
    link/ether fa:5b:3b:00:00:01 brd ff:ff:ff:ff:ff:ff
    inet 172.30.0.2/16 scope global vmbr1
       valid_lft forever preferred_lft forever
    inet6 fe80::6ca3:68ff:0000:0001/64 scope link
       valid_lft forever preferred_lft forever
5: enp1z0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2800 qdisc pfifo_fast master vmbr1 state UNKNOWN group default qlen 1000
    link/ether fa:5b:3b:00:00:01 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::f85b:0000:0000:0001/64 scope link
       valid_lft forever preferred_lft forever

root@npve:~# brctl show
bridge name     bridge id               STP enabled     interfaces
vmbr0           8000.ab***       no              ens18
vmbr1           8000.bc***       no              enp1z0

root@pve:~# ip route
default via 192.168.1.2 dev vmbr0 proto kernel onlink
172.30.0.0/16 dev vmbr1 proto kernel scope link src 172.30.0.2
192.168.1.0/24 dev vmbr0 proto kernel scope link src 192.168.1.234

Works on a freshly installed PVE 8.2.4 (6.8.12-1-pve).
I'm no so good in bash scripting, will glad if someone refactor it.

shallowal · Oct 15, 2024

Struggling with this,
First of all, I tried creating the shell script and running it, but it failed with an unexpected EOF. So I then tried to run sections at a time from the command line, but discovered that the script assignment of shell variables didnt work. So, after getting past those couple of issues I managed to get all the procedure run, however I'm still not getting the result.
So, A couple of questions if I may:
1. Where do the vmbr0 and ens18 interfaces come from. I suspect the vmbr0 is supposed to be the proxmox virtual interface and ens18 is eth0 on my setup. Can you comment please?
When I reboot I end up with the vmbr1 bridge showing up from the "brctl show" command, but nothing else
2. the "ip route" command results in a quite different response.
3. I had to add the line mentioned in this post "https://www.reddit.com/r/Proxmox/comments/jctd6x/zerotier_on_proxmox/" to my container config before I could get any info for the "zerotier_cli listnetworks" command.
My home network is on the 10.0.0.0 network, and the CT has an address of 10.0.0.117. ZT has assigned an address of 192.168.192.80 to the VM.
4. I have to manually run the zerotier-bridge.sh script twice manually for it to work??

I'm very much a novice with ZeroTier and Proxmox.

Code:

root@Zerotier:~# brctl show
bridge name     bridge id               STP enabled     interfaces
vmbr1           8000.daa8c101694d       no              enp1z0


root@Zerotier:~# ip route
default via 10.0.0.138 dev eth0
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.117
10.0.0.0/24 via 192.168.192.1 dev enp1z0 proto static metric 5000
192.168.192.0/24 dev vmbr1 proto kernel scope link src 192.168.192.1
192.168.192.0/24 dev enp1z0 proto kernel scope link src 192.168.192.80


root@Zerotier:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: eth0@if57: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether bc:24:11:e7:72:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.0.0.117/24 brd 10.0.0.255 scope global dynamic eth0
       valid_lft 863988sec preferred_lft 863988sec
    inet6 2406:2d40:474e:e100:be24:11ff:fee7:7202/64 scope global dynamic mngtmpaddr
       valid_lft 245sec preferred_lft 95sec
    inet6 fe80::be24:11ff:fee7:7202/64 scope link
       valid_lft forever preferred_lft forever
3: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether e2:fa:bb:53:1c:61 brd ff:ff:ff:ff:ff:ff
    inet 192.168.192.1/24 brd 192.168.192.255 scope global vmbr1
       valid_lft forever preferred_lft forever
    inet6 fe80::e0fa:bbff:fe53:1c61/64 scope link
       valid_lft forever preferred_lft forever
4: enp1z0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2800 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    link/ether da:a8:c1:01:69:4d brd ff:ff:ff:ff:ff:ff
    inet 192.168.192.80/24 brd 192.168.192.255 scope global enp1z0
       valid_lft forever preferred_lft forever
    inet6 fe80::d8a8:c1ff:fe01:694d/64 scope link
       valid_lft forever preferred_lft forever

root@Zerotier:~# zerotier-cli listnetworks
200 listnetworks <nwid> <name> <mac> <status> <type> <dev> <ZT assigned ips>
200 listnetworks <my network ID> <Mynetwork name> da:a8:c1:01:69:4d OK PRIVATE enp1z0 192.168.192.80/24

crycheck · Oct 15, 2024

shallowal said:
Struggling with this,
First of all, I tried creating the shell script and running it, but it failed with an unexpected EOF. So I then tried to run sections at a time from the command line, but discovered that the script assignment of shell variables didnt work. So, after getting past those couple of issues I managed to get all the procedure run, however I'm still not getting the result.
So, A couple of questions if I may:
1. Where do the vmbr0 and ens18 interfaces come from. I suspect the vmbr0 is supposed to be the proxmox virtual interface and ens18 is eth0 on my setup. Can you comment please?
When I reboot I end up with the vmbr1 bridge showing up from the "brctl show" command, but nothing else
2. the "ip route" command results in a quite different response.
3. I had to add the line mentioned in this post "https://www.reddit.com/r/Proxmox/comments/jctd6x/zerotier_on_proxmox/" to my container config before I could get any info for the "zerotier_cli listnetworks" command.
My home network is on the 10.0.0.0 network, and the CT has an address of 10.0.0.117. ZT has assigned an address of 192.168.192.80 to the VM.
4. I have to manually run the zerotier-bridge.sh script twice manually for it to work??

I'm very much a novice with ZeroTier and Proxmox.

Code:

root@Zerotier:~# brctl show bridge name bridge id STP enabled interfaces vmbr1 8000.daa8c101694d no enp1z0 root@Zerotier:~# ip route default via 10.0.0.138 dev eth0 10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.117 10.0.0.0/24 via 192.168.192.1 dev enp1z0 proto static metric 5000 192.168.192.0/24 dev vmbr1 proto kernel scope link src 192.168.192.1 192.168.192.0/24 dev enp1z0 proto kernel scope link src 192.168.192.80 root@Zerotier:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host noprefixroute valid_lft forever preferred_lft forever 2: eth0@if57: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether bc:24:11:e7:72:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.0.0.117/24 brd 10.0.0.255 scope global dynamic eth0 valid_lft 863988sec preferred_lft 863988sec inet6 2406:2d40:474e:e100:be24:11ff:fee7:7202/64 scope global dynamic mngtmpaddr valid_lft 245sec preferred_lft 95sec inet6 fe80::be24:11ff:fee7:7202/64 scope link valid_lft forever preferred_lft forever 3: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether e2:fa:bb:53:1c:61 brd ff:ff:ff:ff:ff:ff inet 192.168.192.1/24 brd 192.168.192.255 scope global vmbr1 valid_lft forever preferred_lft forever inet6 fe80::e0fa:bbff:fe53:1c61/64 scope link valid_lft forever preferred_lft forever 4: enp1z0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2800 qdisc pfifo_fast state UNKNOWN group default qlen 1000 link/ether da:a8:c1:01:69:4d brd ff:ff:ff:ff:ff:ff inet 192.168.192.80/24 brd 192.168.192.255 scope global enp1z0 valid_lft forever preferred_lft forever inet6 fe80::d8a8:c1ff:fe01:694d/64 scope link valid_lft forever preferred_lft forever root@Zerotier:~# zerotier-cli listnetworks 200 listnetworks <nwid> <name> <mac> <status> <type> <dev> <ZT assigned ips> 200 listnetworks <my network ID> <Mynetwork name> da:a8:c1:01:69:4d OK PRIVATE enp1z0 192.168.192.80/24

Hi, I think the problem is an existing of this two routes:
10.0.0.0/24 via 192.168.192.1 dev enp1z0 proto static metric 5000
192.168.192.0/24 dev vmbr1 proto kernel scope link src 192.168.192.1

It is created by default by zerotier-cli. The command below disable it:

Code:

zerotier-cli set <NetworkID> allowManaged=0

After that you can check it by:

Code:

zerotier-cli get <NetworkID> allowManaged

It have to be equal zero.
After that restart network and zerotier-cli (or reboot) and check again routes list.

You can run it twice but before you have to remove entries connected with vmbr1 in /etc/network/interfaces.

shallowal · Oct 16, 2024

Thanks for that quick answer. My delayed response was because I was sleeping.
Anyway, thanks to your help I can now ping and ssh to my zerotier gateway CT from outside (ie the public phone network), however I cant connect to anything else on my LAN. I presume this is because there is no bridge working between the zerotier virtual network and the lan bridge on the proxmox VM. ie:

Code:

root@Zerotier:~# brctl show
bridge name     bridge id               STP enabled     interfaces
vmbr1           8000.daa8c101694d       no              enp1z0

I tried adding the eth0 IF to the bridge, but that didnt work as I can only add it to vmbr1. I tried adding it to the vmbr0 IF but it says it doesnt exist:

Code:

root@Zerotier:~# brctl addif vmbr0 eth0
bridge vmbr0 does not exist!

The VM shows that vmbr0 is linked to eth0

How do I get the bridge configured?.

Code:

root@Zerotier:~# ip route
default via 10.0.0.138 dev eth0
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.117
192.168.192.0/24 dev vmbr1 proto kernel scope link src 192.168.192.1
root@Zerotier:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: eth0@if73: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether bc:24:11:e7:72:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.0.0.117/24 brd 10.0.0.255 scope global dynamic eth0
       valid_lft 863831sec preferred_lft 863831sec
    inet6 2406:2d40:474e:e100:be24:11ff:fee7:7202/64 scope global dynamic mngtmpaddr
       valid_lft 288sec preferred_lft 138sec
    inet6 fe80::be24:11ff:fee7:7202/64 scope link
       valid_lft forever preferred_lft forever
3: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2800 qdisc noqueue state UP group default qlen 1000
    link/ether da:a8:c1:01:69:4d brd ff:ff:ff:ff:ff:ff
    inet 192.168.192.1/24 brd 192.168.192.255 scope global vmbr1
       valid_lft forever preferred_lft forever
    inet6 fe80::ece9:f0ff:fef4:af00/64 scope link
       valid_lft forever preferred_lft forever
4: enp1z0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2800 qdisc pfifo_fast master vmbr1 state UNKNOWN group default qlen 1000
    link/ether da:a8:c1:01:69:4d brd ff:ff:ff:ff:ff:ff
    inet6 fe80::d8a8:c1ff:fe01:694d/64 scope link
       valid_lft forever preferred_lft forever

[TUTORIAL] ZeroTier + Proxmox PVE

Well-Known Member

Well-Known Member

Famous Member

Well-Known Member

Famous Member

Member

Well-Known Member

Member

New Member

Member

New Member

Well-Known Member

Member

New Member

New Member

Distinguished Member

New Member

New Member

New Member

New Member