Q: Proxmox 5 on Debian / weird 'route' issue

fortechitsolutions

Well-Known Member
Jun 4, 2008
393
38
48
Hi all,

this is a bit of long shot but just in case. I've got a client still using a few legacy proxmox servers running Proxmox5.x which we haven't migrated up to Proxmox6 quiet yet. (sigh).

I had to deploy a new host for them yesterday. Their environment is all on OVH_SYS. The new box was deployed by starting with stock debian9 template, then once deployed, adding in proxmox-on-debian install config. Which normally I've done without any serious drama.

One new-weird thing appears to be happening, that is making me pull out some hair. And I am curious if anyone can comment / has seen this before.

Initial stock config of the debian host was non-bridge / one IP address on the public physical interface.
I changed config so it has a more proxmox-friendly setup, 2 bridges - one on physical interface / and vmbr0 has the public IP address.
then vmbr1 is bound to dummy0 interface / and has private LAN range behind it / also linked to tinc VPN between the cluster nodes / and VMs bridge only to this IP range/interface. I'm doing NAT on the public interface / so things bound on the private range use the proxmox physical host as gateway to get to internets.

What is strange, is that when I boot up this new proxmox host. I have unwanted references to the old physical interface present in the route table. Which I must then manually delete before VMs routing out to internet works as desired. And I cannot find any traces in my /etc/network directory that is putting this config here, but it definitely is re-appearing if I reboot. I've put in a ticket to OVH Support to ask nicely, "hey is there some customization on network/script hidden somewhere that manually sets this route info?" but I am not holding my breath, ie, I expect them to say "ha ha, sorry, that is internal OS config work, not our problem, have a nice day!". (But maybe they will surprise me?!)

So... here is what I've got in my /etc/network/interfaces file presently.

Code:
root@proxmox:/etc/network# cat interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

iface eno1 inet manual

# vmbr0: Bridging. Make sure to use only MAC adresses that were assigned to you.
# NOTE X AND Y ARE HERE JUST TO OBSCURE MY ACTUAL PUBLIC IP IN FORUM POST.
# THEY REPRESENT A SINGLE DIGIT EACH FOR VALID IP-GW ...
auto vmbr0
iface vmbr0 inet static
        address 1X2.Y5.31.93/24
        gateway 1X2.Y5.31.254
        bridge_ports eno1
        bridge_stp off
        bridge_fd 0
#public

# Internal Network
auto vmbr1
iface vmbr1 inet static
    address 192.168.95.251/24
    bridge_ports dummy0
    bridge_stp off
    bridge_fd 0
#private

root@proxmox:/etc/network#


and when I reboot this dude, here is what I've got active at first for output from 'route'

Code:
root@proxmox:/etc/network# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         1X2.Y5.31.254   0.0.0.0         UG    0      0        0 eno1
default         1X2.Y5.31.254   0.0.0.0         UG    0      0        0 vmbr0
1X2.Y5.31.0     0.0.0.0         255.255.255.0   U     0      0        0 eno1
1X2.Y5.31.0     0.0.0.0         255.255.255.0   U     0      0        0 vmbr0
192.168.95.0    0.0.0.0         255.255.255.0   U     0      0        0 vmbr1
224.0.0.0       0.0.0.0         240.0.0.0       U     0      0        0 vmbr1
root@proxmox:/etc/network#

ie, I've got duplicate entries for the (default gateway designation) / one references eno1 and one references vmbr0
and also duplicate entry for the 1X2.Y5.31.0 destination / each again references the eno1 and vmbr0 interfaces.

at this point VMs running here cannot ping out to google.

Then if I manually delete unwanted route entries thus:

Code:
route delete default gw 1X2.Y5.31.254 eno1
route del -net 1X2.Y5.31.0 gw 0.0.0.0 netmask 255.255.255.0 dev eno1

Then those unwanted lines are gone, and I am left with:


Code:
root@proxmox:/etc/network# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         1X2.Y5.31.254   0.0.0.0         UG    0      0        0 vmbr0
1X2.Y5.31.0     0.0.0.0         255.255.255.0   U     0      0        0 vmbr0
192.168.95.0    0.0.0.0         255.255.255.0   U     0      0        0 vmbr1
224.0.0.0       0.0.0.0         240.0.0.0       U     0      0        0 vmbr1
root@proxmox:/etc/network#

and now inside my VM guests, ping to google works just dandy.

If I reboot the physical server, the unwanted route entries re-appear and I am back where I started.

So ... gobs of fun.


If anyone has ever banged their head against this and has any thoughts, please let me know.

(BTW it seems that OVH have removed their pre-cooked proxmox templates? or at least on the product line I bought a server for yesterday, this is the case. Not sure if they were finding proxmox was interfering with their VMware resale? :) or if they just are letting people use proxmox if they are more comfortable with a DIY method rather than a pre-cooked template method.)


Anyhoo.

Many thanks if you read this far.


-Tim
 

fortechitsolutions

Well-Known Member
Jun 4, 2008
393
38
48
ha ha, after writing my question. I think I may have found it. Just need to dig a bit more.

There appears to be a stock file present, /etc/systemd/network/50-default.network

which has some info that looks like it is getting involved.

ie,

Code:
File: /etc/systemd/network/50-default.network

Contains:


# This file sets the IP configuration of the primary (public) network device.
# You can also see this as "OSI Layer 3" config.
# It was created by the OVH installer, please be careful with modifications.
# Documentation: man systemd.network or https://www.freedesktop.org/software/systemd/man/systemd.network.html

[Match]
# FAKED OUT MAC TDC NOTED JUST FOR THIS POSTING
MACAddress=XX:AA:BB:CC:DD:EE

[Network]
Description=network interface on public network, with default route
DHCP=no
# NOTE X AND Y AGAIN ARE VALID NUMERALS ON THE REAL FILE, FAKED OUT HERE FOR POST...
Address=1X2.Y5.31.93/24
Gateway=1X2.Y5.31.254
IPv6AcceptRA=no
NTP=ntp.ovh.net
DNS=127.0.0.1
DNS=213.186.33.99
DNS=2001:41d0:3:163::1
## next line also faked out AA-BB-CC-DD
Gateway=2AA7:5BB0:0CC0:2DDf:ff:ff:ff:ff

[Address]
Address=2AA7:5BB0:6CC0:2DDd::/64

[Route]
Destination=2AA7:5BB0:0CC0:2DDf:ff:ff:ff:ff
Scope=link


So .. now I just need to confirm if I can delete this file so this default info is not present / if that will cause any unwanted side effect. Or not.

But most likely this is the thing, I am guessing.

Great fun!

I'll leave the thread here in case anyone else (or me? in future, with forgetful memory) finds this thread via google search etc.



Tim
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
8,158
1,594
164
yeah, this is the equivalent of /etc/network/interfaces but for systemd-networkd - so it will cause exactly the symptoms you describe, and can be removed (provided you have DNS configured in /etc/resolv/conf ;)). note that it also contains the default ipv6 setup which you don't have in your interfaces file - you might want to carry that o ver (or not)
 

fortechitsolutions

Well-Known Member
Jun 4, 2008
393
38
48
thanks for confirming! I moved/removed the 2 files in the directory /etc/systemd/network/
rebooted the host
things came up cleanly without the unwanted route being forced-configured
so - all is well.

(I am not using ipv6 so did not need to bring it over.)

so, nice when things work 'as planned' even if we don't know in advance all of the details for what that plan seems to be :)

Thanks!

Tim
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!