Not getting routed setup with three single IPs plus one Subnet on hetzner dedicated

S

ssldn

Member
Jul 13, 2020
59
3
8
48
Global
Hi,
I have tried a lot of different setups the last days,... But I do not understand the routed setup to get to work. Am finally devastated and I would really like someone help me, so that I can try on my own also...(setting up a Vbox for testing aside.)
I had created the following /etc/network/interfaces.new and due to big problems, when I had installed ifupdown2 day before yesterday, I am activating the setup, now... it over the shell with:
ifdown --force enp0s31f6 lo && ifup -a
which brings output:
Bash: 
Waiting for vmbr0 to get ready (MAXWAIT is 2 seconds).
RTNETLINK answers: File exists
ifup: failed to bring up vmbr0

Waiting for vmbr2 to get ready (MAXWAIT is 2 seconds).
RTNETLINK answers: File exists
ifup: failed to bring up vmbr2

Waiting for vmbr4 to get ready (MAXWAIT is 2 seconds).
RTNETLINK answers: File exists (exactly..!)
ifup: failed to bring up vmbr4
I wonder why vmbr1, vmbr3 and vmbr5 not show up at all. :cool::eek:


This is the setup:
Bash: 
root@ghost0 ~ # 
[URL='https://pastebin.com/FxDQRjTr']get into: [/URL]
Bash: 
root@ghost0 ~ # 
[URL='https://pastebin.com/k0kjfNct']Netweurk[/URL]
 
yes, @jlebherz and @spirit I missed to mention all steps and am also sorry, for the lack of completion in describing. Finally, I was really tired after several days of "misconcepted" failures.
My prequel around Proxmox was the time from 2015 to 2017 and I had run it all the time with a bridged mode. But the confused setup comes from the overwhelming complexity which Proxmox has gained. I then started to find out and test (yes, I am aware it sould have better been done on a Vbox) and I learned some quite new things, tested around and testeed , whilst hoping.
The different other two or three problems I need to get help with am gonna post around the next hours also. So, I still need really help, thats whz am gonna make a view public , which is capable ...
My setup construction was heading for *got a host machine standard single IP
* the guests are gonna be on kVM an d windows desktop (brings in easy to use software of VPN -providers or proxytools)
* the network consists of with the many IPs on the each bridges, ....
*to be sure abot the easy minimaldeployment of windows running there made me believe: the IP ideal subnet
*because I was walking with nobody because I had t no mutual goals each VM will be able tto go messaging and using more I used my curiousity, than my combinative mindset. I did want to reside in place where I was with my other 10 or 12 problems, "to be solved", just a nother step I got energy for to make the image guest installation, will phrase a short statement about it in soon next post
 
spirit said:
note that ifup/ifdown/ifreloasd don't read /etc/network/interfaces.new, only /etc/network/interfaces. (proxmox do a replace on restart or reload through proxmox api).
Click to expand...
Yes, I know it is to be taken to the original /etc/network/interfaces which I did through cp /etc/network/interfaces.new /etc/network/interfaces and a reboot of the node
The config looks now like : Current ifconfig
lebherz said:
note that ifup/ifdown/ifreloasd don't read /etc/network/interfaces.new, only /etc/network/interfaces. (proxmox do a replace on restart or reload through proxmox api).
Click to expand...
The reason why I have set up several vmbrXX is not knowing, only trayxing to get a clue, about what comes next to fullfill my desired host/guest scenario.
I am still sorting my mind to make it clear, I think I will give you a little graphic.
But,... I want to run a bigger crowd of guest windows VMs for testing skills of some participants in our staff academy in Berlin. Because am with Hetzner hosting I need to do routed setup. So, my network adjustment after reading all the tutorials got cluttered and confused somehow. Sorry.
 
Do you want to assign one IP per Windows VM ?
Or do you want to have internal IPs on the VMs and just have internet access ?


depending on this decision I think we can help you finding the right setup method...
 
jlebherz said:
Do you want to assign one IP per Windows VM ?
Or do you want to have internal IPs on the VMs and just have internet access ?


depending on this decision I think we can help you finding the right setup method...
Click to expand...
Reply to Q1 & Q2.
Q1: Depends on the better operability and flexibility. I need to have internet connection from inside the windos VM. One Assigned IP will then be using a broadcast and gateway by the host, right? Probably not what I need in this case here. If I can assign one IP to the each VM then I could probably use different scenarios to change them later, routed from the host system. But I need external IPs for each case, right? (Or a dedicated proxy, somekind of...)

Q2: Yes, I want the each VM to get on the internet. I guess over the one NODE again all VMs get a routed connection ... and then inside the VM I could use a VPN software with changing IPs or some dedicated proxies from then on, if I want this. The least effort I think is method B.

Well, I was about starting with the link you gave me to see how it can help me, but frankly said, my capabilites in terms of networks are still not evolved tnough to find my way (am about taking courses the next week, to fill this lack..)
So, yes, better work is to shortly lay it out to you. I' ll get my pen and drawing pad and show it quickly.
 
Yeah, my goal is completely clear to me. I just see too many options and wanted to mention them.
This is the wanted setup:
short-scheme-VMs-on-Proxmox.png
 
Hi,

(Just to be sure, they are fake ip for the example right ?;) 23.456.210.117, 34.567.197.208/26 )


I don't use myself routed setup, I can't tell for /32 because it's tricky

but for a full subnet, it's easy like

Code: 
iface  eth0
    address 23.456.210.123
    post-up echo 1 > /proc/sys/net/ipv4/ip_forward

iface vmbr0 inet static
   address  34.567.197.208.1/26
   bridge-ports none
   bridge-stp off
   bridge-fd 0


You'll lose 1 ip for vmbr0, and it'll be used as gateway by the vms.
 
spirit said:
Hi,

(Just to be sure, they are fake ip for the example right ?;) 23.456.210.117, 34.567.197.208/26 )
Click to expand...
Yes.mainly I hope to have exchange all real and sensitive infos.

spirit said:
I don't use myself routed setup, I can't tell for /32 because it's tricky

but for a full subnet, it's easy like
Click to expand...

Why is it tricky and where can I read about the concrete differences between routed and bridged?

I wonder this, because finally when the IP is a vmbr0 connected IP then the VM itself later takes it as a gateway because the routed IP has an internet connection, which is routed to the NODE of ghost0 , correct? then we should be done.... I assume.
or, wait? confusing me.

And if I'd take a single IP in the vmbr0? I have two extra single IPs PLUS the subnet which itself is pointing at the second single IP ...210.123 (extra booked)

In your setup I can see okay, the 23.456.210.123 is set which is a second single IP already and the subnet has the ...197.208 as first from the subnet. As far as I know the first IP in a subnet is said like here:

The first Ip address of any subnet is used for the identification of the network. The devices use it to identify the network. Whereas the last IP address of a subnet if used for broadcast , if a device in the network wants to broadcast any message to all devices then it uses the last IP. means the part :
Code: 
iface vmbr0 inet static

   address  34.567.197.208.1/26
uses the IP address that I think may NOT be used. Shouldnt it be with the ....197.209.
?? And is it in general correct to add a .1 trailing the original IP address ? like I think it should be address 34.567.197.208/26 or? what effect has it to add the /26?

spirit said:
Code: 
iface  eth0
    address 23.456.210.123
    post-up echo 1 > /proc/sys/net/ipv4/ip_forward

iface vmbr0 inet static
   address  34.567.197.208.1/26
   bridge-ports none
   bridge-stp off
   bridge-fd 0


You'll lose 1 ip for vmbr0, and it'll be used as gateway by the vms.
Click to expand...
Thanks a lot so far, am really happy and glad to receive help and conversation about it. :)
 
hi, about /32 routing, I said it's tricking, because in vm it's generally need pointopoint configuration too.

classing routing is the real world, is with bigger subnet, where you can use 1ip in this subnet for gateway.
But to avoid to sell full subnets, some provider sell /32 ips in differents ranges

Code: 
?? And is it in general correct to add a .1 trailing the original IP address ? like I think it should be address 34.567.197.208/26 or? what effect has it to add the /26?
/26 = netmask 255.255.255.192. (ip/XX , is called cidr notation)
 
Hi,
thx for replying.
spirit said:
hi, about /32 routing, I said it's tricking, because in vm it's generally need pointopoint configuration too.

classing routing is the real world, is with bigger subnet, where you can use 1ip in this subnet for gateway.
But to avoid to sell full subnets, some provider sell /32 ips in differents ranges
Click to expand...
I do not see the connection between the two topics, sorry.
As I also did get a /29 subnet and had no choice I had not learned about it lately. :( More over the problem from the setup is which IP to take from where, and simple answer would be great about: Can I take one of those single IPs, like 23.456.210.123, then create a vmbr0 from it and leave the other single IP unsued. Then use the subnet on the Node ghost0 for provisioning ONE (yes, solely one IP) to all VMs I will create under this Node of ghost0 ???
I mean in virtual environment this could be probably a standard, that I might not be able to find explained anywhere, cause it is so basic.
I did not get that so far, sorry.
Or is it that the NODE ghost0 will need one and following IPs from the subnet to use it for each machine....?
So, when I want 50 VMs running aside same time, I'd need 50 external IPs?? I doubt the latter. When in 2016 I used the bridged host to run the VMs I had lots of VMs connected to the online world, without using more than those IPs. But as I had to use routed, from host, I got confused. and I was using version 3 in the past, also confusing.

And, Can I get the internet connected through to guests, there I will use a VPN Provier or proxies? so anyway each VM could get a different IP ?? But fixed one, for security as well.
spirit said:
Code: 
?? And is it in general correct to add a .1 trailing the original IP address ? like I think it should be address 34.567.197.208/26 or? what effect has it to add the /26?
/26 = netmask 255.255.255.192. (ip/XX , is called cidr notation)
Click to expand...
Yes, I have heard that its called this way.

I wondered if you made an unneeded character in there, if you kindly take a look, you wrote: address 34.567.197.208.1/26 xx.xxx.208.1/26
Let's look into what the PVE docs say.
And sorry for my lack of information. Just this and the BIOS stuff I always excluded, cause asa Linux user I not had a problem for 10 years, or so.

P.S.:
I truly have a bit a problem to understand the difference between the routed and the bridge mode from point of view when setting it up in proxmox, resp. the host. Cause we route the config but on the guest its used as a bridge again (or a NAT). So, why the hack can I not find anythign which makes this so clear (okay, I heard about the routed being more safe than bridged, so Hosters wnt it) so lear, that a Network Novice like I understands it a bit... Why something is done, you know? I mean, I dont ask for reworking any documentation in whole, yes, I of course am responsible to reach a state of knowledge from external sources, but I want to understand WHY (for example Hetzner) hosters decided for it.
 
I wondered if you made an unneeded character in there, if you kindly take a look, you wrote:
address 34.567.197.208.1/26 xx.xxx.208.1/26
Click to expand...

oh, ok sorry, wrong typo. lol ;)

Or is it that the NODE ghost0 will need one and following IPs from the subnet to use it for each machine....?
So, when I want 50 VMs running aside same time, I'd need 50 external IPs?? I doubt the latter. When in 2016 I used the bridged host to run the VMs I had lots of VMs connected to the online world, without using more than those IPs. But as I had to use routed, from host, I got confused. and I was using version 3 in the past, also confusing.
Click to expand...
I'm not sure to understand your schema. What is the ghost0 node ? (I'm seeing it on the left, and again on the right under proxmox)

About number of ip, you need 1 ip by vm if you want to reach them from outside (internet-->vm).
but if you want vm--->internet, you can do nat with only 1 ip.

I truly have a bit a problem to understand the difference between the routed and the bridge mode from point of view when setting it up in proxmox, resp. the host. Cause we route the config but on the guest its used as a bridge again (or a NAT). So, why the hack can I not find anythign which makes this so clear (okay, I heard about the routed being more safe than bridged, so Hosters wnt it) so lear, that a Network Novice like I understands it a bit... Why something is done, you know? I mean, I dont ask for reworking any documentation in whole, yes, I of course am responsible to reach a state of knowledge from external sources, but I want to understand WHY (for example Hetzner) hosters decided for it.
Click to expand...
the main difference between bridged vs routed, is where is the gateway of the vm. with bridged setup, your gateway is outside the server, on provider network.
Some provider don't have bridged setup (I think it's hetzner case), because they need to implement mac filtering for example, and gateway need to be in the same subnet. (and it can be difficult with public ipv4 in differents ranges).


I'm currently implemented a new sdn feature in proxmox, to try to automate this kind of setup, but I need some public server to look at clean configuration. (they are a lots of docs with differents config, but I think some of them are wrong)

If you want, I can help you to configure your node (if you are not yet in production). you can send me a pm on this forum.
 
spirit said:
If you want, I can help you to configure your node (if you are not yet in production). you can send me a pm on this forum.
Click to expand...
Yes, definitely. Let's give it a try. Am gonna send to you PIM. And as I am having many other competences assembled with my teams maybe there is even something I can help you with...;)
 
spirit said:
I'm not sure to understand your schema. What is the ghost0 node ? (I'm seeing it on the left, and again on the right under proxmox)
Click to expand...
The scheme is simple. I have proxmox installed, there is a node, I called it ghost0 (as the starting point of all gets a zero)-
the host and sub are called like this too.
spirit said:
About number of ip, you need 1 ip by vm if you want to reach them from outside (internet-->vm).
but if you want vm--->internet, you can do nat with only 1 ip.
Click to expand...
I am so sorry, but I still cannot understand why this worked all in the past, so that the people were able to access the VMs from outside. OR maybe they were given some kind of login over the proxmox ?? Proxmox to guest with a NAT ViSUAL control over any client is still available, or?
 
I am so sorry, but I still cannot understand why this worked all in the past, so that the people were able to access the VMs from outside.
Click to expand...
the only way to reach multiple vm with 1 public ip from outside, is to do port direction + nat
like

client--->internet---->1.1.1.1:80 (proxmoxhost)---nat+port rediction----->vm: 192.168.0.1:80
1.1.1.1:81(proxmoxhost)---nat+port rediction----->vm: 192.168.0.2:80

OR maybe they were given some kind of login over the proxmox ?? Proxmox to guest with a NAT ViSUAL control over any client is still available, or?
Click to expand...
Sorry, I don't understand this part. what is a "nat visual control" ?




Here a sample configuration for hertzner:

Code: 
auto eth0
iface eth0 inet static
   # Your main Hetzner IP
   address {{ hetzner_ip }}
  # Point to Point subnet mask
  netmask 255.255.255.255
  # Your main Hetzner Gateway
  pointopoint {{ gateway_ip }}
  # Your main Hetzner Gateway
  gateway {{ gateway_ip }}


# for /32 failover ip
iface vmbr1 inet static
  address {{ hetzner_ip }}
  netmask 255.255.255.255
  bridge_ports none
  bridge_stp off
  bridge_fd 0
  up ip route add {{ additional_ip }}/32 dev vmbr0
# up ip route add <second additional-IP>/32 dev vmbr0
  ...
  
  
# for a /29 ip range
auto vmbr2
   iface vmbr2 inet static
   address {{ public_subnet_first_ip }}
   netmask {{ public_subnet_netmask }}
   bridge_ports none
   bridge_stp off
   bridge_fd 0


and for vm (debian) configuration:

for /32 ips: (need pointopoint)

Code: 
auto eth0
iface eth0 inet static
       address {{ additional_ip }}
       netmask 255.255.255.255
       pointopoint {{ hetzner_ip }}
       gateway {{ hetzner_ip }}

for /29 subnet

Code: 
auto eth0
iface eth0 inet static
   address {{ 1ip_of_your_subnet }}
   netmask {{ private_subnet_netmask }}
   gateway {{ public_subnet_first_ip_defined_on_vmbr2 }}
 
I was in the set up over the GUI right now, and it did not let me set the same GATEWAY IP address (main) on the network card from host ( enp0s31f6 ) as on the vmbr0. As it was described by Dominic Pratt Hetzner Netzwerk
When not accepted in the GUI (which is told to be a safety belt for users, as proxmox said) then I am a bit concerned to just put it up VIA console...

About your post:
spirit said:
the only way to reach multiple vm with 1 public ip from outside, is to do port direction + nat
like

client--->internet---->1.1.1.1:80 (proxmoxhost)---nat+port rediction----->vm: 192.168.0.1:80
1.1.1.1:81(proxmoxhost)---nat+port rediction----->vm: 192.168.0.2:80
Click to expand...
Okay, I think will learn this.
spirit said:
Sorry, I don't understand this part. what is a "nat visual control" ?
Click to expand...
I only meant that I got a VNC or SPICE or so, from within the host, the proxmox and this way could only reach the inside the VM (like using windos, etc) yes?
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom