Failover virtual port to physical ?

[toxic]

Hello,
I use proxmox to virtualize my main router, passing it all physical NICs for the moment, and also passing a virtual port to vmbr0 so that my router gives access to all other VMs to my wider network (and to the proxmox admin GUI)

I face an issue when I backup my router VM since it's beeing paused and therefore no trafic can be routed, especially to my NAS to store the backup using CIFS...

I'm about to add a failover router so that when the main router VM is down I have something still beeing able to route trafic between my VLANs and such.

My issue/question is that I don't know how to add one of my physical NIC to the vmbr0 since I will definitely create a loop if I do so : 99% of the time I would prefer the trafic from my switch to go through the NIC passed to the virtualRouter and then through the virtual 10GB/s interface to vmbr0. But when the virtualRouter is down I need the trafic to go through a physical NIC that I could assign to vmbr0...

I think I could do this using openvSwitch since I could explicitely define a port on the bridge , add a physicial NIC as failover for this port, and pass all this to the virtualRouter. But even that I'm not very clear how to do with openVswitch.... And I also had some issues early on with openvSwitch not starting on boot or not shutting down and adding 20 minutes or so to my reboot time... So I'm very open to any help on how to do this, either without openvSwitch or with it...

What I am realy looking for is a failover since when the virtualRouter is up and running I do have proper connectivity for all I need, and I do use more than 1GB/s of traffic to my VMs (I pass several 1GB/s NICs to the virtualRouter).

But if you can reassure me that having a loop is not too bad, and that I would not limit myself to 1GB/s to my VMs if I just add a physical IC to vmbr0 and plug that into my switch... I realy fear the my switch would see all MAC of the VMs behind this physical NIC instead of behind the physical NICs that are passed to the virtualRouter and are part of a LAG group...

Once my failover router is setup I will probably find a way to test it out : just add a physical NIC to vmbr0 and hope that vmbr0 beeing connected to my virtualRouter both by a vtnet 10GB/s interfce AND through NIC+switch+NICsInLAG will not result in degraded performance....

Thanks in advance for any insight or help !
Best regards,
Toxic.

 

[ph0x]

I'm not sure if you try to solve the right issues here ...

How should two routers on the same network work together? And how would you want to handover the traffic? There is no such thing as a fallback gateway.
Why do you pass all physical interfaces to a vm? Usually vmbr0 already has a physical interface, which gets assigned to the router vm as WAN. And your vmbr0 being a virtual interface would probably report as up all the time, so how should a failover kick in?
Why does a router have to be backuped so often? And which backup mode did you choose? With a snapshot of the filesystem while the vm is running, the downtime should be around several milliseconds.

 

[toxic]

Thanks for your reply.

I am planning to use CARP to have the 2 routers on the same network like this :https://docs.opnsense.org/manual/how-tos/carp.html
The traffic handover is done by the backup router taking the VIP when it detects the first member is down and it can even sync packet filter status to keep existing flows running it seems. Since I can give both the virtual router and my failover router access to WAN through a dedicated interface I was indeed thinking this would provide a failover gateway but maybe I'm wrong?

The reason I pass all physical NICs to opnSense VM is performance, this way I can use LAG with my clients and then 10GB/s to the virtual network, that leaves me with several GB/s of traffic possible between my physical wired network and the virtual vmbr0. If I put a physical NIC on vmbr0 I will get 1GB/s less in the LAG to my wired network and it will also limit traffic to vmbr0 to 1GB/s unless I do LAG again on vmbr0 but I don't have so many NICs...

Vmbr0 being virtual it will indeed always be up, I am looking to failover the "port" of vmbr0 to which the virtual router is "plugged in", this one seems to not explicitly exist in linux bridge. When thinking in openVSwitch the virtual ports are explicit and I could create a "fast failover group" between the virtual port and the physical NIC it seems.

But guys at openVSwitch had the same point as you: how to detect the port is down. They pointed me to BFD or CFM wich seems will require quite a bit of reading to understand and especially implement...

As to why I backup so often, I'm just paranoid, and downtime is indeed not too bad. But I'm trying this to understand how things work, and also for maintenance downtimes and pure testing purposes: I could take my main routeur down, test new settings on the backup routeur, and just trash everything if I messed it up and simply spin the main vRouter back up while I fix my mistakes on the backup one

I'm still curious if having the loop in the network is really that bad, maybe there's no need at all for failover, since my clients are anyway on another VLAN the switch would always forward the traffic to the CARP VIP, so let's see how my virtual router decides to forward that to my VMs when it can do so through the virtual 10gb/s NIC and the physical NIC on my server VLAN, will it choose wisely...

 

[ph0x]

Okay, I heard about carp but haven't tried that so far. Maybe they handle the handover on their own quite well.
If you only serve tagged VLANs over vmbr0 then a physical interface would probably not create a loop. And still, loops are only bad if you don't take any measures beforehand. With STP enabled your hardware can handle that.
Yet again, if you want to try and test settings, why would you do that on your production network? Setup a small virtual network with its own router and test the settings there.

 

[toxic]

It's my home network, so production yes but test also I'm still waiting to receive some hardware to be able to test without disturbing the rest too much, but on weekends production is totally interruptible ^^

I'll have a look into STP as I don't know what it is yet.

As of now, vmbr0 is not VLAN aware, but it's simply because I don't have the failover router now and since it exists only between pve and my virtual router for now I saw no need to use VLAn in it, but as soon as I add a physical NIC to it I will indeed use only VLAN tagged traffic on it.

 

[toxic]

Ok, just looked up STP and it seems that will do the trick, only my vRouter needs it and opnSense lets me define priority for each interface so I will definitely be able to tell OpnSense to prefer the 10GB/s link.
Even my coming TL-SG1024DE has support for loop detection although it claims to fully disable a port, so I just hope it will play nice with STP.

 

[toxic]

Ok, my switch doesn't support STP so that's it. I'll have to look into something else.
I'm thinking about something that would check the status of the virtual router VM and if down it would attach the physical NIC to vmbr0, and find a hook before VM startup to detach the NIC from vmbr0 and pass it through to the starting VM using pci passthrough... I'm able to write a daemon to check that my virtual router is up, but :
- I'm not sure a NIC can be used by pve host and then be disabled to be passed to a VM, I think that needs a full reboot of the pve host which is not perfect.
- No idea how to plug my code before VM startup yet...

Will have to look into it.

 

[ph0x]

Happy hacking