Force LXC CT to use secondary IP for all outbound connections? (how to force scope LINK?)

[mathx]

An ancient container I inherited in a /25 at location 1 with ip Q on host Z needs to be moved to location 2 on host Y and retain ip Q. We cannot move the /25, there are other hosts+vms+cts on it at 1. We can only route Q/32 to Y.

The CT's software cannot be touched or reconfigured or otherwise modified except at the OS level (ie ip route add, ip addr addr or ip tables -I). We have full control of host Y to do what magic we want.

My solution was to discover what ip address SCOPE is about, and this works:

Shutdown CT Q at Z. Copy CT files from Z to Y.

1) At router: route $Q/32 to host Y.

2) on host Y add a new vmbr253 on an unused interface (ens2, coulda used a dummy virt iface instead), and add 192.168.55.1/24 to it.

3) in the CT lxc .conf file:

net0 with ip $Q/32 as eth0 on bridge vmbr0
net1 with ip 192.168.55.2/24 as eth1 on bridge vmbr253

4) then on the CT after starting it:

ip addr del 192.168.55.2/24 dev eth1
ip addr add 192.168.55.2/24 scope LINK dev eth1
ip route add default via 192.168.55.1

(ip addr change and ip addr replace do not work to change scope. you must del + re-add.)

5) and finally on host Y:

ip route add $Q via 192.168.55.2

This causes all outbound connections on the CT (except to 192.168.55.0/24) to use the (scope GLOBAL) ip $Q for outbound, and route it out 192.168.55.1. It also causes all inbound traffic to of course work fine going to the CT (that's the easy part).

My question is how do I make that scope LINK permanent in the .conf file for net1 aka eth1? And is there an easier/more natural way to operate a CT on a /32 routed to a host?

Nothing in here mentions scope for net[n]: https://pve.proxmox.com/wiki/Manual:_pct.conf#_file_format

Other solutions might be iptables MASQ, but I dont think that that works because I have to reconfigure the software to use some masq'd RFC1918 IP, and the software is imprinted with its real IP all over the place. (Ancient server, do not want to touch til it's replaced/upgraded with entirely different software).

There could be an ip rule (source routing) solution here, but it seems more complex and unnatural.

 

[bobmc]

how is networking between location 1 and location 2 configured - are there routing devices at either end and do you have control of these?

 

[mathx]

top level router splits traffic to site 1 and site 2 based on dest ip. 1 and 2 communicate via this router at layer 3 with routing, there's no opportunity/ability to vlan or otherwise share a broadcast ethernet between them. (I am also not interested in GRE tunnels, etc)

nonetheless I am interested in the generic solution of using a /32 for a container, as this vastly increases IP availability when faced with multitudes of /29s and /30s routed around as well as we can reclaim network and broadcast IPs (2 ips per subnet)

the mechanics of doing what I need multiple ways (ip rule?) are interesting, but my real single question is:

how do i express scope LINK in /etc/pve/lxc/*.conf?

 

[mathx]

btw friend suggest alt solution, but does not work: ip route add default via 192.168.55.1 src $Q

So SCOPE seems to be the way to do this.

The host doenst seem to route the IP's back (test ping to something on local physical networ on foreign subnet, it sees pings and replies via route back to ct host). The ct can't even ping its host's ip on that foreign subnet.

(Yes forwarding=1.)

 

[mathx]

Namely my question is "why is there an ip networking configuration not supported in lxc/*.conf files"?

Please let me know how to set SCOPE LINK in a lxc/*.conf file.

 

[mathx]

4) then on the CT after starting it:

ip addr del 192.168.55.2/24 dev eth1
ip addr add 192.168.55.2/24 scope LINK dev eth1
ip route add default via 192.168.55.1

Better is to just put this all into the /etc/network/interfaces file:
auto eth1 iface eth1 inet static address 192.168.55.2 netmask 255.255.255.0 gateway 192.168.55.1 scope link

The scope link is retained by proxmox when it recreates the file - but I assume on first start of the ct you must go and edit this and restart it.