Hi there,
I've been doing IPv6 routing for OpenVZ CTs with venet for a while and it works like a charm. Notably this allows me *not* to bridge them on the main interface. This is a requirement for us because of the policy of the hosting environment.
Now, I'd like to treat KVMs, with the same requirements, so I set up a secondary bridge (vmbr1), with no physical interface on it, whereas the usual vmbr0 has the main physical interface. The end result is something that works for a few minutes unless I specify all routes manually between the guest and the host using link-local addresses. After a while or a reboot, ping6 does not pass between them. On the other hand, the very same bridge provides IPv4 for NATing without a hitch.
(details on PVE version at the end)
From inside the VM, if I find the link-local address of vmbr1 and use it as default route, and give the host a route to the VM's global address via the VM's link-local address, it works. Even this impractical way of doing is not enough, because the bridge's link-local address changes at each reboot. I had great hopes that fixing the bridge's MAC address would help, but it did not.
Another way to make it work is to give the host an explicit route to the guest, using this time knowledge of the guests' link-local address.
The bridge has only VMs and a CT :
The CT has been added because it starts first in the host boot sequence, and that helps with stability of the MAC address, but the problems were identical before I tried that.
I played a bit with neighbor proxy to no convincing result.
I tried enabling STP, with no result.
I played a bit with tcpdump to investigate. It seems that NDP packets from the host don't reach the guest and vice-versa. I had the feeling for a while that launching tcpdump actually fixed (again, temporarily) the ping, but if there's a side effect of that kind, it's not systematic. For instance, after a reboot, everything is stuck until I set up a static route using link-local addresses
I'm not very knowledgeable in layer2 matters, but it looks as an issue at this level of the stack. Maybe there's something very simple that I'm overlooking, but I'm running out of ideas.
Does all of this ring a bell to anyone ?
Configuration details:
On the host:
On the guest:
Pinging a VM from the host
And from inside a VM (CentOS 7 if that matters) :
On the host:
Now a setup that works (using link-local addresses everywhere).
Host:
Guest:
With this ping6 between host and guest work, and on top of that, I can reach the VM from the internet.
After rebooting the host, the LL address of vmbr1 has changed:
and now for something really puzzling, right after the reboot I did to write this message, the ping worked although the guest's default route is on a LL address that does not exist any more, but this lasted less than 10 mins.
Proxmox software versions:
I've been doing IPv6 routing for OpenVZ CTs with venet for a while and it works like a charm. Notably this allows me *not* to bridge them on the main interface. This is a requirement for us because of the policy of the hosting environment.
Now, I'd like to treat KVMs, with the same requirements, so I set up a secondary bridge (vmbr1), with no physical interface on it, whereas the usual vmbr0 has the main physical interface. The end result is something that works for a few minutes unless I specify all routes manually between the guest and the host using link-local addresses. After a while or a reboot, ping6 does not pass between them. On the other hand, the very same bridge provides IPv4 for NATing without a hitch.
(details on PVE version at the end)
Code:
auto vmbr1
iface vmbr1 inet static
address 192.168.57.1
netmask 255.255.255.0
bridge_ports none
bridge_stp off
bridge_fd 0
iface vmbr0 inet6 static
address 2001:xxx:200::1/64 # this is also the range used on CTs' venets
iface vmbr1 inet6 static
address 2001:xxx:201::1/64 # this is also the range for KVM guestsFrom inside the VM, if I find the link-local address of vmbr1 and use it as default route, and give the host a route to the VM's global address via the VM's link-local address, it works. Even this impractical way of doing is not enough, because the bridge's link-local address changes at each reboot. I had great hopes that fixing the bridge's MAC address would help, but it did not.
Another way to make it work is to give the host an explicit route to the guest, using this time knowledge of the guests' link-local address.
The bridge has only VMs and a CT :
Code:
galilee-~ $ sudo brctl show vmbr1
bridge name bridge id STP enabled interfaces
vmbr1 8000.0001002944ed no tap106i0
tap107i0
veth100.0The CT has been added because it starts first in the host boot sequence, and that helps with stability of the MAC address, but the problems were identical before I tried that.
I played a bit with neighbor proxy to no convincing result.
I tried enabling STP, with no result.
I played a bit with tcpdump to investigate. It seems that NDP packets from the host don't reach the guest and vice-versa. I had the feeling for a while that launching tcpdump actually fixed (again, temporarily) the ping, but if there's a side effect of that kind, it's not systematic. For instance, after a reboot, everything is stuck until I set up a static route using link-local addresses
I'm not very knowledgeable in layer2 matters, but it looks as an issue at this level of the stack. Maybe there's something very simple that I'm overlooking, but I'm running out of ideas.
Does all of this ring a bell to anyone ?
Configuration details:
On the host:
Code:
# sysctl net.ipv6.conf.vmbr1
net.ipv6.conf.vmbr1.forwarding = 1
net.ipv6.conf.vmbr1.hop_limit = 64
net.ipv6.conf.vmbr1.mtu = 1500
net.ipv6.conf.vmbr1.accept_ra = 1
net.ipv6.conf.vmbr1.accept_redirects = 1
net.ipv6.conf.vmbr1.autoconf = 0
net.ipv6.conf.vmbr1.dad_transmits = 1
net.ipv6.conf.vmbr1.router_solicitations = 3
net.ipv6.conf.vmbr1.router_solicitation_interval = 4
net.ipv6.conf.vmbr1.router_solicitation_delay = 1
net.ipv6.conf.vmbr1.force_mld_version = 0
net.ipv6.conf.vmbr1.use_tempaddr = 0
net.ipv6.conf.vmbr1.temp_valid_lft = 604800
net.ipv6.conf.vmbr1.temp_prefered_lft = 86400
net.ipv6.conf.vmbr1.regen_max_retry = 5
net.ipv6.conf.vmbr1.max_desync_factor = 600
net.ipv6.conf.vmbr1.max_addresses = 16
net.ipv6.conf.vmbr1.accept_ra_defrtr = 1
net.ipv6.conf.vmbr1.accept_ra_pinfo = 1
net.ipv6.conf.vmbr1.accept_ra_rtr_pref = 1
net.ipv6.conf.vmbr1.router_probe_interval = 60
net.ipv6.conf.vmbr1.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.vmbr1.proxy_ndp = 0
net.ipv6.conf.vmbr1.accept_source_route = 0
net.ipv6.conf.vmbr1.optimistic_dad = 0
net.ipv6.conf.vmbr1.mc_forwarding = 0
net.ipv6.conf.vmbr1.disable_ipv6 = 0
net.ipv6.conf.vmbr1.accept_dad = 1On the guest:
Code:
$ sudo sysctl net.ipv6.conf.ens18
net.ipv6.conf.ens18.accept_dad = 1
net.ipv6.conf.ens18.accept_ra = 1
net.ipv6.conf.ens18.accept_ra_defrtr = 0
net.ipv6.conf.ens18.accept_ra_pinfo = 0
net.ipv6.conf.ens18.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.ens18.accept_ra_rtr_pref = 0
net.ipv6.conf.ens18.accept_redirects = 1
net.ipv6.conf.ens18.accept_source_route = 0
net.ipv6.conf.ens18.autoconf = 1
net.ipv6.conf.ens18.dad_transmits = 1
net.ipv6.conf.ens18.disable_ipv6 = 0
net.ipv6.conf.ens18.force_mld_version = 0
net.ipv6.conf.ens18.force_tllao = 0
net.ipv6.conf.ens18.forwarding = 0
net.ipv6.conf.ens18.hop_limit = 64
net.ipv6.conf.ens18.max_addresses = 16
net.ipv6.conf.ens18.max_desync_factor = 600
net.ipv6.conf.ens18.mc_forwarding = 0
net.ipv6.conf.ens18.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.ens18.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.ens18.mtu = 1500
net.ipv6.conf.ens18.ndisc_notify = 0
net.ipv6.conf.ens18.optimistic_dad = 0
net.ipv6.conf.ens18.proxy_ndp = 0
net.ipv6.conf.ens18.regen_max_retry = 3
net.ipv6.conf.ens18.router_probe_interval = 60
net.ipv6.conf.ens18.router_solicitation_delay = 1
net.ipv6.conf.ens18.router_solicitation_interval = 4
net.ipv6.conf.ens18.router_solicitations = 3
net.ipv6.conf.ens18.temp_prefered_lft = 86400
net.ipv6.conf.ens18.temp_valid_lft = 604800
net.ipv6.conf.ens18.use_tempaddr = 0Pinging a VM from the host
Code:
$ ping6 2001:xxx:201::106
PING 2001:xxx:201::106(2001:xxx:201::106) 56 data bytes
From 2001:xxx:201::1 icmp_seq=2 Destination unreachable: Address unreachableAnd from inside a VM (CentOS 7 if that matters) :
Code:
$ ip -6 addr show ens18
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qlen 1000
inet6 2001:xxx:201::106/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::6041:c0ff:fee4:68bc/64 scope link
valid_lft forever preferred_lft forever
$ ip -6 addr show ens18
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qlen 1000
inet6 2001:xxx:201::106/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::6041:c0ff:fee4:68bc/64 scope link
valid_lft forever preferred_lft forever
$ ping6 -w 1 2001:xxx:201::1
PING 2001:xxx:201::1(2001:xxx:201::1) 56 data bytes
--- 2001:xxx:201::1 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 999ms
$ ip -6 neigh # (the stable MAC address worked, but the link-local of the router is still random)
2001:xxx:201::1 dev ens18 lladdr 00:01:00:29:44:ed router REACHABLE
fe80::eca0:8aff:feed:272a dev ens18 lladdr 00:01:00:29:44:ed router REACHABLEOn the host:
Code:
$ ip -6 route | grep vmbr
2001:xxx:200::/64 dev vmbr0 proto kernel metric 256 mtu 1500 advmss 1440 hoplimit 0
2001:xxx:201::/64 dev vmbr1 proto kernel metric 256 mtu 1500 advmss 1440 hoplimit 0
fe80::/64 dev vmbr0 proto kernel metric 256 mtu 1500 advmss 1440 hoplimit 0
fe80::/64 dev vmbr1 proto kernel metric 256 mtu 1500 advmss 1440 hoplimit 0
default via fe80::225:45ff:fef6:947f dev vmbr0 metric 1024 mtu 1500 advmss 1440 hoplimit 0Now a setup that works (using link-local addresses everywhere).
Host:
Code:
$ ip -6 route | grep vmbr
2001:xxx:200::1 dev vmbr0 proto kernel metric 256 mtu 1500 advmss 1440 hoplimit 0
2001:xxx:201::106 via fe80::6041:c0ff:fee4:68bc dev vmbr1 metric 1024 mtu 1500 advmss 1440 hoplimit 0
2001:xxx:201::/64 dev vmbr1 proto kernel metric 256 mtu 1500 advmss 1440 hoplimit 0
fe80::/64 dev vmbr0 proto kernel metric 256 mtu 1500 advmss 1440 hoplimit 0
fe80::/64 dev vmbr1 proto kernel metric 256 mtu 1500 advmss 1440 hoplimit 0
default via fe80::225:45ff:fef6:947f dev vmbr0 metric 1024 mtu 1500 advmss 1440 hoplimit 0Guest:
Code:
$ ip -6 route | grep -v unreachable
fe80::eca0:8aff:feed:272a dev ens18 proto static metric 100
fe80::/64 dev ens18 proto kernel metric 256
default via fe80::eca0:8aff:feed:272a dev ens18 proto static metric 100With this ping6 between host and guest work, and on top of that, I can reach the VM from the internet.
After rebooting the host, the LL address of vmbr1 has changed:
Code:
$ /sbin/ifconfig vmbr1 | grep -v 2001
vmbr1 Link encap:Ethernet HWaddr 00:01:00:29:44:ed
inet adr:192.168.57.1 Bcast:192.168.57.255 Masque:255.255.255.0
adr inet6: 2001:xxx:201::1/64 Scope:Global
adr inet6: fe80::9cfa:64ff:fe7c:ab41/64 Scope:Lien
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4 errors:0 dropped:0 overruns:0 frame:0
TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 lg file transmission:0
RX bytes:184 (184.0 B) TX bytes:686 (686.0 B)and now for something really puzzling, right after the reboot I did to write this message, the ping worked although the guest's default route is on a LL address that does not exist any more, but this lasted less than 10 mins.
Proxmox software versions:
Code:
$ dpkg -l \*pve\*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-=================================
ii corosync-pve 1.4.7-1 amd64 Standards-based cluster framework
ii fence-agents-p 4.0.10-2 amd64 fence agents for redhat cluster s
ii libcorosync4-p 1.4.7-1 amd64 Standards-based cluster framework
ii libopenais3-pv 1.1.4-3 amd64 Standards-based cluster framework
ii libpve-access- 3.0-16 amd64 Proxmox VE access control library
ii libpve-common- 3.0-24 all Proxmox VE base library
ii libpve-storage 3.0-31 all Proxmox VE storage management lib
ii novnc-pve 0.4-7 amd64 HTML5 VNC client
ii openais-pve 1.1.4-3 amd64 Standards-based cluster framework
ii pve-cluster 3.0-16 amd64 Cluster Infrastructure for Proxmo
ii pve-firewall 1.0-18 amd64 Proxmox VE Firewall
ii pve-firmware 1.1-3 all Binary firmware code for the pve-
un pve-kernel <none> (no description available)
ii pve-kernel-2.6 2.6.32-140 amd64 The Proxmox PVE Kernel Image
ii pve-kernel-2.6 2.6.32-147 amd64 The Proxmox PVE Kernel Image
un pve-kvm <none> (no description available)
ii pve-libspice-s 0.12.4-3 amd64 SPICE remote display system serve
ii pve-manager 3.4-1 amd64 The Proxmox Virtual Environment
ii pve-qemu-kvm 2.1-12 amd64 Full virtualization on x86 hardwa
un pve-qemu-kvm-2 <none> (no description available)
ii redhat-cluster 3.2.0-2 amd64 Red Hat cluster suite
ii resource-agent 3.9.2-4 amd64 resource agents for redhat cluste