LXC ipv6 only - Really slow.

[Nainterceptor]

Hello,

I'm trying to configure some LXC containers with an ipv6 public address only (without fallback in ipv4, for some non-critical services). All works but it's really slow.

My sysctl custom file :

net.ipv6.conf.all.accept_ra = 2
net.ipv6.conf.all.forwarding = 1
net.ipv6.conf.default.forwarding = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.ip_forward = 1
kernel.sysrq = 1
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.eth0.proxy_arp = 1

Some metrics :
* ping 11ms

* wget small file "hello world" 40 to 90 secondes.
* wget 100Mo file : Not a timeout, but ETA close to 20 days.

* apt-get update : Timeout


Same tests from proxmox host are normal. (110MB/s)

Any idea ?

Thanks,

 

[Nainterceptor]

Hello,



I've just found the solution : inet6 loopback was missing.

Please add "iface lo inet6 loopback" in /etc/network/interfaces to solve this issue

Have a nice day,

 

[wbumiller]

That seems weird. I don't have that line on any of my systems and they all get the ::1 address configured.
In fact, if I put the interface down, remove the loopback addresses from it and then simply do ip link set lo up it'll automatically get both addresses configured on its own.
Do you have any special settings on that system?

 

[Nainterceptor]

Hi,

I think I have the real issue in another place.
This fix work on CT Debian and fix outcoming trafic only. I have some timeouts incoming.
On CentOS CT, All things seems to be perfect.
On Archlinux CT, It's a nightmare.

I've done a diff between sysctl.conf of centos and debian, it's strictly identical. I've done a diff between ip -6 route, it's seems to be the same (after some additions/removings, the issue is still here)
I've started my setup on CentOS, but it's true, I prefer debian, so if we can fix that thing...

Regards,
Gaël

 

[wbumiller]

Well, what kind of template did you use and what happens on a fresh standard debian template?

 

[Nainterceptor]

Hello,
I use debian-8.0-standard_8.0-1_amd64.tar.gz, and with this template without any modification, with IPv6 only on eth0 linked to vmbr0, I'm in the case described in the first message. When I add inet6 loopback, I'm in the case described in my previous message.

 

[wbumiller]

I can't reproduce this.

# pct create 407 iso-templates:vztmpl/debian-8.0-standard_8.0-1_amd64.tar.gz -hostname test6 -password 12345 -net0 name=eth0,bridge=vmbr2 -rootfs 4
Formatting '/var/lib/vz/images/407/vm-407-disk-1.raw', fmt=raw size=4294967296
mke2fs 1.42.12 (29-Aug-2014)
Discarding device blocks: done
Creating filesystem with 1048576 4k blocks and 262144 inodes
Filesystem UUID: 487918e9-e57e-47fa-bd0b-e2a14b423737
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Multiple mount protection is enabled with update interval 5 seconds.
Writing superblocks and filesystem accounting information: done

extracting archive '/mnt/pve/iso-templates/template/cache/debian-8.0-standard_8.0-1_amd64.tar.gz'
Total bytes read: 533012480 (509MiB, 184MiB/s)
Detected container architecture: amd64
Creating SSH host key 'ssh_host_key' - this may take some time ...
Creating SSH host key 'ssh_host_ecdsa_key' - this may take some time ...
Creating SSH host key 'ssh_host_ed25519_key' - this may take some time ...
Creating SSH host key 'ssh_host_rsa_key' - this may take some time ...
Creating SSH host key 'ssh_host_dsa_key' - this may take some time ...
# pct start 407
# pct enter 407
root@test6:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback
root@test6:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
40: eth0@if41: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 62:34:33:32:37:33 brd ff:ff:ff:ff:ff:ff
root@test6:~# exit
# pct stop 407
# pct destroy 407

As you can see the ::1 loopback address shows up.
Note that not all sysctl values propagate to the container. For instance I have:
net.ipv6.conf.default.disable_ipv6 = 1

Even tried the same with:
net.ipv6.conf.lo.disable_ipv6 = 1

 

[Nainterceptor]

Hello,

I use an Ansible script, you can find on my repo, all configurations steps for my proxmox 4 installed by Online.net installer : https://github.com/Nainterceptor/Ansible-playbooks
My IPv6 on my VM : prefix::100/64
Gateway : prefix::1

(Created from web gui)

Formatting '/var/lib/vz/images/100/vm-100-disk-1.raw', fmt=raw size=8589934592
mke2fs 1.42.12 (29-Aug-2014)
Discarding device blocks: 4096/2097152 done
Creating filesystem with 2097152 4k blocks and 524288 inodes
Filesystem UUID: 28e36d54-ac41-45d8-9a1e-d0c8dc6624de
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: 0/64 done
Writing inode tables: 0/64 done
Creating journal (32768 blocks): done
Multiple mount protection is enabled with update interval 5 seconds.
Writing superblocks and filesystem accounting information: 0/64 done

extracting archive '/var/lib/vz/template/cache/debian-8.0-standard_8.0-1_amd64.tar.gz'
Total bytes read: 533012480 (509MiB, 169MiB/s)
Detected container architecture: amd64
Creating SSH host key 'ssh_host_rsa_key' - this may take some time ...
Creating SSH host key 'ssh_host_ecdsa_key' - this may take some time ...
Creating SSH host key 'ssh_host_dsa_key' - this may take some time ...
Creating SSH host key 'ssh_host_key' - this may take some time ...
Creating SSH host key 'ssh_host_ed25519_key' - this may take some time ...

# pct start 100
# pct enter 100
cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet6 static
    address prefix::100
    netmask 64
    gateway prefix::1
ip -6 a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
71: eth0@if72: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qlen 1000
    inet6 prefix::100/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::3065:61ff:fe64:3563/64 scope link
       valid_lft forever preferred_lft forever

But the issue is not really here. In fact, at this point, request like wget or apt-get will timeout or take infinite time.

root@test:/# apt-get update
Get:1 http://security.debian.org jessie/updates InRelease [63.1 kB]
Get:2 http://security.debian.org jessie/updates/main amd64 Packages [148 kB]
Get:3 http://security.debian.org jessie/updates/contrib amd64 Packages [2512 B]
Get:4 http://security.debian.org jessie/updates/contrib Translation-en [1211 B]
Get:5 http://security.debian.org jessie/updates/main Translation-en [82.5 kB]  
100% [Waiting for headers]^C                                                   
root@test:/# ping6 google.com
PING google.com(fra02s18-in-x06.1e100.net) 56 data bytes
64 bytes from fra02s18-in-x06.1e100.net: icmp_seq=1 ttl=54 time=11.1 ms
64 bytes from fra02s18-in-x06.1e100.net: icmp_seq=2 ttl=54 time=11.2 ms
^C
--- google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 11.131/11.195/11.259/0.064 ms
root@test:/# wget ping6.online.net/100Mo.dat
converted 'http://ping6.online.net/100Mo.dat' (ANSI_X3.4-1968) -> 'http://ping6.online.net/100Mo.dat' (UTF-8)
--2015-11-18 15:15:35--  http://ping6.online.net/100Mo.dat
Resolving ping6.online.net (ping6.online.net)... 2001:bc8:1::40
Connecting to ping6.online.net (ping6.online.net)|2001:bc8:1::40|:80... connected.
HTTP request sent, awaiting response... ^C

If I add the iface lo inet6 loopback in my interfaces file :

root@test:/# nano /etc/network/interfaces
root@test:/# service networking restart
[....] Running /etc/init.d/networking restart is deprecated because it may not r[warnble some interfaces ... (warning).
[....] Reconfiguring network interfaces...Waiting for DAD... Done
done.
root@test:/# cat /etc/network/interfaces
auto lo
iface lo inet loopback
iface lo inet6 loopback

auto eth0
iface eth0 inet6 static
    address prefix::100
    netmask 64
    gateway prefix::1

root@test:/# ip -6 a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
71: eth0@if72: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qlen 1000
    inet6 prefix::100/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::3065:61ff:fe64:3563/64 scope link
       valid_lft forever preferred_lft forever

So, the previous commands will success.

root@test:/# apt-get update
Hit http://security.debian.org jessie/updates InRelease
Ign http://ftp.debian.org jessie InRelease
Get:1 http://ftp.debian.org jessie-updates InRelease [135 kB]
Get:2 http://ftp.debian.org jessie Release.gpg [2373 B]      
Get:3 http://ftp.debian.org jessie Release [148 kB]                            
Hit http://security.debian.org jessie/updates/main amd64 Packages
Get:4 http://ftp.debian.org jessie-updates/contrib amd64 Packages [32 B]
Hit http://security.debian.org jessie/updates/contrib amd64 Packages
Hit http://security.debian.org jessie/updates/contrib Translation-en
Get:5 http://ftp.debian.org jessie-updates/contrib Translation-en [14 B]
Get:6 http://ftp.debian.org jessie-updates/main Translation-en [2506 B]
Hit http://security.debian.org jessie/updates/main Translation-en
Get:7 http://ftp.debian.org jessie/main amd64 Packages [6764 kB]            
Get:8 http://ftp.debian.org jessie/contrib amd64 Packages [50.1 kB]
Get:9 http://ftp.debian.org jessie/contrib Translation-en [38.4 kB]
Get:10 http://ftp.debian.org jessie/main Translation-en [4585 kB]          
Get:11 http://ftp.debian.org jessie-updates/main amd64 Packages [3452 B]    
Fetched 11.7 MB in 2s (5737 kB/s)                                              
Reading package lists... Done

Oh. When I'm writing this lines, my fix no longer seems to work. Happiness. My apt-get works but the timeout occurs on my wget. Always have (with or without this "fix") timeout over incoming connections (like SSH from my computer from my ipV6 public IP, ping6 works)

So I'm lost, I don't know what is the issue.

More informations :

root@test:/# ip -6 route
prefix::/64 dev eth0  proto kernel  metric 256
fe80::/64 dev eth0  proto kernel  metric 256
default via prefix::1 dev eth0  metric 1024

root@test:/# traceroute6 google.com
traceroute to google.com (2a00:1450:4001:807::1006), 30 hops max, 80 byte packets
 1  prefix::1 (prefix::1)  0.020 ms  0.005 ms  0.004 ms
 2  2001:bc8:2::1:142:1 (2001:bc8:2::1:142:1)  0.735 ms  0.727 ms  0.853 ms
 3  2001:bc8:0:1::115 (2001:bc8:0:1::115)  0.726 ms  0.842 ms 2001:bc8:0:1::119 (2001:bc8:0:1::119)  0.835 ms
 4  google2.franceix.net (2001:7f8:54::2)  0.893 ms  0.890 ms  0.885 ms
 5  2001:4860::1:0:9f2 (2001:4860::1:0:9f2)  47.238 ms 2001:4860::1:0:4a3a (2001:4860::1:0:4a3a)  1.396 ms google.45x-1.pni.th2.poneytelecom.eu (2001:bc8:400:2::17)  0.880 ms
 6  2001:4860::1:0:9f2 (2001:4860::1:0:9f2)  47.288 ms  46.166 ms 2001:4860::8:0:5e18 (2001:4860::8:0:5e18)  1.236 ms
 7  2001:4860::8:0:5e18 (2001:4860::8:0:5e18)  1.100 ms  1.337 ms 2001:4860::8:0:5e19 (2001:4860::8:0:5e19)  27.346 ms
 8  2001:4860::1:0:abf5 (2001:4860::1:0:abf5)  13.114 ms 2001:4860::8:0:abf2 (2001:4860::8:0:abf2)  11.325 ms 2001:4860::8:0:abf3 (2001:4860::8:0:abf3)  11.720 ms
 9  2001:4860::1:0:abf5 (2001:4860::1:0:abf5)  12.993 ms  13.092 ms 2001:4860:0:1::6ed (2001:4860:0:1::6ed)  11.117 ms
10  2a00:1450:4001:807::a (2a00:1450:4001:807::a)  11.111 ms * 2001:4860:0:1::6ed (2001:4860:0:1::6ed)  11.532 ms

 

[wbumiller]

Maybe your gateway has random hiccups?

 

[Nainterceptor]

RAS from my centos with the same configuration, So I think my gateway is clean.

(yum update ; wget is working at 110MB/s)