PVE hosts without IPv6 connectivity still try to use IPv6

[ahorner]

I have an issue with two Proxmox hosts which are misbehaving when establishing connections with pretty much anything. My own applications, apt, curl, ping, you name it.

Both on the host and within LXC containers, things keep attempting to connect via IPv6, even though no IPv6 service is available:

:~# apt update
Hit:1 http://download.proxmox.com/debian/pve bookworm InRelease
Get:2 https://pkgs.tailscale.com/stable/debian bookworm InRelease
Get:3 http://security.debian.org bookworm-security InRelease [48.0 kB]      
Get:4 http://security.debian.org bookworm-security/main amd64 Packages [169 kB]
Ign:5 http://ftp.uk.debian.org/debian bookworm InRelease          
Ign:6 http://ftp.uk.debian.org/debian bookworm-updates InRelease
Err:7 http://ftp.uk.debian.org/debian bookworm Release
  Cannot initiate the connection to ftp.uk.debian.org:80 (2001:1b40:5600:ff80:f8ee::1). - connect (101: Network is unreachable)
Err:8 http://ftp.uk.debian.org/debian bookworm-updates Release
  Cannot initiate the connection to ftp.uk.debian.org:80 (2001:1b40:5600:ff80:f8ee::1). - connect (101: Network is unreachable)
Reading package lists... Done
E: The repository 'http://ftp.uk.debian.org/debian bookworm Release' no longer has a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository 'http://ftp.uk.debian.org/debian bookworm-updates Release' no longer has a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

The DNS server returns both AAAA and A records. There are no default routes configured for IPv6:

:~# ip -6 route show
fd7a:115c:a1e0::3 dev tailscale0 proto kernel metric 256 pref medium
fe80::/64 dev tailscale0 proto kernel metric 256 pref medium
fe80::/64 dev vmbr1000 proto kernel metric 256 pref medium
fe80::/64 dev vmbr1001 proto kernel metric 256 pref medium
fe80::/64 dev vmbr0 proto kernel metric 256 pref medium
fe80::/64 dev vmbr2000 proto kernel metric 256 linkdown pref medium
fe80::/64 dev vmbr95 proto kernel metric 256 pref medium
 
:~# ip route show
default via 10.0.10.1 dev vmbr0 proto kernel onlink
10.0.10.0/24 dev vmbr0 proto kernel scope link src 10.0.10.116

:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp1s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr2000 state DOWN group default qlen 1000
    link/ether ac:16:2d:9a:eb:fc brd ff:ff:ff:ff:ff:ff
3: enp1s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr2001 state DOWN group default qlen 1000
    link/ether ac:16:2d:9a:eb:fd brd ff:ff:ff:ff:ff:ff
4: enp1s0f2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr2002 state DOWN group default qlen 1000
    link/ether ac:16:2d:9a:eb:fe brd ff:ff:ff:ff:ff:ff
5: enp1s0f3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr2003 state DOWN group default qlen 1000
    link/ether ac:16:2d:9a:eb:ff brd ff:ff:ff:ff:ff:ff
6: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000
    link/ether f8:75:a4:5c:60:db brd ff:ff:ff:ff:ff:ff
    altname enp0s31f6
7: wlp3s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 34:cf:f6:a0:8d:1d brd ff:ff:ff:ff:ff:ff
8: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc pfifo_fast state UNKNOWN group default qlen 500
    link/none
    inet 100.64.0.3/32 scope global tailscale0
       valid_lft forever preferred_lft forever
    inet6 fd7a:115c:a1e0::3/128 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::a04b:9259:56f9:7469/64 scope link stable-privacy
       valid_lft forever preferred_lft forever
9: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f8:75:a4:5c:60:db brd ff:ff:ff:ff:ff:ff
    inet 10.0.10.116/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::fa75:a4ff:fe5c:60db/64 scope link
       valid_lft forever preferred_lft forever
10: vmbr1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether b6:cf:59:11:cd:68 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::c4c3:65ff:fe55:1cf2/64 scope link
       valid_lft forever preferred_lft forever
11: vmbr2000: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether ac:16:2d:9a:eb:fc brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ae16:2dff:fe9a:ebfc/64 scope link
       valid_lft forever preferred_lft forever
12: vmbr2001: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether ac:16:2d:9a:eb:fd brd ff:ff:ff:ff:ff:ff
13: vmbr2002: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether ac:16:2d:9a:eb:fe brd ff:ff:ff:ff:ff:ff
14: vmbr2003: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether ac:16:2d:9a:eb:ff brd ff:ff:ff:ff:ff:ff
15: vmbr1001: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 12:91:7f:4b:9e:81 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::1091:7fff:fe4b:9e81/64 scope link
       valid_lft forever preferred_lft forever
...
62: vmbr95: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 8e:d0:a3:8d:81:19 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::84a3:3aff:fe75:6955/64 scope link
       valid_lft forever preferred_lft forever

It takes 2 to 3 attempts to actually get whatever operation is making the request to work, at which point it finally selects IPv4. By attempt, I do mean running the command multiple times or so in the scenarios of apt and curl for example.

I do not wish to disable IPv6 at the system level, as this should be completely unnecessary, other machines are perfectly capable of handling this without having a tantrum.

Any ideas here would be greatly appreciated!

 

[UdoB]

I do not wish to disable IPv6 at the system level

A random search engines first link tells me echo 'Acquire::ForceIPv4 "true";' > /etc/apt/apt.conf.d/99disable-ipv6...

 

[ahorner]

A random search engines first link tells me echo 'Acquire::ForceIPv4 "true";' > /etc/apt/apt.conf.d/99disable-ipv6...

This essentially disables IPv6 connectivity, masking the underlying issue, which as I said is not what I want to do. And also only applies to apt.

 

[sw-omit]

Most likely the reason it goes for ipv6, is that on the bridge-device you have the ipv4 set up with a gateway, you also have an ipv6 set up on.

So the two options I would think you would need would be to:
a) Remove the ipv6 from the vmbr0 (or move it to a different bridge)
b) Set the "default" to ipv4 over ipv6, which would still allow ipv6 if specifically requested (aka does not disable it fully), as by default any system will prefer ipv6 over ipv4 as recommended as per the RFC's. First example of how to do this that I found:
https://weblog.lkiesow.de/20220311-make-linux-prefer-ipv4.html

 

[ahorner]

Most likely the reason it goes for ipv6, is that on the bridge-device you have the ipv4 set up with a gateway, you also have an ipv6 set up on.

So the two options I would think you would need would be to:
a) Remove the ipv6 from the vmbr0 (or move it to a different bridge)
b) Set the "default" to ipv4 over ipv6, which would still allow ipv6 if specifically requested (aka does not disable it fully), as by default any system will prefer ipv6 over ipv4 as recommended as per the RFC's. First example of how to do this that I found:
https://weblog.lkiesow.de/20220311-make-linux-prefer-ipv4.html

This doesn't seem right, as other systems are capable of handling this no issue. A link local address is perfectly normal and is something designated by the system to an interface regardless of connectivity, and is not used for publicly routed network access. There are no default routes for IPv6. I should not have to force IPv4 nor disable IPv6 for this to work, like every other system, it should know it cannot use IPv6.

If you look at any other system which has IPv6 enabled but no IPv6 connectivity, you'll see it has a link local address automatically assigned. You'll also see that regardless of this, the system has no trouble selecting IPv4 without any special configuration.

 

[apalrd]

Just following up here: OP and I did a whole lot of digging and found the issue is caused by sporadic DNS servfail's on his network. When this happens, his DNS server caches the response for 60s, so subsequent 'apt update's get the same servfails and exhibit the same behavior but then succeed a minute later.

The basic order of events is:
* Apt queries _http._tcp.<host> SRV for srv records, gets back nxdomain, so it continues resolving <host> (if it got back anything it would use the srv results instead). We actually noticed a servfail here as well for a different repo, which apt treats the same as nxdomain.
* apt calls getaddrinfo() with the hostname to get a list of address candidates to try (Happy Eyeballs is implemented properly)
* getaddrinfo() calls gethostbyname() twice, once for A and once for AAAA, to resolve the names. AAAA succeeds, A returns servfail
* Since at least one of the queries returned valid, getaddrinfo() sorts the list (only the AAAA record) according to gai.conf and returns it successfully
* apt iterates over the list in order, trying to connect() to each one. On error, it stores the error and tries the next item in the list. On no error, it would have waited 250ms for the connection to reach tcp established before trying connect() on the next item in the list (and leaving the first one open with a chance to establish)
* Since there is no IPv4 address in the list to try (due to servfail), none of the address candidates succeed, so apt returns the error from the last connection attempt (the IPv6 network unreachable)

Forcing apt to use only IPv4 would have made apt call getaddrinfo() with an AF_INET hint, which causes it to only query for A records, and apt would return a failure to resolve instead of an IPv6 error.