Problem with repository access

[czechsys]

Hi,

we have regularly problem with some apt access to repository last weeks.

Err:10 http://download.proxmox.com/debian/ceph-quincy bookworm InRelease
  Could not connect to download.proxmox.com:80 (212.224.123.70), connection timed out
Err:11 http://download.proxmox.com/debian/pve bookworm InRelease
  Unable to connect to download.proxmox.com:http:

;; ANSWER SECTION:

download.proxmox.com.    257    IN    CNAME    download.cdn.proxmox.com.
download.cdn.proxmox.com. 7    IN    CNAME    cz.eu.cdn.proxmox.com.
cz.eu.cdn.proxmox.com.    7    IN    CNAME    de.cdn.proxmox.com.
de.cdn.proxmox.com.    7    IN    A    212.224.123.70

Err:9 http://download.proxmox.com/debian/ceph-quincy bookworm InRelease
  Could not connect to download.proxmox.com:80 (51.91.38.34), connection timed out
Err:10 http://download.proxmox.com/debian/pve bookworm InRelease
  Unable to connect to download.proxmox.com:http:

Please fix it.

 

[Stoiko Ivanov]

Can your hosts connect to the internet in general?

the outputs show that you get timeouts when connecting to 2 of our mirrors - I doubt that both had issues simultaneously

 

[czechsys]

I did another test.

# date
Thu Jun 13 11:37:14 AM CEST 2024

# ping -4 download.proxmox.com
PING  (212.224.123.70) 56(84) bytes of data.
64 bytes from de.cdn.proxmox.com (212.224.123.70): icmp_seq=1 ttl=55 time=8.01 ms
64 bytes from de.cdn.proxmox.com (212.224.123.70): icmp_seq=2 ttl=55 time=7.60 ms
64 bytes from de.cdn.proxmox.com (212.224.123.70): icmp_seq=3 ttl=55 time=7.67 ms
^C
---  ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 7.602/7.762/8.012/0.179 ms

# apt update
Hit:1 http://ftp.cz.debian.org/debian bookworm InRelease
Hit:2 http://ftp.cz.debian.org/debian bookworm-backports InRelease                                                                                                                                           
Ign:9 http://download.proxmox.com/debian/ceph-quincy bookworm InRelease                                                                 
Ign:10 http://download.proxmox.com/debian/pve bookworm InRelease                                                 
Hit:11 http://security.debian.org/debian-security bookworm-security InRelease
Ign:9 http://download.proxmox.com/debian/ceph-quincy bookworm InRelease
Ign:10 http://download.proxmox.com/debian/pve bookworm InRelease
Ign:9 http://download.proxmox.com/debian/ceph-quincy bookworm InRelease
Ign:10 http://download.proxmox.com/debian/pve bookworm InRelease
Err:9 http://download.proxmox.com/debian/ceph-quincy bookworm InRelease
  Could not connect to download.proxmox.com:80 (212.224.123.70), connection timed out
Err:10 http://download.proxmox.com/debian/pve bookworm InRelease
  Unable to connect to download.proxmox.com:http:
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
81 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: Failed to fetch http://download.proxmox.com/debian/ceph-quincy/dists/bookworm/InRelease  Could not connect to download.proxmox.com:80 (212.224.123.70), connection timed out
W: Failed to fetch http://download.proxmox.com/debian/pve/dists/bookworm/InRelease  Unable to connect to download.proxmox.com:http:
W: Some index files failed to download. They have been ignored, or old ones used instead.
N: Repository 'Debian bookworm' changed its 'non-free component' value from 'non-free' to 'non-free non-free-firmware'
N: More information about this can be found online in the Release notes at: https://www.debian.org/releases/bookworm/amd64/release-notes/ch-information.html#non-free-split

# ping -4 download.proxmox.com
PING  (212.224.123.70) 56(84) bytes of data.
64 bytes from de.cdn.proxmox.com (212.224.123.70): icmp_seq=1 ttl=55 time=7.88 ms
64 bytes from de.cdn.proxmox.com (212.224.123.70): icmp_seq=2 ttl=55 time=7.65 ms
64 bytes from de.cdn.proxmox.com (212.224.123.70): icmp_seq=3 ttl=55 time=7.59 ms
64 bytes from de.cdn.proxmox.com (212.224.123.70): icmp_seq=4 ttl=55 time=7.58 ms
64 bytes from de.cdn.proxmox.com (212.224.123.70): icmp_seq=5 ttl=55 time=7.65 ms
64 bytes from de.cdn.proxmox.com (212.224.123.70): icmp_seq=6 ttl=55 time=7.59 ms
^C
---  ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5006ms
rtt min/avg/max/mdev = 7.578/7.655/7.875/0.102 ms

# date
Thu Jun 13 11:38:20 AM CEST 2024

And firewall using acl with allowing fqdn download.proxmox.com:

FWDeny, Denied, pri=4, disp=Deny, policy=Unhandled-Internal-Packet-00, protocol=http/tcp, src_ip=SERVERIP, src_port=40890, dst_ip=212.224.123.70, dst_port=80, src_intf=SERVERVLAN, dst_intf=PUBLICVLAN, rc=101, pckt_len=60, ttl=63, pr_info=offset 10 S 2181458746 win 61690, duration=0; sent_bytes=60; rcvd_bytes=0, 3000-0148

I think there is dns desync translation between firewall and proxmox apt, both are using set of recursors and cdn dns is swapping too fast, sometimes even in seconds.

 

[Stoiko Ivanov]

And firewall using acl with allowing fqdn download.proxmox.com:

you'd need to allow all of our mirrors in that case (or allow outbound traffic from your PVE nodes on port 80 in general ...)
currently traffic that seems to originate from .cz (based on geo-ip lookups) will be distributed to de.cdn.proxmox.com, de2.cdn.proxmox.com, fr.proxmox.com - however this is subject to change without any notice ...

 

[czechsys]

however this is subject to change without any notice ...

Isn't better to check if cdn dns geo/ttl configuration is sane? Rather unnoticed breaking ?

 

[Stoiko Ivanov]

Isn't better to check if cdn dns geo/ttl configuration is sane? Rather unnoticed breaking ?

Not quite sure if I understand the issue completely - but ...
DNS based loadbalancing works by returning you one of a few of the CDN nodes according to their weight and reachability in your region.
So you might get A record pointing to de.cdn.proxmox.com at one point, and later one pointing to fr.cdn.proxmox.com (or any other of the nodes configured for your region). (if one of the nodes is down for maintenance you just don't get this one as a response.

If the need arises we can add more nodes to the CDN and you might get one of the new records instead.

Do you allow outbound traffic on port 80 and 443 from your nodes to the 3 CDN nodes I mentioned above?