download.proxmox.com unreachable

[David Herselman]

Hi,

We've noticed increasing occurrences where we are unable to retrieve updates from download.proxmox.com from South Africa. We get directed towards af.cdn.proxmox.com and are unable to establish a connection to the resulting IP on tcp:80 (http).

[admin@backup1 ~]# host download.proxmox.com
download.proxmox.com is an alias for download.cdn.proxmox.com.
download.cdn.proxmox.com is an alias for af.cdn.proxmox.com.
af.cdn.proxmox.com is an alias for at.cdn.proxmox.com.
at.cdn.proxmox.com has address 94.136.30.185
at.cdn.proxmox.com has IPv6 address 2a01:7e0:0:424::249

[admin@backup1 ~]# telnet download.proxmox.com 80
Trying 94.136.30.185...
Trying 2a01:7e0:0:424::249...
telnet: Unable to connect to remote host: Network is unreachable

Our source IP:

[admin@backup1 ~]# dig +short myip.opendns.com @resolver1.opendns.com
41.79.23.231

[admin@backup1 ~]# apt-get -y dist-upgrade;
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following NEW packages will be installed:
  libpve-cluster-api-perl libpve-cluster-perl pve-kernel-5.0.21-5-pve zfs-zed
The following packages will be upgraded:
  base-files ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-dashboard ceph-mon ceph-osd cron ifupdown intel-microcode libarchive13 libcephfs2
  libfreetype6 libfribidi0 libglib2.0-0 libmariadb3 libncurses5 libncurses6 libncursesw6 libnss-systemd libnvpair1linux libpam-systemd libpve-access-control
  libpve-common-perl libpve-guest-common-perl libpve-storage-perl libpython2.7 libpython2.7-minimal libpython2.7-stdlib librados2 libradosstriper1 librbd1
  librgw2 libsystemd0 libtinfo5 libtinfo6 libudev1 libuutil1linux libxslt1.1 libzfs2linux libzpool2linux linux-libc-dev lxc-pve mariadb-common ncurses-base
  ncurses-bin ncurses-term openvswitch-common openvswitch-switch postfix proxmox-widget-toolkit pve-cluster pve-container pve-docs pve-edk2-firmware
  pve-firewall pve-firmware pve-ha-manager pve-kernel-5.0 pve-kernel-helper pve-manager pve-qemu-kvm python-ceph-argparse python-cephfs python-cryptography
  python-rados python-rbd python-rgw python-werkzeug python2.7 python2.7-minimal qemu-server radosgw rpcbind spl systemd systemd-sysv udev xsltproc
  zfs-initramfs zfsutils-linux
84 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 224 MB of archives.
After this operation, 473 MB of additional disk space will be used.
Get:1 http://security.debian.org buster/updates/main amd64 linux-libc-dev amd64 4.19.67-2+deb10u2 [1,234 kB]
Get:2 http://security.debian.org buster/updates/non-free amd64 intel-microcode amd64 3.20191112.1~deb10u1 [2,326 kB]
Get:3 http://ftp.debian.org/debian buster/main amd64 base-files amd64 10.3+deb10u2 [69.7 kB]
Get:4 http://ftp.debian.org/debian buster/main amd64 ncurses-bin amd64 6.1+20181013-2+deb10u2 [406 kB]
Get:5 http://ftp.debian.org/debian buster/main amd64 ncurses-base all 6.1+20181013-2+deb10u2 [258 kB]
Get:6 http://ftp.debian.org/debian buster/main amd64 libnss-systemd amd64 241-7~deb10u2 [204 kB]
Get:7 http://ftp.debian.org/debian buster/main amd64 libsystemd0 amd64 241-7~deb10u2 [331 kB]
Get:8 http://ftp.debian.org/debian buster/main amd64 libpam-systemd amd64 241-7~deb10u2 [208 kB]
Get:9 http://ftp.debian.org/debian buster/main amd64 systemd amd64 241-7~deb10u2 [3,497 kB]
Get:10 http://ftp.debian.org/debian buster/main amd64 udev amd64 241-7~deb10u2 [1,276 kB]
Get:11 http://ftp.debian.org/debian buster/main amd64 libudev1 amd64 241-7~deb10u2 [150 kB]
Get:12 http://ftp.debian.org/debian buster/main amd64 systemd-sysv amd64 241-7~deb10u2 [99.3 kB]
Get:13 http://ftp.debian.org/debian buster/main amd64 cron amd64 3.0pl1-134+deb10u1 [99.0 kB]
Get:14 http://ftp.debian.org/debian buster/main amd64 libpython2.7 amd64 2.7.16-2+deb10u1 [1,036 kB]
Get:15 http://ftp.debian.org/debian buster/main amd64 python2.7 amd64 2.7.16-2+deb10u1 [305 kB]
Get:16 http://ftp.debian.org/debian buster/main amd64 python2.7-minimal amd64 2.7.16-2+deb10u1 [1,369 kB]
Get:17 http://ftp.debian.org/debian buster/main amd64 libpython2.7-stdlib amd64 2.7.16-2+deb10u1 [1,912 kB]
Get:18 http://ftp.debian.org/debian buster/main amd64 libpython2.7-minimal amd64 2.7.16-2+deb10u1 [395 kB]
Get:19 http://ftp.debian.org/debian buster/main amd64 libncurses6 amd64 6.1+20181013-2+deb10u2 [102 kB]
Get:20 http://ftp.debian.org/debian buster/main amd64 libtinfo6 amd64 6.1+20181013-2+deb10u2 [325 kB]
Get:21 http://ftp.debian.org/debian buster/main amd64 libncursesw6 amd64 6.1+20181013-2+deb10u2 [132 kB]
Get:22 http://ftp.debian.org/debian buster/main amd64 ncurses-term all 6.1+20181013-2+deb10u2 [490 kB]
Get:23 http://ftp.debian.org/debian buster/main amd64 python-werkzeug all 0.14.1+dfsg1-4+deb10u1 [196 kB]
Get:24 http://ftp.debian.org/debian buster/main amd64 libarchive13 amd64 3.3.3-4+deb10u1 [313 kB]
Get:25 http://ftp.debian.org/debian buster/main amd64 libfreetype6 amd64 2.9.1-3+deb10u1 [380 kB]
Get:26 http://ftp.debian.org/debian buster/main amd64 libfribidi0 amd64 1.0.5-3.1+deb10u1 [63.7 kB]
Get:27 http://ftp.debian.org/debian buster/main amd64 libglib2.0-0 amd64 2.58.3-2+deb10u2 [1,258 kB]
Get:28 http://ftp.debian.org/debian buster/main amd64 mariadb-common all 1:10.3.18-0+deb10u1 [31.7 kB]
Get:29 http://ftp.debian.org/debian buster/main amd64 libmariadb3 amd64 1:10.3.18-0+deb10u1 [169 kB]
Get:30 http://ftp.debian.org/debian buster/main amd64 libncurses5 amd64 6.1+20181013-2+deb10u2 [96.2 kB]
Get:31 http://ftp.debian.org/debian buster/main amd64 libtinfo5 amd64 6.1+20181013-2+deb10u2 [321 kB]
Get:32 http://ftp.debian.org/debian buster/main amd64 postfix amd64 3.4.7-0+deb10u1 [1,515 kB]
Get:33 http://ftp.debian.org/debian buster/main amd64 libxslt1.1 amd64 1.1.32-2.2~deb10u1 [237 kB]
Get:34 http://ftp.debian.org/debian buster/main amd64 openvswitch-common amd64 2.10.0+2018.08.28+git.8ca7c82b7d+ds1-12+deb10u1 [2,047 kB]
Get:35 http://ftp.debian.org/debian buster/main amd64 openvswitch-switch amd64 2.10.0+2018.08.28+git.8ca7c82b7d+ds1-12+deb10u1 [50.8 kB]
Get:36 http://ftp.debian.org/debian buster/main amd64 python-cryptography amd64 2.6.1-3+deb10u2 [218 kB]
Get:37 http://ftp.debian.org/debian buster/main amd64 rpcbind amd64 1.2.5-0.3+deb10u1 [47.1 kB]
Get:38 http://ftp.debian.org/debian buster/main amd64 xsltproc amd64 1.1.32-2.2~deb10u1 [122 kB]
Err:39 http://download.proxmox.com/debian buster/pve-no-subscription amd64 ifupdown amd64 0.8.35+pve1
  Cannot initiate the connection to download.proxmox.com:80 (2a01:7e0:0:424::249). - connect (101: Network is unreachable) Could not connect to download.proxmox.com:80 (94.136.30.185), connection timed out [IP: 94.136.30.185 80]
Err:40 http://download.proxmox.com/debian/ceph-nautilus buster/main amd64 radosgw amd64 14.2.4.1-pve1
  Cannot initiate the connection to download.proxmox.com:80 (2a01:7e0:0:424::249). - connect (101: Network is unreachable) [IP: 94.136.30.185 80]
Err:41 http://download.proxmox.com/debian/ceph-nautilus buster/main amd64 python-ceph-argparse all 14.2.4.1-pve1
  Cannot initiate the connection to download.proxmox.com:80 (2a01:7e0:0:424::249). - connect (101: Network is unreachable) [IP: 94.136.30.185 80]
Err:42 http://download.proxmox.com/debian/ceph-nautilus buster/main amd64 ceph-mon amd64 14.2.4.1-pve1
  Cannot initiate the connection to download.proxmox.com:80 (2a01:7e0:0:424::249). - connect (101: Network is unreachable) [IP: 94.136.30.185 80]
Err:43 http://download.proxmox.com/debian/ceph-nautilus buster/main amd64 ceph-mgr-dashboard all 14.2.4.1-pve1
  Cannot initiate the connection to download.proxmox.com:80 (2a01:7e0:0:424::249). - connect (101: Network is unreachable) [IP: 94.136.30.185 80]
Err:44 http://download.proxmox.com/debian/ceph-nautilus buster/main amd64 ceph-mgr amd64 14.2.4.1-pve1
  Cannot initiate the connection to download.proxmox.com:80 (2a01:7e0:0:424::249). - connect (101: Network is unreachable) [IP: 94.136.30.185 80]
Err:45 http://download.proxmox.com/debian/ceph-nautilus buster/main amd64 ceph-mds amd64 14.2.4.1-pve1
  Cannot initiate the connection to download.proxmox.com:80 (2a01:7e0:0:424::249). - connect (101: Network is unreachable) [IP: 94.136.30.185 80]
Err:46 http://download.proxmox.com/debian/ceph-nautilus buster/main amd64 ceph amd64 14.2.4.1-pve1
  Cannot initiate the connection to download.proxmox.com:80 (2a01:7e0:0:424::249). - connect (101: Network is unreachable) [IP: 94.136.30.185 80]
Err:47 http://download.proxmox.com/debian/ceph-nautilus buster/main amd64 ceph-osd amd64 14.2.4.1-pve1
  Cannot initiate the connection to download.proxmox.com:80 (2a01:7e0:0:424::249). - connect (101: Network is unreachable) [IP: 94.136.30.185 80]
<snip>

Regards
David Herselman

 

[tom]

please try again.

if you still see issues, you can try instead of download.proxmox.com you the german mirror de.cdn.proxmox.com as a workaround for now.

 

[David Herselman]

The outages I've hit typically last about an hour and yes, it is working again now... Any chance of automating DNS updates to avoid situations where we aren't able to complete updates during scheduled maintenance events?

Or Proxmox perhaps considering signing up with a forward caching service such as Fastly (relatively cheap and reliable) or Cloudflare?


Regards
David Herselman

 

[tom]

Thanks for reporting - the issue was not caused by a scheduled maintenance - we will analyse the reason and act accordingly.

 

[COMMARMOND Christian]

Hi,

I had the same problem yesterday from France.

Best regards,
Christian

 

[benoitc]

Hi,

I had the same problem yesterday from France.

Best regards,
Christian

same

 

[AvocadoSupreme]

I'm seeing the same issue right now in Eastern USA

--2020-10-22 13:26:46-- http://download.proxmox.com/images/system/ubuntu-20.04-standard_20.04-1_amd64.tar.gz
Resolving download.proxmox.com (download.proxmox.com)... 144.217.225.162, 2607:5300:203:7dc2::162
Connecting to download.proxmox.com (download.proxmox.com)|144.217.225.162|:80... failed: Connection timed out.
Connecting to download.proxmox.com (download.proxmox.com)|2607:5300:203:7dc2::162|:80... failed: Network is unreachable.

----

...and just like that, it works again. Thanks!

 

[MrPete]

download.proxmox.com is dead right now.
I have set up a worldwide ping monitor, in case staff are interested...
* From OH, DE, TX all appears well
* From Melbourne, slow but working
* From my phone ISP - yes
* From my local (Century Link) ISP - no connection


Perhaps an internet backbone outage of sorts.
Traceroute from here gets to Denver Level3 gateway, then nothing.

 

[fabian]

download.proxmox.com is actually more than one system - which one are you resolving to in the problematic location? but yes, likely it's some sort of peering issue between the node you are hitting and your uplink..

 

[MrPete]

na.cdn.proxmox.com

I get a different response - denied - with af...co and eu...co
Any other alternatives? Still dead this AM from MOST of the USA
https://lookingglass.centurylink.com/
- Eastern seaboard (MA, MD, GA) are ok
- Everything else in the USA is not working

 

[fabian]

tried three random US sources from your LG link and some similar sites, all worked just fine. in any case if there is a problem it's likely some issue with routing on your/your ISP's end..

you can try de.cdn and fr.cdn as a temporary workaround, although obviously the latency will be worse than they would be for na (and none of those should be hard-coded long-term, as they are internal endpoints that can vanish without prior notice).

 

[MrPete]

Thanks. Yep, everything appears to be fixed now. That was quite the outage!