Unable to "apt update" (proxmox apt repos hang)

nakamorichi

New Member
Aug 12, 2024
7
0
1
I feel I've tried everything, and am honestly out of ideas; for some reason I simply can't access the apt repositories, e.g. "deb http://download.proxmox.com/debian/pbs-client bookworm main". I'm able to ping the repos, e.g.:

Code:
root@proxmox:~# ping6 download.proxmox.com
PING download.proxmox.com(sg.cdn.proxmox.com (2402:1f00:8001:f7a::65)) 56 data bytes
64 bytes from sg.cdn.proxmox.com (2402:1f00:8001:f7a::65): icmp_seq=1 ttl=52 time=86.2 ms
64 bytes from sg.cdn.proxmox.com (2402:1f00:8001:f7a::65): icmp_seq=2 ttl=52 time=86.2 ms
^C
--- download.proxmox.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 86.187/86.190/86.194/0.003 ms
root@proxmox:~# ping -4 download.proxmox.com
PING  (51.79.228.122) 56(84) bytes of data.
64 bytes from sg.cdn.proxmox.com (51.79.228.122): icmp_seq=1 ttl=52 time=87.9 ms
64 bytes from sg.cdn.proxmox.com (51.79.228.122): icmp_seq=2 ttl=52 time=88.3 ms
^C
---  ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 87.888/88.088/88.289/0.200 ms

But when trying to simply run apt update, the command just freezes with 0% [Waiting for headers]

Debian repos work, so the issue is only about proxmox apt repos.

I've had similar issue previously with e.g. default Ubuntu apt repos, but was able to fix the issue by switching to mirrors. However, I couldn't find any Proxmox mirrors, so I guess I'm out of luck..?
 
I've had similar issue previously with e.g. default Ubuntu apt repos, but was able to fix the issue by switching to mirrors.
On a hunch - is there a (corporate) firewall configured that might block/intercept HTTP requests? Or do you a proxy configured?

What does http://sg.cdn.proxmox.com/ report?
 
On a hunch - is there a (corporate) firewall configured that might block/intercept HTTP requests? Or do you a proxy configured?

What does http://sg.cdn.proxmox.com/ report?
No corporate firewall, but a self-built one running opnsense. However, there are no logs indicating anything being blocked, and I don't have out-of-ordinary NAT etc. rules either. Also, considering that other apt repos work, I can't quite figure out what could be causing the issue. Something ridiculous, like my source IP range being blocked by Proxmox CDN..? I don't currently have an easy way to test with different source IP, so I'm just wondering whether there are any - even unofficial - mirrors that I could try?

Here's the output:
Code:
root@proxmox:~# apt update
Hit:1 https://cdn-fastly.deb.debian.org/debian bookworm InRelease
Hit:2 https://cdn-fastly.deb.debian.org/debian bookworm-updates InRelease
Hit:3 https://cdn-fastly.deb.debian.org/debian-security bookworm-security InRelease
Ign:4 http://sg.cdn.proxmox.com bookworm InRelease
Ign:4 http://sg.cdn.proxmox.com bookworm InRelease
Ign:4 http://sg.cdn.proxmox.com bookworm InRelease
Err:4 http://sg.cdn.proxmox.com bookworm InRelease
  Connection failed [IP: 51.79.228.122 80]
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
W: Failed to fetch http://sg.cdn.proxmox.com/dists/bookworm/InRelease  Connection failed [IP: 51.79.228.122 80]
W: Some index files failed to download. They have been ignored, or old ones used instead.
 
Can you curl the URL or open it inside a web browser? Should give you a HTML of the nginx file server type
 
Code:
root@proxmox:~# curl -L http://sg.cdn.proxmox.com -v
*   Trying [2402:1f00:8001:f7a::65]:80...
* Connected to sg.cdn.proxmox.com (2402:1f00:8001:f7a::65) port 80 (#0)
> GET / HTTP/1.1
> Host: sg.cdn.proxmox.com
> User-Agent: curl/7.88.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx
< Date: Mon, 12 Aug 2024 16:42:12 GMT
< Content-Type: text/html
< Transfer-Encoding: chunked
< Connection: keep-alive
<
<html>
<head><title>Index of /</title></head>
<body>
<h1>Index of /</h1><hr><pre><a href="../">../</a>
<a href="debian/">debian/</a>                                            08-Sep-2023 11:47                   -
<a href="images/">images/</a>                                            07-May-2024 09:19                   -
<a href="iso/">iso/</a>                                               25-Apr-2024 12:24                   -
<a href="temp/">temp/</a>                                              11-Jul-2024 07:05                   -
</pre><hr></body>
</html>

Code:
root@proxmox:~# curl -L http://sg.cdn.proxmox.com/dists/bookworm/InRelease -v
*   Trying [2402:1f00:8001:f7a::65]:80...
* Connected to sg.cdn.proxmox.com (2402:1f00:8001:f7a::65) port 80 (#0)
> GET /dists/bookworm/InRelease HTTP/1.1
> Host: sg.cdn.proxmox.com
> User-Agent: curl/7.88.1
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Server: nginx
< Date: Mon, 12 Aug 2024 16:42:35 GMT
< Content-Type: text/html
< Content-Length: 146
< Connection: keep-alive
<
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

btw, is the 404 normal?
 
Code:
root@proxmox:~# curl -L http://sg.cdn.proxmox.com -v
*   Trying [2402:1f00:8001:f7a::65]:80...
* Connected to sg.cdn.proxmox.com (2402:1f00:8001:f7a::65) port 80 (#0)
> GET / HTTP/1.1
> Host: sg.cdn.proxmox.com
> User-Agent: curl/7.88.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx
< Date: Mon, 12 Aug 2024 16:42:12 GMT
< Content-Type: text/html
< Transfer-Encoding: chunked
< Connection: keep-alive
<
<html>
<head><title>Index of /</title></head>
<body>
<h1>Index of /</h1><hr><pre><a href="../">../</a>
<a href="debian/">debian/</a>                                            08-Sep-2023 11:47                   -
<a href="images/">images/</a>                                            07-May-2024 09:19                   -
<a href="iso/">iso/</a>                                               25-Apr-2024 12:24                   -
<a href="temp/">temp/</a>                                              11-Jul-2024 07:05                   -
</pre><hr></body>
</html>
Looks like Proxmox's side also works fine.
Code:
root@proxmox:~# curl -L http://sg.cdn.proxmox.com/dists/bookworm/InRelease -v
*   Trying [2402:1f00:8001:f7a::65]:80...
* Connected to sg.cdn.proxmox.com (2402:1f00:8001:f7a::65) port 80 (#0)
> GET /dists/bookworm/InRelease HTTP/1.1
> Host: sg.cdn.proxmox.com
> User-Agent: curl/7.88.1
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Server: nginx
< Date: Mon, 12 Aug 2024 16:42:35 GMT
< Content-Type: text/html
< Content-Length: 146
< Connection: keep-alive
<
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

btw, is the 404 normal?
I get that too, and I have no update issues.
 
Arent your apt sources completely wrong?
PVE repos are supposed to be:

Code:
deb http://ftp.debian.org/debian bookworm main contrib
deb http://ftp.debian.org/debian bookworm-updates main contrib

# security updates
deb http://security.debian.org/debian-security bookworm-security main contrib

# PVE Enterprise
deb https://enterprise.proxmox.com/debian/pve bookworm pve-enterprise
# OR
# Proxmox VE pve-no-subscription repository provided by proxmox.com,
# NOT recommended for production use
deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription
 
Arent your apt sources completely wrong?
PVE repos are supposed to be:

Code:
deb http://ftp.debian.org/debian bookworm main contrib
deb http://ftp.debian.org/debian bookworm-updates main contrib

# security updates
deb http://security.debian.org/debian-security bookworm-security main contrib

# PVE Enterprise
deb https://enterprise.proxmox.com/debian/pve bookworm pve-enterprise
# OR
# Proxmox VE pve-no-subscription repository provided by proxmox.com,
# NOT recommended for production use
deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription
Why would they be "wrong"? I'm merely using geographically close Debian mirrors and no-subscription PVE repos.
 
Last edited:
Have you also tried manually selecting a different server (that might be further away but also has different paths and the like) for your proxmox downloads?
For example de.cdn.proxmox.com
deb http://de.cdn.proxmox.com/debian/pve bookworm pve-no-subscription
 
The issue was after all due to my local network setup. Silly amateurish mistake; I recently changed setting in opnsense from a single LAN to dual LAN, and forgot to adjust NAT settings accordingly. I thought the issue was due to DNS, but totally forgot to check NATting. The reason why the issue was only with proxmox machine turned out to be simply due to other machines being in different opnsense LAN network.. ^^; Also the reason why ipv6 connections work fine was naturally due to ipv6 not being NATted.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!