ISO download from URL is not working

[joace]

I have a 5-node cluster setup on-prem and behind a corp proxy, it's configured correctly under datacenter options, I can update system or download CT templates without problem. But if I tried to download an ISO from URL, it always failed with timeout error (clicking Query URL button), if start to download directly, eventually also failed, from job output, looks like it tried not to use configured proxy, definitely it would fail. Can you help me nail down where is the probelm and how to fix? Thanks!

BTW, If I set the same proxy env var under console and use wget to download iso, it worked.

ISO Link: https://mirror.fcix.net/ubuntu-releases/22.10/ubuntu-22.10-live-server-amd64.iso
Tried both verify or don't verify cert





Syslog for this failure:
Apr 12 14:13:10 adtpven1 pvedaemon[2267056]: download failed: Connecting to mirror.fcix.net (mirror.fcix.net)|2620:13b:0:1000::16|:443... failed: Network is unreachable.
Apr 12 14:13:10 adtpven1 pvedaemon[2184188]: <root@pam> end task UPID:adtpven1:002297B0:1C0F1482:64370244:download:ubuntu-22.10-live-server-amd64.iso:root@pam: download failed: Connecting to mirror.fcix.net (mirror.fcix.net)|2620:13b:0:1000::16|:443... failed: Network is unreachable.
Apr 12 14:13:36 adtpven1 pvedaemon[2184188]: <root@pam> successful auth for user 'root@pam'

Below is my PVE package version list:

proxmox-ve: 7.3-1 (running kernel: 5.15.85-1-pve)
pve-manager: 7.3-6 (running version: 7.3-6/723bb6ec)
pve-kernel-helper: 7.3-4
pve-kernel-5.15: 7.3-2
pve-kernel-5.13: 7.1-9
pve-kernel-5.15.85-1-pve: 5.15.85-1
pve-kernel-5.15.64-1-pve: 5.15.64-1
pve-kernel-5.15.53-1-pve: 5.15.53-1
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph: 16.2.9-pve1
ceph-fuse: 16.2.9-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.3
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.3-1
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-2
libpve-guest-common-perl: 4.2-3
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.3-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-1
lxcfs: 5.0.3-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.3.3-1
proxmox-backup-file-restore: 2.3.3-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.1-1
proxmox-widget-toolkit: 3.5.5
pve-cluster: 7.3-2
pve-container: 4.4-2
pve-docs: 7.3-1
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-7
pve-firmware: 3.6-3
pve-ha-manager: 3.5.1
pve-i18n: 2.8-2
pve-qemu-kvm: 7.1.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.3-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+2
vncterm: 1.7-1
zfsutils-linux: 2.1.9-pve1

 

[Moayad]

Hi,

Apr 12 14:13:10 adtpven1 pvedaemon[2267056]: download failed: Connecting to mirror.fcix.net (mirror.fcix.net)|2620:13b:0:1000::16|:443... failed: Network is unreachable.
Apr 12 14:13:10 adtpven1 pvedaemon[2184188]: <root@pam> end task UPID:adtpven1:002297B0:1C0F1482:64370244:download:ubuntu-22.10-live-server-amd64.iso:root@pam: download failed: Connecting to mirror.fcix.net (mirror.fcix.net)|2620:13b:0:1000::16|:443... failed: Network is unreachable.

Can you download any other small ISO for example VirtIO this is a small ISO file - If yes, and no issue, I would check the network on the nodes, since the error said (Network is unreachable)? Another thing you can check for it is, during downloading the ISO file do you see in the Syslog any error related to the Corosync or network between the nodes?

 

[joace]

Thanks for reply! Yes, I tried with smaller size ISO, but the same result. On network, the node has multiple NICs used, e.g., dedicated for Corosync, Ceph storage, Ceph public, but only 1 uplink with external gateway, which has been configured correctly, as node system update or CT template download work fine.

One observation of different behavior between downloading CT template and ISO to the same storage: the CT template download seems correctly connected through proxy and retrieved required tar.xz, but ISO download was not trying to go through proxy, instead directly.

Download CT template:


Download ISO:


Also double checked there is no other warning, error seen in the syslog, only failure on ISO download which posted in OP. And we have another 3-node cluster has the exact same issue.

 

[joace]

bump

 

[joace]

bump again if other check I can try, thanks!

 

[Moayad]

Hello,

Since I can't replicate it, can I ask you for your setup for the proxy the Proxmox VE uses?

 

[joace]

sure, below is the proxy we set in datacenter options, proxy is provided by corp IT using an internal dns, other than that, no other configuration made on the nodes.

 

[darac]

Having a dig through the code, I believe that the problem is that the proxy is being applied for http:// urls, but not https:// URLs.

The "Download from URL" dialog seems to end up at this function call - https://github.com/proxmox/pve-mana...d892098fe865557c96b40/PVE/API2/Nodes.pm#L1617

This calls the `download_file_from_url` function and sets the `http_proxy` option. However, the function itself (defined here - https://github.com/proxmox/pve-comm...ded13adc6772d440d8a804/src/PVE/Tools.pm#L2030) takes an `http_proxy` and an `https_proxy` option.

This would explain why your CT image was downloaded (using an http URL) but your ISO wasn't (using an https URL).

https://bugzilla.proxmox.com/show_bug.cgi?id=3716 is the relevant bug (raised in 2021 with a one-line patch attached).