[SOLVED] upgrade process hangs

A

avidesh

Member
Dec 28, 2023
51
2
8
I have been running Virtual Environment 8.4.1 on a custom mini pc.
I have faced issues during upgrade once is a while.
When I faced the issue today, I tried to investigate further.
Here what I observed:
the processes runs ok until it reached read change log step. it is getting stuck at 88% during my 3-4 attempts.

Code: 
Starting system upgrade: apt-get dist-upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  gnutls-bin grub-common grub-efi-amd64 grub-efi-amd64-bin grub-efi-amd64-signed
  grub-pc-bin grub2-common libgnutls-dane0 libgnutls30 libgnutlsxx30
  libpve-common-perl proxmox-archive-keyring proxmox-grub pve-container pve-firewall
  pve-qemu-kvm qemu-server
17 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/42.2 MB of archives.
After this operation, 236 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Reading changelogs... 88%

I then went to pve updates menu where each up-gradable package is listed. selecting individual package and then clicking on change log button at the top menu bar, I could confirm that change log was accessible for all the proxmox packages. however I was getting an error for all the debian packages.


Code: 
E: Failed to fetch https://metadata.ftp-master.debian.org/changelogs/main/g/gnutls28/gnutls28_3.7.9-2%2bdeb12u5_changelog  Changelog unavailable for gnutls28=3.7.9-2+deb12u5 (404  Not Found [IP: 167.82.62.132 443])
RC: 100

Is that why my upgrade process is getting stuck?

I tried to disable my firewall, but faces the same error.

Is that a unique thing for me. i.e. Have I made some mistakes in my setup or is this a genuine issue?

Avinash
 
additional information:
when the upgrade process gets stuck, the proxmox instance become unavailable. I need to reboot the server for it to work again.
 
Hi,
do you see any error messages in your system logs/journal? What if you run ps faxl from another shell, maybe we can better see where the process is stuck there. Do you have enough disk space/are the disks healthy?
 
hi,
did not notice any error messages in journal. here is the output of ps faxl before the system hangs.
 

Attachments

avidesh said:
hi,
did not notice any error messages in journal. here is the output of ps faxl before the system hangs.
Click to expand...
Unfortunately, the output is horizontally truncated, so the interesting bits (tasks below apt-get) are missing.
 
Was this while it was hanging?
Code: 
4     0 1458046 1458045  20   0  67716 61280 do_sel S+   pts/3      0:00                  \_ apt-get dist-upgrade
4    42 1458105 1458046  20   0  18796  8832 do_sys S+   pts/3      0:00                      \_ /usr/lib/apt/methods/http
1     0    1024       1  20   0 229136 120112 hrtime Ss  ?          0:16 pve-ha-crm
This doesn't look very surprising (but if it was hanging, it might be network-related).

Last time there were many more tasks below apt-get in the tree (just not readable):
Code: 
4     0  500698  500697  20   0  70384 61772 do_wai S+   pts/0      0:00  |               \_ apt-get 
0     0  500971  500698  20   0   2580  1536 do_wai S+   pts/0      0:00  |                   \_ /bin
0     0  500972  500971  20   0  29868 21436 do_wai S+   pts/0      0:00  |                       \_ 
0     0  501364  500972  20   0   2580  1536 do_wai S+   pts/0      0:00  |                          
0     0  501365  501364  20   0   4336  1792 do_wai S+   pts/0      0:00  |                          
1     0  501367  501365  20   0   4336   384 pipe_w S+   pts/0      0:00  |                          
1     0  501368  501365  20   0 372504 136320 futex_ Sl+ pts/0      0:00  |                          
0     0  501366  501364  20   0   3808  2304 pipe_r S+   pts/0      0:00  |                          
1     0    1022    1019  20   0 251208 152656 do_sel S   ?          0:03  \_ pvedaemon worker
 
fiona said:
Was this while it was hanging?
Code: 
4     0 1458046 1458045  20   0  67716 61280 do_sel S+   pts/3      0:00                  \_ apt-get dist-upgrade
4    42 1458105 1458046  20   0  18796  8832 do_sys S+   pts/3      0:00                      \_ /usr/lib/apt/methods/http
1     0    1024       1  20   0 229136 120112 hrtime Ss  ?          0:16 pve-ha-crm
This doesn't look very surprising (but if it was hanging, it might be network-related).

Last time there were many more tasks below apt-get in the tree (just not readable):
Code: 
4     0  500698  500697  20   0  70384 61772 do_wai S+   pts/0      0:00  |               \_ apt-get
0     0  500971  500698  20   0   2580  1536 do_wai S+   pts/0      0:00  |                   \_ /bin
0     0  500972  500971  20   0  29868 21436 do_wai S+   pts/0      0:00  |                       \_
0     0  501364  500972  20   0   2580  1536 do_wai S+   pts/0      0:00  |                         
0     0  501365  501364  20   0   4336  1792 do_wai S+   pts/0      0:00  |                         
1     0  501367  501365  20   0   4336   384 pipe_w S+   pts/0      0:00  |                         
1     0  501368  501365  20   0 372504 136320 futex_ Sl+ pts/0      0:00  |                         
0     0  501366  501364  20   0   3808  2304 pipe_r S+   pts/0      0:00  |                         
1     0    1022    1019  20   0 251208 152656 do_sel S   ?          0:03  \_ pvedaemon worker
Click to expand...
network seems to be ok. everything else is accessible. even the apt-get process was able to download the change log for proxmox packages. I think it is unable to get change log for the Debian packages which are part of the upgrade. when I try to use 'change log' menu on the datacenter->pve-> updates page (UI) I get an error as follows

Code: 
E: Failed to fetch https://metadata.ftp-master.debian.org/changelogs/main/g/gnutls28/gnutls28_3.7.9-2%2bdeb12u5_changelog  Changelog unavailable for gnutls28=3.7.9-2+deb12u5 (404  Not Found [IP: 167.82.62.132 443])
RC: 100

When I check that portal, I see that the change log file is not present at the URL which the apt-get dist-upgrade process is trying to access. Not sure if that is the cause. Seems a bit odd though.
 
additional data point, the same proxmox server has a lxc that uses Debian 12 base. apt dist-upgrade works fine there.
 
avidesh said:
I think it is unable to get change log for the Debian packages which are part of the upgrade. when I try to use 'change log' menu on the datacenter->pve-> updates page (UI) I get an error as follows
Click to expand...
I've always had a similar experience with PVE where 90% of the time I'm unable to get the change logs for Debian-sourced updates but always get the Proxmox-sourced ones. I just go searching for them manually on the Web. I've never been concerned by it, as I put it down to some metadata unavailable at the time since the upgrade procedure always works.

Over the years, threads/posts crop up on this issue. Here & here are examples from last year.

I would also be happy with a permanent solution.
 
Hi,

can you provide the configuration of your apt sources?
Not all debian mirrors are "equal", some maybe differing in small details.

For that reason it might also be feasible your LXC-instance sucessfully updates, while your host is not.

Have you tried, to port the LXC mirror configuration, to the host? Or the one from the official iso?

Was the Mini-PC installed from a Debian baseimage?
That installer gives you the freedom to choose your mirror during the installation process.

BR, Lucas
 
here it is
Code: 
root@pve:~# more /etc/apt/sources.list
deb http://deb.debian.org/debian bookworm main contrib
deb http://deb.debian.org/debian bookworm-updates main contrib
deb http://security.debian.org/debian-security bookworm-security main contrib

no, I have not tried to port the LXC mirror to the host.
I used official Proxmox installation iso for the MiniPC base.
 
I have the same Debian ones:
Code: 
~# cat /etc/apt/sources.list
deb http://ftp.debian.org/debian bookworm main contrib
deb http://ftp.debian.org/debian bookworm-updates main contrib
deb http://security.debian.org/debian-security bookworm-security main contrib
 
mhm ok, that both looks fine on the first view.

Like the debian documentations states with
Code: 
http://deb.debian.org/debian
the content delivery network is utilized to select a mirror, but anyways the error reports a missing changelog pointing to the https://metadata.ftp-master.debian.org server.

Maybe removing the by default installed apt-listchanges package helps.

Code: 
aptitude remove apt-listchanges

BR, Lucas
 
  • Like
Reactions: avidesh
bl1mp said:
mhm ok, that both looks fine on the first view.

Like the debian documentations states with
Code: 
http://deb.debian.org/debian
the content delivery network is utilized to select a mirror, but anyways the error reports a missing changelog pointing to the https://metadata.ftp-master.debian.org server.

Maybe removing the by default installed apt-listchanges package helps.

Code: 
aptitude remove apt-listchanges

BR, Lucas
Click to expand...
@bl1mp, thanks for that. this one helped a bit. this enabled the upgrade process to proceed further. It got stuck for some other reasons which I was able to fix later. So for all practical reasons, I'll mark this one as solved. thank you everyone.
 
You must log in or register to reply here.
Top Bottom