[SOLVED] Node won't upgrade to newest version

E

Eilif Johannsen

New Member
Aug 1, 2019
7
0
1
47
I have multiple nodes and one of them won't upgrade to newest version.

The one that won't upgrade is running 4.15.18-12 where the others are running 4.15.18-19.

I see that it's both of these it haven't installed:
pve-kernel-4.15.18-19-pve: 4.15.18-45
pve-kernel-4.15.18-13-pve: 4.15.18-37​

All nodes have been updated previously without problems.
Running apt-get update and apt-get dist-upgrade show that there isn't any updates.

Both servers have been restarted multiple times.

Does someone have an idea how to fix this ?

pveversion -v
Node 3
proxmox-ve: 5.4-1 (running kernel: 4.15.18-12-pve)
pve-manager: 5.3-11 (running version: 5.3-11/d4907f84)
pve-kernel-4.15: 5.3-3
pve-kernel-4.15.18-12-pve: 4.15.18-36
pve-kernel-4.15.18-11-pve: 4.15.18-34
pve-kernel-4.15.18-9-pve: 4.15.18-30
pve-kernel-4.15.18-4-pve: 4.15.18-23
pve-kernel-4.15.17-3-pve: 4.15.17-14
pve-kernel-4.13.13-1-pve: 4.13.13-31
pve-kernel-4.13.8-2-pve: 4.13.8-28
pve-kernel-4.10.17-2-pve: 4.10.17-20
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-51
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-13
libpve-storage-perl: 5.0-42
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-3
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-3
proxmox-widget-toolkit: 1.0-26
pve-cluster: 5.0-37
pve-container: 2.0-37
pve-docs: 5.4-2
pve-edk2-firmware: 1.20190312-1
pve-firewall: 3.0-20
pve-firmware: 2.0-6
pve-ha-manager: 2.0-9
pve-i18n: 1.1-4
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 3.0.1-2
pve-xtermjs: 3.12.0-1
qemu-server: 5.0-51
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.13-pve1~bpo2
​
Node 6
proxmox-ve: 5.4-2 (running kernel: 4.15.18-19-pve)
pve-manager: 5.4-13 (running version: 5.4-13/aee6f0ec)
pve-kernel-4.15: 5.4-7
pve-kernel-4.15.18-19-pve: 4.15.18-45
pve-kernel-4.15.18-13-pve: 4.15.18-37
pve-kernel-4.15.18-12-pve: 4.15.18-36
pve-kernel-4.15.18-11-pve: 4.15.18-34
pve-kernel-4.15.18-9-pve: 4.15.18-30
pve-kernel-4.15.18-4-pve: 4.15.18-23
pve-kernel-4.15.17-3-pve: 4.15.17-14
pve-kernel-4.13.13-1-pve: 4.13.13-31
pve-kernel-4.13.8-2-pve: 4.13.8-28
pve-kernel-4.10.17-2-pve: 4.10.17-20
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-12
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-54
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-14
libpve-storage-perl: 5.0-44
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-3
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-3
proxmox-widget-toolkit: 1.0-28
pve-cluster: 5.0-37
pve-container: 2.0-40
pve-docs: 5.4-2
pve-edk2-firmware: 1.20190312-1
pve-firewall: 3.0-22
pve-firmware: 2.0-7
pve-ha-manager: 2.0-9
pve-i18n: 1.1-4
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 3.0.1-4
pve-xtermjs: 3.12.0-1
qemu-server: 5.0-54
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.13-pve1~bpo2​
 
Do you have the same repositories configured on all nodes?
In
Code: 
/etc/apt/sources.list
and
Code: 
/etc/apt/sources.list.d/*.list
 
please post the complete output from `apt update`, `apt full-upgrade` on the nodes (post them directly into the forum inside code tags)
 
apt update
Node 3:
Code: 
Hit:1 http://security.debian.org stretch/updates InRelease
Ign:2 http://ftp.debian.org/debian stretch InRelease
Hit:3 http://ftp.debian.org/debian stretch Release
Hit:4 http://download.proxmox.com/debian/pve stretch InRelease
Hit:5 http://apt.puppetlabs.com stretch InRelease
Ign:7 http://downloads.linux.hpe.com/SDR/repo/mcp stretch/current InRelease
Hit:8 http://downloads.linux.hpe.com/SDR/repo/mcp stretch/current Release
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.

Node 6:
Code: 
Get:1 http://security.debian.org stretch/updates InRelease [94.3 kB]
Ign:2 http://ftp.dk.debian.org/debian stretch InRelease
Hit:3 http://ftp.dk.debian.org/debian stretch Release
Hit:4 http://download.proxmox.com/debian/pve stretch InRelease
Hit:5 http://apt.puppetlabs.com stretch InRelease
Ign:6 http://downloads.linux.hpe.com/SDR/repo/mcp stretch/current InRelease
Hit:8 http://downloads.linux.hpe.com/SDR/repo/mcp stretch/current Release
Fetched 94.3 kB in 0s (115 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
N: Skipping acquire of configured file 'pve-no-subscription/binary-i386/Packages' as repository 'http://download.proxmox.com/debian/pve stretch InRelease' doesn't support architecture 'i386'

apt full-upgrade
Node 3:
Code: 
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Node 6:
Code: 
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  augeas-lenses fonts-lato hiera javascript-common libaugeas0 libjs-jquery libruby2.3 libyaml-0-2 rake ruby ruby-augeas
  ruby-deep-merge ruby-did-you-mean ruby-json ruby-minitest ruby-net-telnet ruby-nokogiri ruby-pkg-config ruby-power-assert
  ruby-rgen ruby-safe-yaml ruby-selinux ruby-shadow ruby-test-unit ruby2.3 rubygems-integration unzip virt-what zip
Use 'apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
 
We have had the i386 archithecture installed on all nodes. We haven't uninstalled it from node 6 yet, but that haven't been an issue earlier, and it's node 3 that have the issue.
 
Hmm - good point .

Still this is quite odd - could you try posting the sources.list files (all of them) - if necessary obfuscate the url (put a _ inside the ht_tp) ?

what's the output of:
`apt-cache policy pve-kernel-4.15`
on both nodes?
 
apt-cache policy pve-kernel-4.15
Node 3
Code: 
pve-kernel-4.15:
  Installed: 5.3-3
  Candidate: 5.3-3
  Version table:
 *** 5.3-3 100
        100 /var/lib/dpkg/status
Node 6
Code: 
pve-kernel-4.15:
  Installed: 5.4-7
  Candidate: 5.4-7
  Version table:
 *** 5.4-7 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
        100 /var/lib/dpkg/status
     5.4-6 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.4-5 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.4-4 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.4-3 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.4-2 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.4-1 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.3-3 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.3-2 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.3-1 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.2-12 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.2-11 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.2-10 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.2-8 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.2-7 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.2-6 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.2-5 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.2-4 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.2-3 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.2-2 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.2-1 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.1-4 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.1-3 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.1-2 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.1-1 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
​
/etc/apt/sources.list
Node 3
Code: 
deb ht_tp://ftp.debian.org/debian stretch main contrib

# PVE pve-no-subscription repository provided by proxmox.com,
# NOT recommended for production use
# deb http://download.proxmox.com/debian/pve stretch pve-no-subscription

# security updates
deb ht_tp://security.debian.org stretch/updates main contrib
Node 6
Code: 
deb ht_tp://ftp.dk.debian.org/debian stretch main contrib

# security updates
deb ht_tp://security.debian.org stretch/updates main contrib
/etc/apt/sources.list.d/pve-enterprise.list
Node 3
Code: 
# deb ht_tps://enterprise.proxmox.com/debian/pve stretch pve-enterprise
Node 6
Code: 
# deb ht_tps://enterprise.proxmox.com/debian/pve stretch pve-enterprise
/etc/apt/sources.list.d/pve-no-sub.list
Node 3
Code: 
deb ht_tp://download.proxmox.com/debian/pve stretch pve-no-subscription
Node 6
Code: 
deb ht_tp://download.proxmox.com/debian/pve stretch pve-no-subscription

​
 
This is getting more odd...
I agree that both nodes should have the same view on the package repositories and thus the same versions - but they do not!
(see the apt-cache policy output on node3 )

any chance there is a proxy/firewall in the middle?
or any difference in '/etc/apt/'?
(copy the dirs from both nodes and run `diff -ruN node3aptdir node6aptdir`)
or a wrong entry in /etc/hosts the dns?
(when you try to ping download.proxmox.com from both nodes - do you see the same ip?)

hope this helps!
 
Hi Stoiko

There is a firewall between the servers and the internet, but no firewall on the servers themselves.

The only difference is in listchanges.conf:
Code: 
diff -ruN node3 node6
diff -ruN node3/apt.conf.d/01autoremove-kernels node6/apt.conf.d/01autoremove-kernels
--- node3/apt.conf.d/01autoremove-kernels       2019-08-01 18:19:44.101885240 +0200
+++ node6/apt.conf.d/01autoremove-kernels       2019-08-01 18:16:49.446460480 +0200
@@ -1,17 +1,28 @@
 // DO NOT EDIT! File autogenerated by /etc/kernel/postinst.d/apt-auto-removal
 APT::NeverAutoRemove
 {
-   "^linux-image-4\.15\.18-12-pve$";
-   "^linux-headers-4\.15\.18-12-pve$";
-   "^linux-image-extra-4\.15\.18-12-pve$";
-   "^linux-signed-image-4\.15\.18-12-pve$";
-   "^kfreebsd-image-4\.15\.18-12-pve$";
-   "^kfreebsd-headers-4\.15\.18-12-pve$";
-   "^gnumach-image-4\.15\.18-12-pve$";
-   "^.*-modules-4\.15\.18-12-pve$";
-   "^.*-kernel-4\.15\.18-12-pve$";
-   "^linux-backports-modules-.*-4\.15\.18-12-pve$";
-   "^linux-tools-4\.15\.18-12-pve$";
+   "^linux-image-4\.15\.18-13-pve$";
+   "^linux-image-4\.15\.18-19-pve$";
+   "^linux-headers-4\.15\.18-13-pve$";
+   "^linux-headers-4\.15\.18-19-pve$";
+   "^linux-image-extra-4\.15\.18-13-pve$";
+   "^linux-image-extra-4\.15\.18-19-pve$";
+   "^linux-signed-image-4\.15\.18-13-pve$";
+   "^linux-signed-image-4\.15\.18-19-pve$";
+   "^kfreebsd-image-4\.15\.18-13-pve$";
+   "^kfreebsd-image-4\.15\.18-19-pve$";
+   "^kfreebsd-headers-4\.15\.18-13-pve$";
+   "^kfreebsd-headers-4\.15\.18-19-pve$";
+   "^gnumach-image-4\.15\.18-13-pve$";
+   "^gnumach-image-4\.15\.18-19-pve$";
+   "^.*-modules-4\.15\.18-13-pve$";
+   "^.*-modules-4\.15\.18-19-pve$";
+   "^.*-kernel-4\.15\.18-13-pve$";
+   "^.*-kernel-4\.15\.18-19-pve$";
+   "^linux-backports-modules-.*-4\.15\.18-13-pve$";
+   "^linux-backports-modules-.*-4\.15\.18-19-pve$";
+   "^linux-tools-4\.15\.18-13-pve$";
+   "^linux-tools-4\.15\.18-19-pve$";
 };
 /* Debug information:
 # dpkg list:
@@ -19,12 +30,13 @@

 # list of different kernel versions:

-# Installing kernel:  (4.15.18-12-pve)
-# Running kernel: ignored (4.15.18-12-pve)
+# Installing kernel:  (4.15.18-19-pve)
+# Running kernel: ignored (4.15.18-13-pve)
 # Last kernel:
 # Previous kernel:
 # Kernel versions list to keep:

 # Kernel packages (version part) to protect:
-4\.15\.18-12-pve
+4\.15\.18-13-pve
+4\.15\.18-19-pve
 */
diff -ruN node3/listchanges.conf node6/listchanges.conf
--- node3/listchanges.conf      1970-01-01 01:00:00.000000000 +0100
+++ node6/listchanges.conf      2019-08-01 18:16:49.450460376 +0200
@@ -0,0 +1,7 @@
+[apt]
+frontend=pager
+confirm=false
+email_address=root
+save_seen=/var/lib/apt/listchanges.db
+which=news
+
diff -ruN node3/sources.list node6/sources.list
--- node3/sources.list  1970-01-01 01:00:00.000000000 +0100
+++ node6/sources.list  2019-08-01 18:16:49.450460376 +0200
@@ -0,0 +1,4 @@
+deb http://ftp.dk.debian.org/debian stretch main contrib
+
+# security updates
+deb http://security.debian.org stretch/updates main contrib
Binary files node3/trusted.gpg and node6/trusted.gpg differ
Binary files node3/trusted.gpg~ and node6/trusted.gpg~ differ

I can't ping download.proxmox.com, but doing a nslookup give me the ip 212.224.123.70
Code: 
ping download.proxmox.com
PING de.cdn.proxmox.com (212.224.123.70) 56(84) bytes of data.

/etc/hosts are the same on all servers.
 
you need to either remove the i386 architecture (with "dpkg --remove-architecture i386") or explicitly tell apt that the PVE repositories are amd64 only (by replacing the leading "deb" with "deb [arch=amd64]" (or adding "arch=amd64" to the existing list of options, if you have one - see "man sources.list").

edit: misread, but above does not hurt. can you try deleting the APT list files in "/var/lib/apt/lists/" and then calling "apt update" again?
 
Hi Fabian

Deleting the files in /var/lib/apt/lists/ solved it.

I did the following:
Code: 
rm -rf /var/lib/apt/lists/*
apt-get update
apt-get dist-upgrade

Thanks for your help.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom