[SOLVED] Node won't upgrade to newest version

[Eilif Johannsen]

I have multiple nodes and one of them won't upgrade to newest version.

The one that won't upgrade is running 4.15.18-12 where the others are running 4.15.18-19.

I see that it's both of these it haven't installed:

pve-kernel-4.15.18-19-pve: 4.15.18-45
pve-kernel-4.15.18-13-pve: 4.15.18-37​

All nodes have been updated previously without problems.
Running apt-get update and apt-get dist-upgrade show that there isn't any updates.

Both servers have been restarted multiple times.

Does someone have an idea how to fix this ?

pveversion -v
Node 3

proxmox-ve: 5.4-1 (running kernel: 4.15.18-12-pve)
pve-manager: 5.3-11 (running version: 5.3-11/d4907f84)
pve-kernel-4.15: 5.3-3
pve-kernel-4.15.18-12-pve: 4.15.18-36
pve-kernel-4.15.18-11-pve: 4.15.18-34
pve-kernel-4.15.18-9-pve: 4.15.18-30
pve-kernel-4.15.18-4-pve: 4.15.18-23
pve-kernel-4.15.17-3-pve: 4.15.17-14
pve-kernel-4.13.13-1-pve: 4.13.13-31
pve-kernel-4.13.8-2-pve: 4.13.8-28
pve-kernel-4.10.17-2-pve: 4.10.17-20
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-51
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-13
libpve-storage-perl: 5.0-42
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-3
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-3
proxmox-widget-toolkit: 1.0-26
pve-cluster: 5.0-37
pve-container: 2.0-37
pve-docs: 5.4-2
pve-edk2-firmware: 1.20190312-1
pve-firewall: 3.0-20
pve-firmware: 2.0-6
pve-ha-manager: 2.0-9
pve-i18n: 1.1-4
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 3.0.1-2
pve-xtermjs: 3.12.0-1
qemu-server: 5.0-51
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.13-pve1~bpo2
​

Node 6

proxmox-ve: 5.4-2 (running kernel: 4.15.18-19-pve)
pve-manager: 5.4-13 (running version: 5.4-13/aee6f0ec)
pve-kernel-4.15: 5.4-7
pve-kernel-4.15.18-19-pve: 4.15.18-45
pve-kernel-4.15.18-13-pve: 4.15.18-37
pve-kernel-4.15.18-12-pve: 4.15.18-36
pve-kernel-4.15.18-11-pve: 4.15.18-34
pve-kernel-4.15.18-9-pve: 4.15.18-30
pve-kernel-4.15.18-4-pve: 4.15.18-23
pve-kernel-4.15.17-3-pve: 4.15.17-14
pve-kernel-4.13.13-1-pve: 4.13.13-31
pve-kernel-4.13.8-2-pve: 4.13.8-28
pve-kernel-4.10.17-2-pve: 4.10.17-20
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-12
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-54
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-14
libpve-storage-perl: 5.0-44
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-3
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-3
proxmox-widget-toolkit: 1.0-28
pve-cluster: 5.0-37
pve-container: 2.0-40
pve-docs: 5.4-2
pve-edk2-firmware: 1.20190312-1
pve-firewall: 3.0-22
pve-firmware: 2.0-7
pve-ha-manager: 2.0-9
pve-i18n: 1.1-4
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 3.0.1-4
pve-xtermjs: 3.12.0-1
qemu-server: 5.0-54
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.13-pve1~bpo2​

 

[aaron]

Do you have the same repositories configured on all nodes?
In

/etc/apt/sources.list

and

/etc/apt/sources.list.d/*.list

 

[Eilif Johannsen]

Yes all files are identical.

I'm not allowed to post external url in a comment so can't post the files content.

 

[Stoiko Ivanov]

please post the complete output from `apt update`, `apt full-upgrade` on the nodes (post them directly into the forum inside code tags)

 

[Eilif Johannsen]

apt update

Node 3:

Hit:1 http://security.debian.org stretch/updates InRelease
Ign:2 http://ftp.debian.org/debian stretch InRelease
Hit:3 http://ftp.debian.org/debian stretch Release
Hit:4 http://download.proxmox.com/debian/pve stretch InRelease
Hit:5 http://apt.puppetlabs.com stretch InRelease
Ign:7 http://downloads.linux.hpe.com/SDR/repo/mcp stretch/current InRelease
Hit:8 http://downloads.linux.hpe.com/SDR/repo/mcp stretch/current Release
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.

Node 6:

Get:1 http://security.debian.org stretch/updates InRelease [94.3 kB]
Ign:2 http://ftp.dk.debian.org/debian stretch InRelease
Hit:3 http://ftp.dk.debian.org/debian stretch Release
Hit:4 http://download.proxmox.com/debian/pve stretch InRelease
Hit:5 http://apt.puppetlabs.com stretch InRelease
Ign:6 http://downloads.linux.hpe.com/SDR/repo/mcp stretch/current InRelease
Hit:8 http://downloads.linux.hpe.com/SDR/repo/mcp stretch/current Release
Fetched 94.3 kB in 0s (115 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
N: Skipping acquire of configured file 'pve-no-subscription/binary-i386/Packages' as repository 'http://download.proxmox.com/debian/pve stretch InRelease' doesn't support architecture 'i386'

apt full-upgrade

Node 3:

Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Node 6:

Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  augeas-lenses fonts-lato hiera javascript-common libaugeas0 libjs-jquery libruby2.3 libyaml-0-2 rake ruby ruby-augeas
  ruby-deep-merge ruby-did-you-mean ruby-json ruby-minitest ruby-net-telnet ruby-nokogiri ruby-pkg-config ruby-power-assert
  ruby-rgen ruby-safe-yaml ruby-selinux ruby-shadow ruby-test-unit ruby2.3 rubygems-integration unzip virt-what zip
Use 'apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

 

[Stoiko Ivanov]

N: Skipping acquire of configured file 'pve-no-subscription/binary-i386/Packages' as repository 'http://download.proxmox.com/debian/pve stretch InRelease' doesn't support architecture 'i386'

This is quite odd - did you by chance setup the node with i386 ?
or have an apt-config setting to that end?

PVE supports only amd64 as architecture

 

[Eilif Johannsen]

We have had the i386 archithecture installed on all nodes. We haven't uninstalled it from node 6 yet, but that haven't been an issue earlier, and it's node 3 that have the issue.

 

[Stoiko Ivanov]

Hmm - good point .

Still this is quite odd - could you try posting the sources.list files (all of them) - if necessary obfuscate the url (put a _ inside the ht_tp) ?

what's the output of:
`apt-cache policy pve-kernel-4.15`
on both nodes?

 

[Eilif Johannsen]

apt-cache policy pve-kernel-4.15

Node 3

pve-kernel-4.15:
  Installed: 5.3-3
  Candidate: 5.3-3
  Version table:
 *** 5.3-3 100
        100 /var/lib/dpkg/status

Node 6

pve-kernel-4.15:
  Installed: 5.4-7
  Candidate: 5.4-7
  Version table:
 *** 5.4-7 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
        100 /var/lib/dpkg/status
     5.4-6 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.4-5 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.4-4 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.4-3 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.4-2 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.4-1 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.3-3 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.3-2 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.3-1 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.2-12 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.2-11 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.2-10 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.2-8 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.2-7 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.2-6 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.2-5 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.2-4 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.2-3 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.2-2 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.2-1 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.1-4 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.1-3 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.1-2 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages
     5.1-1 500
        500 ht_tp://download.proxmox.com/debian/pve stretch/pve-no-subscription amd64 Packages

​

/etc/apt/sources.list

Node 3

deb ht_tp://ftp.debian.org/debian stretch main contrib

# PVE pve-no-subscription repository provided by proxmox.com,
# NOT recommended for production use
# deb http://download.proxmox.com/debian/pve stretch pve-no-subscription

# security updates
deb ht_tp://security.debian.org stretch/updates main contrib

Node 6

deb ht_tp://ftp.dk.debian.org/debian stretch main contrib

# security updates
deb ht_tp://security.debian.org stretch/updates main contrib

/etc/apt/sources.list.d/pve-enterprise.list

Node 3

# deb ht_tps://enterprise.proxmox.com/debian/pve stretch pve-enterprise

Node 6

# deb ht_tps://enterprise.proxmox.com/debian/pve stretch pve-enterprise

/etc/apt/sources.list.d/pve-no-sub.list

Node 3

deb ht_tp://download.proxmox.com/debian/pve stretch pve-no-subscription

Node 6

deb ht_tp://download.proxmox.com/debian/pve stretch pve-no-subscription

​

 

[Stoiko Ivanov]

This is getting more odd...
I agree that both nodes should have the same view on the package repositories and thus the same versions - but they do not!
(see the apt-cache policy output on node3 )

any chance there is a proxy/firewall in the middle?
or any difference in '/etc/apt/'?
(copy the dirs from both nodes and run `diff -ruN node3aptdir node6aptdir`)
or a wrong entry in /etc/hosts the dns?
(when you try to ping download.proxmox.com from both nodes - do you see the same ip?)

hope this helps!

 

[Eilif Johannsen]

Hi Stoiko

There is a firewall between the servers and the internet, but no firewall on the servers themselves.

The only difference is in listchanges.conf:

diff -ruN node3 node6
diff -ruN node3/apt.conf.d/01autoremove-kernels node6/apt.conf.d/01autoremove-kernels
--- node3/apt.conf.d/01autoremove-kernels       2019-08-01 18:19:44.101885240 +0200
+++ node6/apt.conf.d/01autoremove-kernels       2019-08-01 18:16:49.446460480 +0200
@@ -1,17 +1,28 @@
 // DO NOT EDIT! File autogenerated by /etc/kernel/postinst.d/apt-auto-removal
 APT::NeverAutoRemove
 {
-   "^linux-image-4\.15\.18-12-pve$";
-   "^linux-headers-4\.15\.18-12-pve$";
-   "^linux-image-extra-4\.15\.18-12-pve$";
-   "^linux-signed-image-4\.15\.18-12-pve$";
-   "^kfreebsd-image-4\.15\.18-12-pve$";
-   "^kfreebsd-headers-4\.15\.18-12-pve$";
-   "^gnumach-image-4\.15\.18-12-pve$";
-   "^.*-modules-4\.15\.18-12-pve$";
-   "^.*-kernel-4\.15\.18-12-pve$";
-   "^linux-backports-modules-.*-4\.15\.18-12-pve$";
-   "^linux-tools-4\.15\.18-12-pve$";
+   "^linux-image-4\.15\.18-13-pve$";
+   "^linux-image-4\.15\.18-19-pve$";
+   "^linux-headers-4\.15\.18-13-pve$";
+   "^linux-headers-4\.15\.18-19-pve$";
+   "^linux-image-extra-4\.15\.18-13-pve$";
+   "^linux-image-extra-4\.15\.18-19-pve$";
+   "^linux-signed-image-4\.15\.18-13-pve$";
+   "^linux-signed-image-4\.15\.18-19-pve$";
+   "^kfreebsd-image-4\.15\.18-13-pve$";
+   "^kfreebsd-image-4\.15\.18-19-pve$";
+   "^kfreebsd-headers-4\.15\.18-13-pve$";
+   "^kfreebsd-headers-4\.15\.18-19-pve$";
+   "^gnumach-image-4\.15\.18-13-pve$";
+   "^gnumach-image-4\.15\.18-19-pve$";
+   "^.*-modules-4\.15\.18-13-pve$";
+   "^.*-modules-4\.15\.18-19-pve$";
+   "^.*-kernel-4\.15\.18-13-pve$";
+   "^.*-kernel-4\.15\.18-19-pve$";
+   "^linux-backports-modules-.*-4\.15\.18-13-pve$";
+   "^linux-backports-modules-.*-4\.15\.18-19-pve$";
+   "^linux-tools-4\.15\.18-13-pve$";
+   "^linux-tools-4\.15\.18-19-pve$";
 };
 /* Debug information:
 # dpkg list:
@@ -19,12 +30,13 @@

 # list of different kernel versions:

-# Installing kernel:  (4.15.18-12-pve)
-# Running kernel: ignored (4.15.18-12-pve)
+# Installing kernel:  (4.15.18-19-pve)
+# Running kernel: ignored (4.15.18-13-pve)
 # Last kernel:
 # Previous kernel:
 # Kernel versions list to keep:

 # Kernel packages (version part) to protect:
-4\.15\.18-12-pve
+4\.15\.18-13-pve
+4\.15\.18-19-pve
 */
diff -ruN node3/listchanges.conf node6/listchanges.conf
--- node3/listchanges.conf      1970-01-01 01:00:00.000000000 +0100
+++ node6/listchanges.conf      2019-08-01 18:16:49.450460376 +0200
@@ -0,0 +1,7 @@
+[apt]
+frontend=pager
+confirm=false
+email_address=root
+save_seen=/var/lib/apt/listchanges.db
+which=news
+
diff -ruN node3/sources.list node6/sources.list
--- node3/sources.list  1970-01-01 01:00:00.000000000 +0100
+++ node6/sources.list  2019-08-01 18:16:49.450460376 +0200
@@ -0,0 +1,4 @@
+deb http://ftp.dk.debian.org/debian stretch main contrib
+
+# security updates
+deb http://security.debian.org stretch/updates main contrib
Binary files node3/trusted.gpg and node6/trusted.gpg differ
Binary files node3/trusted.gpg~ and node6/trusted.gpg~ differ

I can't ping download.proxmox.com, but doing a nslookup give me the ip 212.224.123.70

ping download.proxmox.com
PING de.cdn.proxmox.com (212.224.123.70) 56(84) bytes of data.

/etc/hosts are the same on all servers.

 

[fabian]

you need to either remove the i386 architecture (with "dpkg --remove-architecture i386") or explicitly tell apt that the PVE repositories are amd64 only (by replacing the leading "deb" with "deb [arch=amd64]" (or adding "arch=amd64" to the existing list of options, if you have one - see "man sources.list").

edit: misread, but above does not hurt. can you try deleting the APT list files in "/var/lib/apt/lists/" and then calling "apt update" again?

 

[Eilif Johannsen]

Hi Fabian

Deleting the files in /var/lib/apt/lists/ solved it.

I did the following:

rm -rf /var/lib/apt/lists/*
apt-get update
apt-get dist-upgrade

Thanks for your help.