[SOLVED] Upgrade discrepancies!

[Deleted member 33567]

Running a cluster with 3 nodes:

Node1

Linux node01-sxb-pve01 4.13.13-5-pve #1 SMP PVE 4.13.13-38 (Fri, 26 Jan 2018 10:47:09 +0100) x86_64 GNU/Linux

root@node01-sxb-pve01:~# apt-get update && apt-get upgrade && apt-get dist-upgrade
Hit:1 http://security.debian.org stretch/updates InRelease
Hit:2 https://enterprise.proxmox.com/debian/pve stretch InRelease                                                                                                  
Hit:3 https://packages.cisofy.com/community/lynis/deb stretch InRelease                                                                                            
Hit:5 http://repo.cloudlinux.com/kernelcare-debian/8 stable InRelease                                                    
Ign:4 http://cdn-fastly.deb.debian.org/debian stretch InRelease                                                           
Hit:6 http://cdn-fastly.deb.debian.org/debian stretch-updates InRelease 
Hit:7 http://download.ceph.com/debian-luminous stretch InRelease        
Hit:8 http://cdn-fastly.deb.debian.org/debian stretch Release           
Reading package lists... Done
Reading package lists... Done
Building dependency tree      
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists... Done
Building dependency tree      
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
root@node01-sxb-pve01:~#

Node2

Linux node02-sxb-pve01 4.13.13-6-pve #1 SMP PVE 4.13.13-41 (Wed, 21 Feb 2018 10:07:54 +0100) x86_64 GNU/Linux

Node3

Linux node03-sxb-pve01 4.13.13-6-pve #1 SMP PVE 4.13.13-41 (Wed, 21 Feb 2018 10:07:54 +0100) x86_64 GNU/Linux

Note that each node also runs Kernelpatch so reboots are NOT required for kernel patching.

 

[Alwin]

Run the apt again on the node1, it may well be that there was a package update in between the updates of the different hosts.

 

[Deleted member 33567]

Run the apt again on the node1, it may well be that there was a package update in between the updates of the different hosts.

I did run the APT again and again... Nothing changed. The reply is above. Including a full update/upgrade and dist-upgrade.

 

[Alwin]

Repositories entries are the same on all nodes (enterprise/no-subscription)? Are the servers behind a proxy (caching)?

 

[Deleted member 33567]

Repositories entries are the same on all nodes (enterprise/no-subscription)? Are the servers behind a proxy (caching)?

No proxy, and yes repository is the same on all nodes Enterprise. All nodes should and had run on same kernel version up until last upgrade.

 

[Alwin]

Is 'pveversion -v' showing the same output? As the hosts didn't reboot, the updated kernel might not yet run.

 

[Deleted member 33567]

Is 'pveversion -v' showing the same output? As the hosts didn't reboot, the updated kernel might not yet run.

We run kernelcare exactly for this reason not to reboot machines. None of the other 2 nodes were rebooted, and they show the changes.

I just found this during the pveversion:

root@node01-sxb-pve01:/etc/apt/sources.list.d# pveversion -v
proxmox-ve: not correctly installed (running kernel: 4.13.13-5-pve)
pve-manager: 5.1-51 (running version: 5.1-51/96be5354)
pve-kernel-4.13.13-5-pve: 4.13.13-38
pve-kernel-4.13.8-3-pve: 4.13.8-30
pve-kernel-4.13.8-2-pve: 4.13.8-28
pve-kernel-4.13.8-1-pve: 4.13.8-27
pve-kernel-4.4.67-1-pve: 4.4.67-92
pve-kernel-4.4.49-1-pve: 4.4.49-86
ceph: 12.2.4-1~bpo90+1
corosync: 2.4.2-pve4
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-apiclient-perl: 2.0-4
libpve-common-perl: 5.0-30
libpve-guest-common-perl: 2.0-14
libpve-http-server-perl: 2.0-8
libpve-storage-perl: 5.0-18
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 2.1.1-3
lxcfs: 2.0.8-2
novnc-pve: 0.6-4
proxmox-widget-toolkit: 1.0-15
pve-cluster: 5.0-25
pve-container: 2.0-21
pve-docs: 5.1-17
pve-firewall: 3.0-8
pve-firmware: 2.0-4
pve-ha-manager: 2.0-5
pve-i18n: 1.0-4
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.11.1-5
pve-xtermjs: 1.0-2
qemu-server: 5.0-25
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.7-pve1~bpo9

While any other node has:

root@node03-sxb-pve01:/etc/apt/sources.list.d# pveversion -v
proxmox-ve: 5.1-42 (running kernel: 4.13.13-6-pve)
pve-manager: 5.1-51 (running version: 5.1-51/96be5354)
pve-kernel-4.13: 5.1-44
pve-kernel-4.13.16-2-pve: 4.13.16-47
pve-kernel-4.13.13-6-pve: 4.13.13-42
pve-kernel-4.13.13-5-pve: 4.13.13-38
pve-kernel-4.13.13-4-pve: 4.13.13-35
pve-kernel-4.13.13-2-pve: 4.13.13-33
pve-kernel-4.13.13-1-pve: 4.13.13-31
pve-kernel-4.13.8-3-pve: 4.13.8-30
pve-kernel-4.13.8-2-pve: 4.13.8-28
pve-kernel-4.4.95-1-pve: 4.4.95-99
pve-kernel-4.4.83-1-pve: 4.4.83-96
pve-kernel-4.4.79-1-pve: 4.4.79-95
pve-kernel-4.4.76-1-pve: 4.4.76-94
pve-kernel-4.4.67-1-pve: 4.4.67-92
pve-kernel-4.4.44-1-pve: 4.4.44-84
ceph: 12.2.4-1~bpo90+1
corosync: 2.4.2-pve4
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-apiclient-perl: 2.0-4
libpve-common-perl: 5.0-30
libpve-guest-common-perl: 2.0-14
libpve-http-server-perl: 2.0-8
libpve-storage-perl: 5.0-18
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 2.1.1-3
lxcfs: 2.0.8-2
novnc-pve: 0.6-4
proxmox-widget-toolkit: 1.0-15
pve-cluster: 5.0-25
pve-container: 2.0-21
pve-docs: 5.1-17
pve-firewall: 3.0-8
pve-firmware: 2.0-4
pve-ha-manager: 2.0-5
pve-i18n: 1.0-4
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.11.1-5
pve-xtermjs: 1.0-2
qemu-server: 5.0-25
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3

 

[Alwin]

You need to re-install the proxmox-ve (meta) package.

 

[Deleted member 33567]

I did so and it resulted still in older kernel version:

root@node01-sxb-pve01:/etc/apt/sources.list.d# pveversion -v
proxmox-ve: 5.1-42 (running kernel: 4.13.13-5-pve)
pve-manager: 5.1-51 (running version: 5.1-51/96be5354)
pve-kernel-4.13: 5.1-44
pve-kernel-4.13.16-2-pve: 4.13.16-47
pve-kernel-4.13.13-5-pve: 4.13.13-38
pve-kernel-4.13.8-3-pve: 4.13.8-30
pve-kernel-4.13.8-2-pve: 4.13.8-28
pve-kernel-4.13.8-1-pve: 4.13.8-27
pve-kernel-4.4.67-1-pve: 4.4.67-92
pve-kernel-4.4.49-1-pve: 4.4.49-86
ceph: 12.2.4-1~bpo90+1
corosync: 2.4.2-pve4
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-apiclient-perl: 2.0-4
libpve-common-perl: 5.0-30
libpve-guest-common-perl: 2.0-14
libpve-http-server-perl: 2.0-8
libpve-storage-perl: 5.0-18
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 2.1.1-3
lxcfs: 2.0.8-2
novnc-pve: 0.6-4
proxmox-widget-toolkit: 1.0-15
pve-cluster: 5.0-25
pve-container: 2.0-21
pve-docs: 5.1-17
pve-firewall: 3.0-8
pve-firmware: 2.0-4
pve-ha-manager: 2.0-5
pve-i18n: 1.0-4
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.11.1-5
pve-xtermjs: 1.0-2
qemu-server: 5.0-25
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.7-pve1~bpo9

 

[tschanness]

Did you run apt-get upgrade or apt-get dist-upgrade?

 

[Deleted member 33567]

Did you run apt-get upgrade or apt-get dist-upgrade?

Yes I did so.

 

[tschanness]

Yes I did so.

You did what? Upgrade or dist-upgrade?

 

[Deleted member 33567]

You did what? Upgrade or dist-upgrade?

Tried both of them:

apt-get update && apt-get upgrade && apt-get dist-upgrade

 

[RobFantini]

we had the same issue. this fixed for us

apt --reinstall  install  proxmox-ve

in our case a newer kernel got installed. this system has a subscription yet uses pve-no-subscription ... need to fix that...

 

[Deleted member 33567]

I already used that command to reinstall. Still this did not made changes on kernel side. I have only the subscription repo installed.

we had the same issue. this fixed for us

apt --reinstall  install  proxmox-ve

in our case a newer kernel got installed. this system has a subscription yet uses pve-no-subscription ... need to fix that...

eady

 

[RobFantini]

I know what you mean about the version numbers.. still "proxmox-ve: not correctly installed" went away here. How about for you?

 

[Deleted member 33567]

I know what you mean about the version numbers.. still "proxmox-ve: not correctly installed" went away here. How about for you?

In the latest reinstall it installed the right version. You can check my earlier reply. Just thr kernel version is skipped back.

 

[fabian]

In the latest reinstall it installed the right version. You can check my earlier reply. Just thr kernel version is skipped back.

the running kernel version only changes when you reboot. a list of installed kernel packages is provided as well, there you can see that the newest one at the moment (pve-kernel-4.13.16-2-pve) is installed

 

[Deleted member 33567]

the running kernel version only changes when you reboot. a list of installed kernel packages is provided as well, there you can see that the newest one at the moment (pve-kernel-4.13.16-2-pve) is installed

In that case having a KernelCare license is not helping at all against server reboots and kernel upgrades.

 

[fabian]

In that case having a KernelCare license is not helping at all against server reboots and kernel upgrades.

I don't understand your point - if you want to boot into a specific kernel version (and get all the changes contained in it), you need to reboot. if you want to fix security vulnerabilities (or rather, potentially only a subset), you can use kernel live patching (like KernelCare). the latter does not change the running kernel version, the former does. kernel live patching does not mean you never need to reboot, but it might mean that you can postpone the reboot until the next service/maintenance window if there is some high risk security vulnerability in the kernel, instead of having to reboot then and there or risk getting exploited.