Unable to start linux vm's after Proxmox upgrade

R

Rait

Member
Aug 1, 2020
9
1
8
Estonia
Hi!

I upgraded my Proxmox to 7.3 with kernel 5.15.35-2 and now my Oracle Linux servers are not starting up. It's throwing me this error:
Code: 
qm start 118
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 147.
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 182.
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 182.
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 208.
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 208.
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 147.
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 182.
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 182.
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 208.
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 208.
kvm: Address space limit 0xffffffffff < 0x440ffffffff phys-bits too low (40)
start failed: QEMU exited with code 1

Update 1:
I'm guessing this has to do something with Intel CPU's since the phys-bits for my cpu is 39:
Code: 
# grep -m 1 'address sizes' /proc/cpuinfo
address sizes   : 39 bits physical, 48 bits virtual

Update 2:
After removing numa=1 from the VM's they start up fine.

Update3:
I tried to boot the VM's with just numa enabled and memory hotplug disabled, VM's booted up fine. There seems to be an issue with memory hotplug.

Kind of lost here on what to do next to make it work with memory hotplug and how to fix the PVE_TAB_RE parts.
 
Last edited:
Hi,
Rait said:
There seems to be an issue with memory hotplug.
Click to expand...
Yes, but only regarding your host and VM config.

The kvm64 CPU type has a phys-bit size of 40, earlier QEMU versions ignored if the physical bit size was smaller, but that was just wrong and worked only by luck most of the time. So newer QEMU enforces that the phys-size of host and guest must be compatible and that any configured max-mem for hotplug must also fit there.

See this commit for some more details:
https://git.proxmox.com/?p=qemu-server.git;a=commitdiff;h=33b0d3b7bee12c49f3d0b7d53699b5874ab8eb73

If you need memory-hotplug now on that system I'd recommend using either the COU type host or setting phys-bits in a custom CPU model explicitly to 39. We left out auto-falling-back to < 40 as that might have some compatibility issues, but can recheck more closely on that.

Out of interest, what CPU model ist this?
 
Hi!

Thank you for the explanation and showing me the commit. I tried memory hotplug with host CPU, but that did not work either. I will define a custom CPU and see if that works.
Regarding the CPU I'm running:
Code: 
12 x Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz (1 Socket)

Would this work as custom CPU inside cpu-models.conf or would I need to define more flags:
Code: 
cpu-model: i7
    flags +aes
    phys-bits 39
    hidden 0
    hv-vendor-id proxmox
    reported-model kvm64

Do you know where the uninitialized value errors are coming from? This is thrown every time I try to list, start, stop VM's via qm command.
 
Last edited:
Rait said:
I upgraded my Proxmox to 7.3 with kernel 5.15.35-2
Click to expand...

That kernel is quite old.
Please provide the full output of: pveversion -v
Did you reboot the PVE-host after the update?

Edit: Is this a cluster? If yes, did you update all nodes (and reboot them afterwards)?
 
Last edited:
Neobin said:
That kernel is quite old.
Please provide the full output of: pveversion -v
Did you reboot the PVE-host after the update?

Edit: Is this a cluster? If yes, did you update all nodes (and reboot them afterwards)?
Click to expand...
We have figured out that the issue does not come from kernel, but it comes from QEMU changes regarding phys-bit size. I just need to define a custom CPU and define phys-bit size to 39, so I can use memory hotplug with numa.
 
Rait said:
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation
Click to expand...
I think Neobin meant above, which is unrelated to the phys-bits size and indeed could indicate a partial update.
Posting pveversion -v and checking that its output is the same in the whole cluster should help to clear that up.
 
  • Like
Reactions: Neobin
t.lamprecht said:
I think Neobin meant above, which is unrelated to the phys-bits size and indeed could indicate a partial update.
Posting pveversion -v and checking that its output is the same in the whole cluster should help to clear that up.
Click to expand...
Oh I misunderstood then. Here is the full output of pveversion -v command:
Code: 
# pveversion -v
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 147.
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 182.
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 182.
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 208.
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 208.
proxmox-ve: 7.3-1 (running kernel: 5.15.35-2-pve)
pve-manager: 7.2-4 (running version: 7.2-4/ca9d43cc)
pve-kernel-helper: 7.2-14
pve-kernel-5.15: 7.2-4
pve-kernel-5.13: 7.1-9
pve-kernel-5.4: 6.4-11
pve-kernel-5.15.35-2-pve: 5.15.35-5
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.4.157-1-pve: 5.4.157-1
pve-kernel-5.4.140-1-pve: 5.4.140-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 14.2.21-1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.2-4
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
openvswitch-switch: 2.15.0+ds1-2+deb11u1
proxmox-backup-client: 2.2.7-1
proxmox-backup-file-restore: 2.2.7-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.3
pve-cluster: 7.3-1
pve-container: 4.2-1
pve-docs: 7.3-1
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-7
pve-firmware: 3.5-6
pve-ha-manager: 3.5.1
pve-i18n: 2.8-1
pve-qemu-kvm: 7.1.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+2
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1
 
Last edited:
Rait said:
Oh I misunderstood then. Here is the full output of pveversion -v command:
Code: 
# pveversion -v
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 147.
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 182.
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 182.
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 208.
Use of uninitialized value $PVE::JSONSchema::PVE_TAG_RE in concatenation (.) or string at /usr/share/perl5/PVE/DataCenterConfig.pm line 208.
proxmox-ve: 7.3-1 (running kernel: 5.15.35-2-pve)
pve-manager: 7.2-4 (running version: 7.2-4/ca9d43cc)
pve-kernel-helper: 7.2-14
pve-kernel-5.15: 7.2-4
pve-kernel-5.13: 7.1-9
pve-kernel-5.4: 6.4-11
pve-kernel-5.15.35-2-pve: 5.15.35-5
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.4.157-1-pve: 5.4.157-1
pve-kernel-5.4.140-1-pve: 5.4.140-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 14.2.21-1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.2-4
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
openvswitch-switch: 2.15.0+ds1-2+deb11u1
proxmox-backup-client: 2.2.7-1
proxmox-backup-file-restore: 2.2.7-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.3
pve-cluster: 7.3-1
pve-container: 4.2-1
pve-docs: 7.3-1
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-7
pve-firmware: 3.5-6
pve-ha-manager: 3.5.1
pve-i18n: 2.8-1
pve-qemu-kvm: 7.1.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+2
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1
Click to expand...

How exactly did you update? With only: apt upgrade or with: apt full-upgrade/apt dist-upgrade or via the GUI? (Never use only: apt upgrade with Proxmox products! [1])

Please provide the full output each in code-tags of:
  • apt list --installed | grep linux-image
  • grep -r '' /etc/apt/sources.list*
  • apt update
  • apt full-upgrade

[1] https://forum.proxmox.com/threads/proxmox-ve-7-1-released.99847/page-8#post-463941
 
Neobin said:
How exactly did you update? With only: apt upgrade or with: apt full-upgrade/apt dist-upgrade or via the GUI? (Never use only: apt upgrade with Proxmox products! [1])

Please provide the full output each in code-tags of:
  • apt list --installed | grep linux-image
  • grep -r '' /etc/apt/sources.list*
  • apt update
  • apt full-upgrade

[1] https://forum.proxmox.com/threads/proxmox-ve-7-1-released.99847/page-8#post-463941
Click to expand...
I did run apt upgrade. I can provide full output list at the end of this week. Please keep the thread open.
 
Neobin said:
How exactly did you update? With only: apt upgrade or with: apt full-upgrade/apt dist-upgrade or via the GUI? (Never use only: apt upgrade with Proxmox products! [1])

Please provide the full output each in code-tags of:
  • apt list --installed | grep linux-image
  • grep -r '' /etc/apt/sources.list*
  • apt update
  • apt full-upgrade

[1] https://forum.proxmox.com/threads/proxmox-ve-7-1-released.99847/page-8#post-463941
Click to expand...
Here are the outputs:
Code: 
# apt list --installed | grep linux-image

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Code: 
# grep -r '' /etc/apt/sources.list*
/etc/apt/sources.list:deb http://ftp.debian.org/debian bullseye main contrib
/etc/apt/sources.list:
/etc/apt/sources.list:deb http://ftp.debian.org/debian bullseye-updates main contrib
/etc/apt/sources.list:
/etc/apt/sources.list:# PVE pve-no-subscription repository provided by proxmox.com,
/etc/apt/sources.list:# NOT recommended for production use
/etc/apt/sources.list:deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription
/etc/apt/sources.list:
/etc/apt/sources.list:# security updates
/etc/apt/sources.list:deb http://security.debian.org/debian-security bullseye-security main contrib
/etc/apt/sources.list:
/etc/apt/sources.list.bak:deb http://ftp.debian.org/debian bullseye main contrib
/etc/apt/sources.list.bak:deb http://ftp.debian.org/debian bullseye-updates main contrib
/etc/apt/sources.list.bak:
/etc/apt/sources.list.bak:# PVE pve-no-subscription repository provided by proxmox.com,
/etc/apt/sources.list.bak:# NOT recommended for production use
/etc/apt/sources.list.bak:deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription
/etc/apt/sources.list.bak:
/etc/apt/sources.list.bak:# security updates
/etc/apt/sources.list.bak:deb http://security.debian.org/debian-security bullseye-security main contrib
/etc/apt/sources.list.d/pve-enterprise.list:#deb https://enterprise.proxmox.com/debian/pve bullseye pve-enterprise
/etc/apt/sources.list.d/pve-enterprise.list.dpkg-dist:deb https://enterprise.proxmox.com/debian/pve bullseye pve-enterprise

Code: 
# apt update
Hit:1 http://security.debian.org/debian-security bullseye-security InRelease
Hit:2 http://ftp.debian.org/debian bullseye InRelease
Get:3 http://ftp.debian.org/debian bullseye-updates InRelease [44.1 kB]
Hit:4 http://download.proxmox.com/debian/pve bullseye InRelease
Fetched 44.1 kB in 1s (58.2 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
10 packages can be upgraded. Run 'apt list --upgradable' to see them.

Code: 
# apt full-upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following package was automatically installed and is no longer required:
  pve-kernel-5.4.140-1-pve
Use 'apt autoremove' to remove it.
The following NEW packages will be installed:
  binutils binutils-common binutils-x86-64-linux-gnu libbinutils libctf-nobfd0 libctf0 proxmox-mail-forward
  proxmox-offline-mirror-docs proxmox-offline-mirror-helper pve-kernel-5.15.74-1-pve
The following packages will be upgraded:
  libpve-common-perl libpve-guest-common-perl libpve-storage-perl proxmox-archive-keyring proxmox-backup-client
  proxmox-backup-file-restore pve-container pve-kernel-5.15 pve-manager qemu-server
10 upgraded, 10 newly installed, 0 to remove and 0 not upgraded.
Need to get 91.3 MB of archives.
After this operation, 431 MB of additional disk space will be used.
Do you want to continue? [Y/n]
 
Can you please also provide the output of: apt list --upgradable to see the exact versions.
 
Neobin said:
Can you please also provide the output of: apt list --upgradable to see the exact versions.
Click to expand...
Here you go:
Code: 
# apt list --upgradable
Listing... Done
libpve-common-perl/stable 7.3-1 all [upgradable from: 7.2-2]
libpve-guest-common-perl/stable 4.2-3 all [upgradable from: 4.1-2]
libpve-storage-perl/stable 7.3-1 all [upgradable from: 7.2-4]
proxmox-archive-keyring/stable 2.2 all [upgradable from: 2.1]
proxmox-backup-client/stable 2.3.1-1 amd64 [upgradable from: 2.2.7-1]
proxmox-backup-file-restore/stable 2.3.1-1 amd64 [upgradable from: 2.2.7-1]
pve-container/stable 4.4-2 all [upgradable from: 4.2-1]
pve-kernel-5.15/stable 7.2-14 all [upgradable from: 7.2-4]
pve-manager/stable 7.3-3 amd64 [upgradable from: 7.2-4]
qemu-server/stable 7.3-1 amd64 [upgradable from: 7.2-3]

As I can see the upgrade wasn't a full upgrade. Just to be sure that I do not break anything, could I run apt full-upgrade right now to fix the issue?
 
Last edited:
Rait said:
Just to be sure that I do not break anything, could I run apt full-upgrade right now to fix the issue?
Click to expand...

To me, all your outputs look pretty good and I personally would do it; but of course I can not give you any guarantee. So if you want to be on the safe side, you might want to wait of the confirmation from the Proxmox-Staff (perhaps @t.lamprecht? :)).
 
what cpu settings should I use with AMD Ryzen 9 3900X now ?


and apt full-upgrade just lead to a hung machine

I really need to find a way to lock apt so when I get my terminals mixed up the wrong pc does not get updated

ugh 16th Oct was my last snapshot of the boot OS , better than nothing I guess
 
Last edited:
I am facing a similar issue were I migrated a VM from proxmox 6.4.x to 7.3.
1. We implemented a new setup with 3 node cluster of 7.3.
2. Took a backup of the VM on old setup of 6.4
4. Restored the backup snapshot on new setup of 7.3
5. In old setup we had processor type default kvm64 and new we are changing it to host
6. 2 VM are working fine whose memory are equal to or below 8GB but there is one VM with memory: 48GB which is giving an error which starting the VM

ERROR: kvm: -device pc-dimm,id=dimm61,memdev=mem-dimm61,node=1: a used vhost backend has no free memory slots left
TASK ERROR: start failed: QEMU exited with code 1

Kindly help on how should this be resolved on new setup of 7.3. what am I missing here
 
freakits_jino said:
6. 2 VM are working fine whose memory are equal to or below 8GB but there is one VM with memory: 48GB which is giving an error which starting the VM
Click to expand...
In my experience memory hotplug fails with more than 21GB or so. You can hotplug the rest after the VM has started. I have not found a resolution yet, sorry.
 
leesteken said:
In my experience memory hotplug fails with more than 21GB or so. You can hotplug the rest after the VM has started. I have not found a resolution yet, sorry.
Click to expand...
Hello @leesteken, I just resolved this issue by doing the following changes.

Check the following file:
cat /sys/module/vhost/parameters/max_mem_regions which had 64
--------------------------------------------------------------------------------------------------
Make following changes on all nodes of the cluster
Add/edit file: /etc/modprobe.d/vhost.conf

options vhost max_mem_regions=509
--------------------------------------------------------------------------------------------------
Reboot each node after the changes.

Verify the changes:
cat /sys/module/vhost/parameters/max_mem_regions
--------------------------------------------------------------------------------------------------
Start VM which was throwing the error.
 
Last edited:
  • Like
Reactions: leesteken
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back