Proxmox 5 reboot bug?
- Thread starter mada
- Start date
Nothing than the normal
Sep 19 12:12:41 xx pmxcfs[1774]: [status] notice: received log
Sep 19 12:12:42 xx systemd[1]: Started Session 271 of user root.
Sep 19 12:12:43 xx qm[14577]: VM 221 qmp command failed - VM 221 not running
Sep 19 12:12:43 xx pmxcfs[1774]: [status] notice: received log
When i do reset from the Proxmox GUI the VPS go down and won't boot up again without manually boot.
A reboot of the VM works without issues? When you reset the VM then it should show task entries like this.
Code:
Sep 20 09:18:51 pve1 pvedaemon[6249]: <root@pam> starting task UPID:pve1:00002E07:00E08789:59C2165B:qmreset:19141:root@pam:I only see this
Sep 20 06:39:39 xx pmxcfs[2236]: [status] notice: received log
Sep 20 06:39:39 xx pvedaemon[5185]: <root@pam> successful auth for user 'root@pam'
Sep 20 06:39:39 xx pmxcfs[2236]: [status] notice: received log
Sep 20 06:39:39 xx pmxcfs[2236]: [status] notice: received log
Sep 20 06:39:39 xx pvedaemon[4564]: <root@pam> successful auth for user 'root@pam'
Sep 20 06:39:39 xx pvedaemon[5185]: <root@pam> successful auth for user 'root@pam'
The problem is with all VM on all node i do have with Proxmox 5 i just installed new server last 2 days ago and have the same issue. i have around 6 server with Proxmox 5 have the same issue with all VM i do have like 100 VM on them.
i really not sure what is going on i also have 7 server with Proxmox 4 they are fine.
Sorry to start stepping on toes but.....
ksm-control-daemon is not installed when following the wiki guide for installing proxmox on debian...
that said, KSM missing would NEVER be the issue for a VM failing to RESTART if it was already running, lack of KSM could prevent a VM from starting if the total RAM allotment for all currently running VMs was close to, or exceeding the physical RAM available on the host, and ballooning was unable to borrow enough RAM form the currently running VMs to cover the startup of another, but if the VM was already running, a reboot would happen flawlessly with or without KSM.
my above statement is already well tested, it took me almost 4 weeks to notice KSM was not installed, as i dealt with borderline OOM conditions.
as well as anyone whom followed the guide before this moment (as im sure someone will run to update it) who is likely still running without KSM installed.
to be clear KSM has nothing to do at all with VM memory ballooning.
https://pve.proxmox.com/wiki/Dynamic_Memory_Management
KSM can be disabled and still have auto-ballooning, the ballooning driver still arrives in the guest, it is a KVM function, not part of KSM.
As for your reboot issues, when you try to reboot or reset a VM, does your cpu, ram, disk access, or IO Delay, max out or go beyond max?
if your bottle-necking on any of those, it can easily cause the VM to time out, and KVM/Proxmox to give up on it.
You did state you had 100s of VMs on each node.
if CPU or IO delay is the issue, then you will want to uninstall KSM again, it greatly adds to CPU usage.
if your CPU usage is fine, then KSM will help you in saving RAM both for virtual machines, and other processes on the node.
ksm-control-daemon is not installed when following the wiki guide for installing proxmox on debian...
that said, KSM missing would NEVER be the issue for a VM failing to RESTART if it was already running, lack of KSM could prevent a VM from starting if the total RAM allotment for all currently running VMs was close to, or exceeding the physical RAM available on the host, and ballooning was unable to borrow enough RAM form the currently running VMs to cover the startup of another, but if the VM was already running, a reboot would happen flawlessly with or without KSM.
my above statement is already well tested, it took me almost 4 weeks to notice KSM was not installed, as i dealt with borderline OOM conditions.
as well as anyone whom followed the guide before this moment (as im sure someone will run to update it) who is likely still running without KSM installed.
to be clear KSM has nothing to do at all with VM memory ballooning.
https://pve.proxmox.com/wiki/Dynamic_Memory_Management
KSM can be disabled and still have auto-ballooning, the ballooning driver still arrives in the guest, it is a KVM function, not part of KSM.
As for your reboot issues, when you try to reboot or reset a VM, does your cpu, ram, disk access, or IO Delay, max out or go beyond max?
if your bottle-necking on any of those, it can easily cause the VM to time out, and KVM/Proxmox to give up on it.
You did state you had 100s of VMs on each node.
if CPU or IO delay is the issue, then you will want to uninstall KSM again, it greatly adds to CPU usage.
if your CPU usage is fine, then KSM will help you in saving RAM both for virtual machines, and other processes on the node.
correct, KSM has nothing to do with the issue, I just assumed from his description that the installation might be bust.
Good point, but then you would see some log entries. Still worth to investigate further on this matter, in my opinion.
True, though i would like to add, that when I first tested proxmox, I did so on a very low end system. it was not uncommon for the timeout to happen at the gui level itself, I assume the inbuilt web service simply could not keep up, ether way numerous commands could ether fail to execute, or only partially execute. although I did not see the reboot issue, I did see many issues with VMs showing as started in the web gui after clicking start, but they never actually started, and there was no log entry to indicate what went wrong. similarly attempting to force stop VMs sometimes resulted in the webgui showing it had stopped, but no log entry for the stop event, and the VM was still very much alive.
I do not know if the issue is the same here or not, i know those issues of mine were solved by running less VMs on the test box. once moving to production hardware the issue did not present again, even with... less then standard setup. (another issue, another pre-existing thread)
So it is my opinion that the first focus should be the easiest and simplest to get out of the way, that being, is there sufficient resources for all those VMs plus proxmox/overhead on Mada's setup. is something maxing out? does running less VMs or giving them less resources help? are they running same family architecture? (ie, is it a mix of x86 and x64 systems only? or is qemu/kvm running some out of family systems too, such as arm) off family would mean pure software cpu virtualization, which would spike overhead.
if the host is starving for resources, then logs may be of little use as events may never make it to the log in the first place, if the "log this" trigger even fires at all before time outs and give ups.
on the possible install bust, I recommend aptitude, (apt install aptitude)
with aptitude you may do a reinstall of the proxmox packages, aptitude is very adept at finding errors and repairing them during a reinstall, errors that apt otherwise misses.
(example, if i got it all right: "aptitude reinstall pve-manager pve-qemu-kvm pve-ha-manager libpve-access-control libpve-common-perl libpve-guest-common-perl libpve-http-server-perl libpve-storage-perl lxc-pve pve-libspice-server1" that should reinstall the core, fixing any issues it may find, aptitude may ask you what to do in the case of modified config files, keep/replace with default/examine,fix manually)
aptitude is not guaranteed to fix this or any other issue, but often can where apt and dpkg fails, avoiding a full system reinstall to chase the gremlins away.
at the very least if there is an install issue it can not fix, aptitude does give much better information as to why it failed, vs apt basically leaving you to sort it out yourself with vague error messages.
No 100 VM @ all 7 nodes some even they are new have few like 4 or 7 VM so there is no high load or i/o issue sine i'm using high enterprise Raid card & SAS drives
The issue fixed if i using Proxmox ISO but i would like to using Debian ! seems there is something wrong if Proxmox install above Debian9 , so what i have to do? did this will ever fixed ? i can't switch to Poxmox ISO since i required some more staff with Debian and better options for me?
Last edited:
Are your installations according to the instructions on our wiki?
https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Stretch
Proxmox is on top of Debian even when you install from ISO, you can install all the packages from Debian. Which extra stuff do you need?
https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Stretch
Proxmox is on top of Debian even when you install from ISO, you can install all the packages from Debian. Which extra stuff do you need?
At the moment, we only know that there is a difference, not that something would be broken, that would need fixing.
Yes i followed The wiki, unfortunately i can't move over the Debian since this will required at last 10 servers of it.
I don't do anything special my drives followed.
/ root files
/swap
/var/lib/vz
installed the Debian 9 then using Proxmox package
Add the Proxmox VE repository:
echo "deb http://download.proxmox.com/debian/pve stretch pve-no-subscription" > /etc/apt/sources.list.d/pve-install-repo.list
Add the Proxmox VE repository key:
wget http://download.proxmox.com/debian/proxmox-ve-release-5.x.gpg -O /etc/apt/trusted.gpg.d/proxmox-ve-release-5.x.gpg
Update your repository and system by running:
apt update && apt dist-upgrade
Install Proxmox VE packages
Install the Proxmox VE packages
apt install proxmox-ve postfix open-iscsi
That's all.