Bug Proxmox 5.1.35 - Can not start VM

[itvietnam]

This morning my server shutdown and i can not start our VM back, when check pveproxy service is said:

root@node01:~# service pveproxy status
● pveproxy.service - PVE API Proxy Server
Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2018-01-28 11:01:03 +07; 6min ago
Process: 2348 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
Main PID: 2353 (pveproxy)
Tasks: 4 (limit: 4915)
CGroup: /system.slice/pveproxy.service
├─2353 pveproxy
├─2354 pveproxy worker
├─2355 pveproxy worker
└─2356 pveproxy worker

Jan 28 11:02:49 node01 pveproxy[2354]: internal error at /usr/share/perl5/PVE/RESTHandler.pm line 312.
Jan 28 11:03:05 node01 pveproxy[2354]: internal error at /usr/share/perl5/PVE/RESTHandler.pm line 312.
Jan 28 11:03:05 node01 pveproxy[2354]: internal error at /usr/share/perl5/PVE/RESTHandler.pm line 312.
Jan 28 11:03:41 node01 pveproxy[2356]: proxy detected vanished client connection
Jan 28 11:04:09 node01 pveproxy[2355]: proxy detected vanished client connection
Jan 28 11:05:17 node01 pveproxy[2354]: proxy detected vanished client connection
Jan 28 11:05:55 node01 pveproxy[2356]: proxy detected vanished client connection
Jan 28 11:06:17 node01 pveproxy[2356]: proxy detected vanished client connection
Jan 28 11:06:35 node01 pveproxy[2355]: proxy detected vanished client connection
Jan 28 11:06:57 node01 pveproxy[2356]: proxy detected vanished client connection
root@node01:~#

 

[fireon]

Why the serve was shutdown?
Is the server part of an cluster?
Where the vm-images are stored?
What error message is displayed if you start an vm? (qm start <vmid>)

 

[itvietnam]

Why the serve was shutdown?

I guest problem related to this thread

Is the server part of an cluster?

Yes

Where the vm-images are stored?

External CEPH storage

What error message is displayed if you start an vm? (qm start <vmid>)

We use both platform: KVM and LXC. I try separate both platform to separate compute, some host KVM and some host LXC. After monitoring we detected problem just happen to LXC node only. KVM node is still working normal.

This probably number of file limited somewhere in Proxmox. I try increase some value inotify parameter but no luck

 

[fireon]

hmm, ok sound not nice. So no update no changes on the host? Ok, have tryed to grow up the inotify watches?

nano /etc/sysctl.d/pve.conf
...
fs.inotify.max_user_watches=1048576
...

Your system is clean?

apt install -f

If you set new watches, you have to reboot the server. Maybe it will help.

 

[itvietnam]

Hi Fireon,

Yes, this is newly setup system and host under 50 VPS on 3 nodes. I have narrow down VPS caused this issues and temporary suspend them to avoid this error. Our server has run without reboot more than 24 hours already.

But this is just a temporary resolution. It's not the root cause and preventive action in case we have more than 1000+ VPSes.