We have been running Proxmox 2.2.24 in production for a week now. Last night half of our scheduled backups job failed with the same error:
The backup job has run fine for the past 6 nights; this is the first error. Of 8 VM's to be backed up, the last 4 all got this error and failed to backup.
At the same time, the pvestatd daemon died (giving the illusion of all the VM's being down in the web interface!). Restarting pvestatd brought the web interface back, and all the VM's are fine; it is just the backups that failed.
I initially thought ulimit might be at fault, but http://sourceforge.net/p/syslog-analyzer/support-requests/35/ points to a separate ionotify configuration. What I don't understand is why I would need to mess with these values on a system designed for virtualization. Right now there are 10 VM's on the system with 6 active. There is about 18GB of RAM free and plenty more storage.
This machine is a dedicated Proxmox machine installed from the Proxmox installation disc. We only have KVM VM's on it. I don't see any additional info in the syslog on this and we've never seen this on the staging server that we have run for 3 months. If this happens again, it will be cause for alarm (we'll see what happens tonight), but even as an isolated instance, I'd like to know if anyone has any idea what would cause this to happen?
Code:
105: Jan 03 01:22:52 INFO: Unable to create new inotify object: Too many open files at /usr/share/perl5/PVE/INotify.pm line 386.
105: [COLOR=red]Jan 03 01:22:52 ERROR: Backup of VM 105 failed - command 'qm set 105 --lock backup' failed: exit code 24[/COLOR]The backup job has run fine for the past 6 nights; this is the first error. Of 8 VM's to be backed up, the last 4 all got this error and failed to backup.
At the same time, the pvestatd daemon died (giving the illusion of all the VM's being down in the web interface!). Restarting pvestatd brought the web interface back, and all the VM's are fine; it is just the backups that failed.
I initially thought ulimit might be at fault, but http://sourceforge.net/p/syslog-analyzer/support-requests/35/ points to a separate ionotify configuration. What I don't understand is why I would need to mess with these values on a system designed for virtualization. Right now there are 10 VM's on the system with 6 active. There is about 18GB of RAM free and plenty more storage.
This machine is a dedicated Proxmox machine installed from the Proxmox installation disc. We only have KVM VM's on it. I don't see any additional info in the syslog on this and we've never seen this on the staging server that we have run for 3 months. If this happens again, it will be cause for alarm (we'll see what happens tonight), but even as an isolated instance, I'd like to know if anyone has any idea what would cause this to happen?
