PVE 3 stops responding overnight

[anubis]

Hi All,

I'm in the process of setting up a new server with Proxmox 3 on it. It's running about 7 VM's. Last night, only two were powered on and not doing anything (as I'm still in the process of setting the server up) but when I came into work this morning, the server had stopped responding.

It's now done this twice, it did it the day before also.

I've checked through all the logs and can't find anything that might be the cause. Below I've copied what I get from Syslog. It appears the server is responding until around 1am then simply stops. I've also attached to this thread the syslogs from my reboot in the morning in case something stands out.

Has anyone else encountered this problem?

Thanks
Anubis.

Sep  4 00:00:38 vsv001 pveproxy[4027]: worker 26685 finished
Sep  4 00:00:38 vsv001 pveproxy[4027]: starting 1 worker(s)
Sep  4 00:00:38 vsv001 pveproxy[4027]: worker 29250 started
Sep  4 00:02:39 vsv001 pveproxy[4027]: worker 27415 finished
Sep  4 00:02:39 vsv001 pveproxy[4027]: starting 1 worker(s)
Sep  4 00:02:39 vsv001 pveproxy[4027]: worker 29351 started
Sep  4 00:13:09 vsv001 pvedaemon[27039]: <root@pam> successful auth for user 'root@pam'
Sep  4 00:13:49 vsv001 pvedaemon[3998]: worker 26879 finished
Sep  4 00:13:49 vsv001 pvedaemon[3998]: starting 1 worker(s)
Sep  4 00:13:49 vsv001 pvedaemon[3998]: worker 29911 started
Sep  4 00:17:01 vsv001 /USR/SBIN/CRON[30085]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Sep  4 00:25:49 vsv001 pveproxy[4027]: worker 28338 finished
Sep  4 00:25:49 vsv001 pveproxy[4027]: starting 1 worker(s)
Sep  4 00:25:49 vsv001 pveproxy[4027]: worker 30517 started
Sep  4 00:28:10 vsv001 pvedaemon[29911]: <root@pam> successful auth for user 'root@pam'
Sep  4 00:38:02 vsv001 pveproxy[4027]: worker 29250 finished
Sep  4 00:38:02 vsv001 pveproxy[4027]: starting 1 worker(s)
Sep  4 00:38:02 vsv001 pveproxy[4027]: worker 31138 started
Sep  4 00:43:11 vsv001 pvedaemon[28453]: <root@pam> successful auth for user 'root@pam'
Sep  4 00:45:58 vsv001 pveproxy[4027]: worker 29351 finished
Sep  4 00:45:58 vsv001 pveproxy[4027]: starting 1 worker(s)
Sep  4 00:45:58 vsv001 pveproxy[4027]: worker 31532 started
Sep  4 00:47:35 vsv001 rrdcached[3771]: flushing old values
Sep  4 00:47:35 vsv001 rrdcached[3771]: rotating journals
Sep  4 00:47:35 vsv001 rrdcached[3771]: started new journal /var/lib/rrdcached/journal/rrd.journal.1378216055.983644
Sep  4 00:47:35 vsv001 rrdcached[3771]: removing old journal /var/lib/rrdcached/journal/rrd.journal.1378208855.983699
Sep  4 00:55:11 vsv001 pvedaemon[3998]: worker 28453 finished
Sep  4 00:55:11 vsv001 pvedaemon[3998]: starting 1 worker(s)
Sep  4 00:55:11 vsv001 pvedaemon[3998]: worker 31999 started
Sep  4 00:58:12 vsv001 pvedaemon[31999]: <root@pam> successful auth for user 'root@pam'
Sep  4 01:08:15 vsv001 pvedaemon[3998]: worker 29911 finished
Sep  4 01:08:15 vsv001 pvedaemon[3998]: starting 1 worker(s)
Sep  4 01:08:15 vsv001 pvedaemon[3998]: worker 32651 started
Sep  4 01:10:42 vsv001 pveproxy[4027]: worker 30517 finished
Sep  4 01:10:42 vsv001 pveproxy[4027]: starting 1 worker(s)
Sep  4 01:10:42 vsv001 pveproxy[4027]: worker 32774 started
Sep  4 01:13:13 vsv001 pvedaemon[27039]: <root@pam> successful auth for user 'root@pam'
Sep  4 01:17:01 vsv001 /USR/SBIN/CRON[33098]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Sep  4 01:21:00 vsv001 pveproxy[4027]: worker 31138 finished
Sep  4 01:21:00 vsv001 pveproxy[4027]: starting 1 worker(s)
Sep  4 01:21:00 vsv001 pveproxy[4027]: worker 33300 started
Sep  4 01:22:15 vsv001 pvedaemon[3998]: worker 27039 finished
Sep  4 01:22:15 vsv001 pvedaemon[3998]: starting 1 worker(s)
Sep  4 01:22:15 vsv001 pvedaemon[3998]: worker 33359 started
Sep  4 07:22:44 vsv001 kernel: imklog 5.8.11, log source = /proc/kmsg started.
Sep  4 07:22:44 vsv001 rsyslogd: [origin software="rsyslogd" swVersion="5.8.11" x-pid="3672" x-info="http://www.rsyslog.com"] start
Sep  4 07:22:44 vsv001 kernel: Initializing cgroup subsys cpuset
Sep  4 07:22:44 vsv001 kernel: Initializing cgroup subsys cpu
Sep  4 07:22:44 vsv001 kernel: Linux version 2.6.32-20-pve (root@lola) (gcc version 4.7.2 (Debian 4.7.2-5) ) #1 SMP Wed May 15 08:23:27 CEST 2013
Sep  4 07:22:44 vsv001 kernel: Command line: BOOT_IMAGE=/vmlinuz-2.6.32-20-pve root=/dev/mapper/pve-root ro quiet
Sep  4 07:22:44 vsv001 kernel: KERNEL supported cpus:
Sep  4 07:22:44 vsv001 kernel:  Intel GenuineIntel
Sep  4 07:22:44 vsv001 kernel:  AMD AuthenticAMD
Sep  4 07:22:44 vsv001 kernel:  Centaur CentaurHauls

 

[jdrews]

Might want to check /var/log/messages to see if it's some problem with debian...

 

[e100]

I suspect that the kernel panicked.
If the kernel panicked the console would display some information that would be useful in figuring out what is wrong.
This info is never written to disk, only to the console.

I have had a few kernel panics with Proxmox over the years, usually it is a hardware issue and bad RAM has caused this more than once for me.
Just replaced a module Friday from the most recent RAM failure, sometimes even ECC RAM fails to correct errors.
There was at least one time the problem was a bug in the kernel and the Proxmox team quickly applied the patch I found that resolved the problem.

You can setup a serial console or netconsole to log the kernel panic, details for both can be found here:
http://serverfault.com/questions/245530/how-can-i-see-logs-in-a-server-after-a-kernel-panic-hang

Once you have the stack trace post it here and hopefully someone will be able to decipher it.