Host is solid- No problems. Overnight the containers and VM's crash. I reviewed syslog and only found 1 set of errors pointing to the containers and VM's. I checked system time and it is accurate. I verified network time on and NTP synchronized both = yes. Currently running memtester and no errors so far but do not expect hardware errors since the host is working fine. Here is the error even though I am not confident this is causing the problem.
May 11 01:12:42 vh0 rrdcached[1871]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/100) failed with status -1. (/var/lib/rrdcached/db/pve2-vm/100: illegal attempt to update using time 1557554061 when last update time is 1557554351 (minimum one second step))
May 11 01:12:42 vh0 rrdcached[1871]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/202) failed with status -1. (/var/lib/rrdcached/db/pve2-vm/202: illegal attempt to update using time 1557554061 when last update time is 1557554351 (minimum one second step))
May 11 01:12:42 vh0 rrdcached[1871]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/201) failed with status -1. (/var/lib/rrdcached/db/pve2-vm/201: illegal attempt to update using time 1557554061 when last update time is 1557554351 (minimum one second step))
May 11 01:12:43 vh0 rrdcached[1871]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-storage/drk-vh0/local) failed with status -1. (/var/lib/rrdcached/db/pve2-storage/drk-vh0/local: illegal attempt to update using time 1557554071 when last update time is 1557554361 (minimum one second step))
May 11 01:12:43 vh0 rrdcached[1871]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-storage/drk-vh0/local-lvm) failed with status -1. (/var/lib/rrdcached/db/pve2-storage/drk-vh0/local-lvm: illegal attempt to update using time 1557554071 when last update time is 1557554361 (minimum one second step))
May 11 01:12:42 vh0 rrdcached[1871]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/100) failed with status -1. (/var/lib/rrdcached/db/pve2-vm/100: illegal attempt to update using time 1557554061 when last update time is 1557554351 (minimum one second step))
May 11 01:12:42 vh0 rrdcached[1871]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/202) failed with status -1. (/var/lib/rrdcached/db/pve2-vm/202: illegal attempt to update using time 1557554061 when last update time is 1557554351 (minimum one second step))
May 11 01:12:42 vh0 rrdcached[1871]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/201) failed with status -1. (/var/lib/rrdcached/db/pve2-vm/201: illegal attempt to update using time 1557554061 when last update time is 1557554351 (minimum one second step))
May 11 01:12:43 vh0 rrdcached[1871]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-storage/drk-vh0/local) failed with status -1. (/var/lib/rrdcached/db/pve2-storage/drk-vh0/local: illegal attempt to update using time 1557554071 when last update time is 1557554361 (minimum one second step))
May 11 01:12:43 vh0 rrdcached[1871]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-storage/drk-vh0/local-lvm) failed with status -1. (/var/lib/rrdcached/db/pve2-storage/drk-vh0/local-lvm: illegal attempt to update using time 1557554071 when last update time is 1557554361 (minimum one second step))