Hi !
I have a huge vm that is frozen some mornings and when it happens, the backup in progress seems frozen as well...
It's a windows server with lots of softwares and users, so i didn't suspect the hypervisor, but it's getting more and more frequent, and started to happen on it's neighbour, an other windows much cleaner (ADDC)...
I'm suspecting the backup since it happens only at night, plus i have a lot of "Volsnap" errors in the logs before the crash... ( sorry for the french)
Does the backup task use windows VSS to make backups ? guest agent is on and installed and backup is in snapshot mode.
Thanks for any lead you could give me and feel free to ask more details
Here some syslogs
I have a huge vm that is frozen some mornings and when it happens, the backup in progress seems frozen as well...
It's a windows server with lots of softwares and users, so i didn't suspect the hypervisor, but it's getting more and more frequent, and started to happen on it's neighbour, an other windows much cleaner (ADDC)...
I'm suspecting the backup since it happens only at night, plus i have a lot of "Volsnap" errors in the logs before the crash... ( sorry for the french)
Does the backup task use windows VSS to make backups ? guest agent is on and installed and backup is in snapshot mode.
Thanks for any lead you could give me and feel free to ask more details
Here some syslogs
Code:
Oct 09 00:00:08 pve9 systemd[1]: Reloading pveproxy.service - PVE API Proxy Server...
Oct 09 00:00:09 pve9 pveproxy[444628]: send HUP to 1021179
Oct 09 00:00:09 pve9 pveproxy[1021179]: received signal HUP
Oct 09 00:00:09 pve9 pveproxy[1021179]: server closing
Oct 09 00:00:09 pve9 pveproxy[1021179]: server shutdown (restart)
Oct 09 00:00:09 pve9 systemd[1]: Reloaded pveproxy.service - PVE API Proxy Server.
Oct 09 00:00:09 pve9 systemd[1]: Reloading spiceproxy.service - PVE SPICE Proxy Server...
Oct 09 00:00:10 pve9 spiceproxy[444631]: send HUP to 2015
Oct 09 00:00:10 pve9 spiceproxy[2015]: received signal HUP
Oct 09 00:00:10 pve9 spiceproxy[2015]: server closing
Oct 09 00:00:10 pve9 spiceproxy[2015]: server shutdown (restart)
Oct 09 00:00:10 pve9 systemd[1]: Reloaded spiceproxy.service - PVE SPICE Proxy Server.
Oct 09 00:00:10 pve9 pvefw-logger[3804283]: received terminate request (signal)
Oct 09 00:00:10 pve9 pvefw-logger[3804283]: stopping pvefw logger
Oct 09 00:00:10 pve9 systemd[1]: Stopping pvefw-logger.service - Proxmox VE firewall logger...
Oct 09 00:00:10 pve9 systemd[1]: pvefw-logger.service: Deactivated successfully.
Oct 09 00:00:10 pve9 systemd[1]: Stopped pvefw-logger.service - Proxmox VE firewall logger.
Oct 09 00:00:10 pve9 systemd[1]: pvefw-logger.service: Consumed 7.580s CPU time.
Oct 09 00:00:10 pve9 systemd[1]: Starting pvefw-logger.service - Proxmox VE firewall logger...
Oct 09 00:00:10 pve9 systemd[1]: Started pvefw-logger.service - Proxmox VE firewall logger.
Oct 09 00:00:10 pve9 pvefw-logger[444642]: starting pvefw logger
Oct 09 00:00:10 pve9 systemd[1]: logrotate.service: Deactivated successfully.
Oct 09 00:00:10 pve9 systemd[1]: Finished logrotate.service - Rotate log files.
Oct 09 00:00:10 pve9 spiceproxy[2015]: restarting server
Oct 09 00:00:10 pve9 spiceproxy[2015]: starting 1 worker(s)
Oct 09 00:00:10 pve9 spiceproxy[2015]: worker 444647 started
Oct 09 00:00:11 pve9 pveproxy[1021179]: restarting server
Oct 09 00:00:11 pve9 pveproxy[1021179]: starting 3 worker(s)
Oct 09 00:00:11 pve9 pveproxy[1021179]: worker 444648 started
Oct 09 00:00:11 pve9 pveproxy[1021179]: worker 444649 started
Oct 09 00:00:11 pve9 pveproxy[1021179]: worker 444650 started
Oct 09 00:00:15 pve9 spiceproxy[3804288]: worker exit
Oct 09 00:00:15 pve9 spiceproxy[2015]: worker 3804288 finished
Oct 09 00:00:16 pve9 pveproxy[222845]: worker exit
Oct 09 00:00:16 pve9 pveproxy[240805]: worker exit
Oct 09 00:00:16 pve9 pveproxy[228768]: worker exit
Oct 09 00:00:16 pve9 pveproxy[1021179]: worker 222845 finished
Oct 09 00:00:16 pve9 pveproxy[1021179]: worker 240805 finished
Oct 09 00:00:16 pve9 pveproxy[1021179]: worker 228768 finished
Oct 09 00:12:52 pve9 pvestatd[1967]: VM 911 qmp command failed - VM 911 qmp command 'query-proxmox-support' failed - unable to connect to VM 911 qmp socket - timeout after 51 retries
Oct 09 00:12:52 pve9 pvestatd[1967]: status update time (8.355 seconds)
Oct 09 00:13:02 pve9 pvestatd[1967]: VM 911 qmp command failed - VM 911 qmp command 'query-proxmox-support' failed - unable to connect to VM 911 qmp socket - timeout after 51 retries
Oct 09 00:13:02 pve9 pvestatd[1967]: status update time (8.333 seconds)
Oct 09 00:13:12 pve9 pvestatd[1967]: VM 911 qmp command failed - VM 911 qmp command 'query-proxmox-support' failed - unable to connect to VM 911 qmp socket - timeout after 51 retries
Oct 09 00:13:13 pve9 pvestatd[1967]: status update time (8.606 seconds)
Oct 09 00:13:22 pve9 pvestatd[1967]: VM 911 qmp command failed - VM 911 qmp command 'query-proxmox-support' failed - unable to connect to VM 911 qmp socket - timeout after 51 retries
Oct 09 00:13:23 pve9 pvestatd[1967]: status update time (8.599 seconds)
Oct 09 00:13:32 pve9 pvestatd[1967]: VM 911 qmp command failed - VM 911 qmp command 'query-proxmox-support' failed - unable to connect to VM 911 qmp socket - timeout after 51 retries
Oct 09 00:13:32 pve9 pvestatd[1967]: status update time (8.317 seconds)
Oct 09 00:13:42 pve9 pvestatd[1967]: VM 911 qmp command failed - VM 911 qmp command 'query-proxmox-support' failed - unable to connect to VM 911 qmp socket - timeout after 51 retries
Oct 09 00:13:42 pve9 pvestatd[1967]: status update time (8.334 seconds)
Oct 09 00:13:51 pve9 pvestatd[1967]: status update time (6.378 seconds)
Oct 09 00:14:13 pve9 pvestatd[1967]: VM 911 qmp command failed - VM 911 qmp command 'query-proxmox-support' failed - unable to connect to VM 911 qmp socket - timeout after 51 retries
Oct 09 00:14:13 pve9 pvestatd[1967]: status update time (8.355 seconds)
Oct 09 00:14:22 pve9 pvestatd[1967]: VM 911 qmp command failed - VM 911 qmp command 'query-proxmox-support' failed - unable to connect to VM 911 qmp socket - timeout after 51 retries
Oct 09 00:14:22 pve9 pvestatd[1967]: status update time (8.300 seconds)
Oct 09 00:14:32 pve9 pvestatd[1967]: VM 911 qmp command failed - VM 911 qmp command 'query-proxmox-support' failed - unable to connect to VM 911 qmp socket - timeout after 51 retries
Oct 09 00:14:33 pve9 pvestatd[1967]: status update time (8.375 seconds)
Oct 09 00:14:42 pve9 pvestatd[1967]: VM 911 qmp command failed - VM 911 qmp command 'query-proxmox-support' failed - unable to connect to VM 911 qmp socket - timeout after 51 retries
Oct 09 00:14:42 pve9 pvestatd[1967]: status update time (8.333 seconds)
Oct 09 00:14:52 pve9 pvestatd[1967]: VM 911 qmp command failed - VM 911 qmp command 'query-proxmox-support' failed - unable to connect to VM 911 qmp socket - timeout after 51 retries
Oct 09 00:14:52 pve9 pvestatd[1967]: status update time (8.321 seconds)
Oct 09 00:15:02 pve9 pvestatd[1967]: VM 911 qmp command failed - VM 911 qmp command 'query-proxmox-support' failed - unable to connect to VM 911 qmp socket - timeout after 51 retries
Oct 09 00:15:03 pve9 pvestatd[1967]: status update time (8.335 seconds)
Oct 09 00:15:12 pve9 pvestatd[1967]: VM 911 qmp command failed - VM 911 qmp command 'query-proxmox-support' failed - unable to connect to VM 911 qmp socket - timeout after 51 retries
Oct 09 00:15:12 pve9 pvestatd[1967]: status update time (8.358 seconds)
Oct 09 00:15:22 pve9 pvestatd[1967]: VM 911 qmp command failed - VM 911 qmp command 'query-proxmox-support' failed - unable to connect to VM 911 qmp socket - timeout after 51 retries
Oct 09 00:15:22 pve9 pvestatd[1967]: status update time (8.346 seconds)
Oct 09 00:15:32 pve9 pvestatd[1967]: VM 911 qmp command failed - VM 911 qmp command 'query-proxmox-support' failed - unable to connect to VM 911 qmp socket - timeout after 51 retries
Oct 09 00:15:33 pve9 pvestatd[1967]: status update time (8.301 seconds)
Oct 09 00:15:42 pve9 pvestatd[1967]: VM 911 qmp command failed - VM 911 qmp command 'query-proxmox-support' failed - unable to connect to VM 911 qmp socket - timeout after 51 retries
Oct 09 00:15:42 pve9 pvestatd[1967]: status update time (8.325 seconds)
Oct 09 00:15:52 pve9 pvestatd[1967]: status update time (7.946 seconds)
Oct 09 00:16:47 pve9 systemd[1]: Starting man-db.service - Daily man-db regeneration...
Oct 09 00:16:48 pve9 systemd[1]: man-db.service: Deactivated successfully.
Oct 09 00:16:48 pve9 systemd[1]: Finished man-db.service - Daily man-db regeneration.
Oct 09 00:17:01 pve9 CRON[453912]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Oct 09 00:17:01 pve9 CRON[453913]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Oct 09 00:17:01 pve9 CRON[453912]: pam_unix(cron:session): session closed for user root
Oct 09 00:24:01 pve9 CRON[457723]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Oct 09 00:24:01 pve9 CRON[457724]: (root) CMD (if [ $(date +%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/scrub ]; then /usr/lib/zfs-linux/scrub; fi)
Oct 09 00:24:01 pve9 CRON[457723]: pam_unix(cron:session): session closed for user root
Oct 09 01:05:28 pve9 pvestatd[1967]: PBS1: error fetching datastores - 500 read timeout
Oct 09 01:05:28 pve9 pvestatd[1967]: status update time (14.149 seconds)
Oct 09 01:05:35 pve9 pvestatd[1967]: PBS1: error fetching datastores - 500 Can't connect to 172.16.10.103:8007
Oct 09 01:05:35 pve9 pvestatd[1967]: status update time (7.165 seconds)
Oct 09 01:05:47 pve9 systemd[1]: Starting systemd-tmpfiles-clean.service - Cleanup of Temporary Directories...
Oct 09 01:05:47 pve9 systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Oct 09 01:05:47 pve9 systemd[1]: Finished systemd-tmpfiles-clean.service - Cleanup of Temporary Directories.
Oct 09 01:05:47 pve9 systemd[1]: run-credentials-systemd\x2dtmpfiles\x2dclean.service.mount: Deactivated successfully.
Oct 09 01:09:05 pve9 pvestatd[1967]: status update time (6.670 seconds)
Oct 09 01:10:06 pve9 pvestatd[1967]: status update time (47.763 seconds)
Oct 09 01:10:23 pve9 pvestatd[1967]: PBS1: error fetching datastores - 500 read timeout
Oct 09 01:10:23 pve9 pvestatd[1967]: status update time (16.962 seconds)
Oct 09 01:10:47 pve9 pvestatd[1967]: status update time (24.059 seconds)
Oct 09 01:11:12 pve9 pvestatd[1967]: status update time (25.065 seconds)
Oct 09 01:11:18 pve9 pvestatd[1967]: status update time (5.476 seconds)
Oct 09 01:16:10 pve9 pvestatd[1967]: status update time (8.704 seconds)
Oct 09 01:17:01 pve9 CRON[485906]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Oct 09 01:17:01 pve9 CRON[485907]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Oct 09 01:17:01 pve9 CRON[485906]: pam_unix(cron:session): session closed for user root
Oct 09 01:19:10 pve9 pvestatd[1967]: status update time (17.948 seconds)
Oct 09 01:21:48 pve9 pvestatd[1967]: status update time (7.564 seconds)
Oct 09 01:25:00 pve9 pvestatd[1967]: status update time (9.301 seconds)
Oct 09 01:25:58 pve9 pvestatd[1967]: status update time (17.870 seconds)
Oct 09 01:26:20 pve9 pvestatd[1967]: status update time (21.794 seconds)
Oct 09 01:26:39 pve9 pvestatd[1967]: status update time (19.465 seconds)
Oct 09 01:28:20 pve9 pvestatd[1967]: status update time (21.023 seconds)
Oct 09 01:29:30 pve9 pvedaemon[180746]: <root@pam> successful auth for user 'root@pam'
Oct 09 01:31:05 pve9 pvestatd[1967]: status update time (5.311 seconds)
Oct 09 01:39:09 pve9 pvestatd[1967]: status update time (8.854 seconds)
Oct 09 01:39:27 pve9 pvestatd[1967]: PBS1: error fetching datastores - 500 Can't connect to 172.16.10.103:8007
Oct 09 01:39:27 pve9 pvestatd[1967]: status update time (7.172 seconds)
Oct 09 01:41:31 pve9 pvestatd[1967]: status update time (10.986 seconds)
Oct 09 01:41:52 pve9 pvestatd[1967]: status update time (10.633 seconds)
Oct 09 01:42:41 pve9 pvestatd[1967]: status update time (19.408 seconds)
Oct 09 01:48:48 pve9 pvestatd[1967]: status update time (7.098 seconds)
Oct 09 01:50:38 pve9 pvestatd[1967]: status update time (17.469 seconds)
Oct 09 01:50:49 pve9 pvestatd[1967]: status update time (11.325 seconds)
Oct 09 01:50:54 pve9 pvestatd[1967]: status update time (5.072 seconds)
Oct 09 01:54:48 pve9 pvestatd[1967]: status update time (8.787 seconds)
Oct 09 01:55:07 pve9 pvestatd[1967]: status update time (18.033 seconds)
Oct 09 01:57:33 pve9 pvestatd[1967]: status update time (15.153 seconds)
Oct 09 01:58:09 pve9 pvestatd[1967]: status update time (6.224 seconds)
Oct 09 02:00:16 pve9 QEMU[808470]: kvm: Desc next is 2
Oct 09 02:01:58 pve9 pvestatd[1967]: status update time (5.250 seconds)
Oct 09 02:08:01 pve9 pvestatd[1967]: status update time (7.718 seconds)
Last edited: