Hi everyone,
yesterday, I have upgrade 4 Proxmox server from 6.4.13 to 7.2.7 on my 4 Dell R640. I took the opportunity to add RAM and SSD on each of them and since the migration, I have a problem with 2 nodes. Randomly, the virtual machines become very slow and the information transmitted through the web gui takes a long time to arrive. NoVNC also becomes very slow.
During the hardware upgrade,I added two NVME disks to the two problem servers (in the other two, they are SATA SSDs). These disks are Samsung PM1733 1,92Tb.These two servers were already equipped with SSD NVMe but Dell brand.
In journalctl, I see these errors during moments of slowness:
Thanks for help !
yesterday, I have upgrade 4 Proxmox server from 6.4.13 to 7.2.7 on my 4 Dell R640. I took the opportunity to add RAM and SSD on each of them and since the migration, I have a problem with 2 nodes. Randomly, the virtual machines become very slow and the information transmitted through the web gui takes a long time to arrive. NoVNC also becomes very slow.
During the hardware upgrade,I added two NVME disks to the two problem servers (in the other two, they are SATA SSDs). These disks are Samsung PM1733 1,92Tb.These two servers were already equipped with SSD NVMe but Dell brand.
In journalctl, I see these errors during moments of slowness:
Code:
Aug 11 11:06:51 xxxxx pvedaemon[527325]: VM 3503 qmp command failed - unable to open monitor socket
Aug 11 11:06:52 xxxxx pvestatd[517298]: VM 2126 qmp command failed - VM 2126 qmp command 'query-proxmox-support' failed - got timeout
Aug 11 11:06:53 xxxxx pvedaemon[518782]: VM 3503 qmp command failed - unable to open monitor socket
Aug 11 11:06:55 xxxxx pvestatd[517298]: VM 1104 qmp command failed - VM 1104 qmp command 'query-proxmox-support' failed - got timeout
Aug 11 11:06:57 xxxxx pvestatd[517298]: status update time (13.409 seconds)
Aug 11 11:07:00 xxxxx pvedaemon[536118]: VM 2105 qmp command failed - VM 2105 qmp command 'query-proxmox-support' failed - got timeout
Aug 11 11:07:01 xxxxx pvedaemon[527325]: VM 3503 qmp command failed - VM 3503 qmp command 'query-proxmox-support' failed - unable to connect to VM 3503 qmp socket - timeout after 31 retries
Aug 11 11:07:03 xxxxx pvestatd[517298]: VM 2105 qmp command failed - VM 2105 qmp command 'query-proxmox-support' failed - got timeout
Aug 11 11:07:03 xxxxx pvedaemon[518782]: VM 2105 qmp command failed - VM 2105 qmp command 'guest-ping' failed - got timeout
Aug 11 11:07:04 xxxxx systemd-udevd[536842]: Using default interface naming scheme 'v247'.
Aug 11 11:07:04 xxxxx systemd-udevd[536842]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Aug 11 11:07:06 xxxxx pvestatd[517298]: VM 2126 qmp command failed - VM 2126 qmp command 'query-proxmox-support' failed - got timeout
Aug 11 11:07:10 xxxxx pvestatd[517298]: VM 3503 qmp command failed - VM 3503 qmp command 'query-proxmox-support' failed - unable to connect to VM 3503 qmp socket - timeout after 31 retries
Aug 11 11:07:10 xxxxx pvedaemon[527325]: VM 3324 qmp command failed - VM 3324 qmp command 'query-proxmox-support' failed - got timeout
Aug 11 11:07:13 xxxxx pvestatd[517298]: VM 3210 qmp command failed - VM 3210 qmp command 'query-proxmox-support' failed - got timeout
Aug 11 11:07:19 xxxxx pvestatd[517298]: VM 3324 qmp command failed - VM 3324 qmp command 'query-proxmox-support' failed - got timeout
Aug 11 11:07:19 xxxxx pvestatd[517298]: status update time (22.832 seconds)
Aug 11 11:07:21 xxxxx pvedaemon[518782]: VM 3503 qmp command failed - VM 3503 qmp command 'query-proxmox-support' failed - unable to connect to VM 3503 qmp socket - timeout after 31 retries
Aug 11 11:07:22 xxxxx pvedaemon[536118]: VM 2105 qmp command failed - VM 2105 qmp command 'guest-ping' failed - got timeout
Aug 11 11:07:25 xxxxx pvedaemon[527325]: VM 2105 qmp command failed - VM 2105 qmp command 'query-proxmox-support' failed - got timeout
Aug 11 11:07:28 xxxxx pvedaemon[518782]: <root@pam> successful auth for user 'xx@xx'
Aug 11 11:07:28 xxxxx pmxcfs[3438]: [status] notice: received log
Aug 11 11:07:28 xxxxx promtail[2839]: ts=2022-08-11T09:07:28.810236634Z caller=log.go:168 level=info msg="Re-opening truncated file /etc/pve/.clusterlog ..."
Aug 11 11:07:28 xxxxx promtail[2839]: ts=2022-08-11T09:07:28.810631875Z caller=log.go:168 level=info msg="Successfully reopened truncated /etc/pve/.clusterlog"
Aug 11 11:07:28 xxxxx pvestatd[517298]: VM 2105 qmp command failed - VM 2105 qmp command 'query-proxmox-support' failed - got timeout
Aug 11 11:07:35 xxxxx pvestatd[517298]: VM 3503 qmp command failed - VM 3503 qmp command 'query-proxmox-support' failed - unable to connect to VM 3503 qmp socket - timeout after 31 retries
Aug 11 11:07:38 xxxxx pvestatd[517298]: VM 3210 qmp command failed - VM 3210 qmp command 'query-proxmox-support' failed - got timeout
Aug 11 11:07:41 xxxxx pvestatd[517298]: status update time (21.546 seconds)
Aug 11 11:07:41 xxxxx pvedaemon[536118]: VM 3503 qmp command failed - VM 3503 qmp command 'query-proxmox-support' failed - unable to connect to VM 3503 qmp socket - timeout after 31 retries
Aug 11 11:07:42 xxxxx pvedaemon[527325]: VM 2105 qmp command failed - VM 2105 qmp command 'guest-ping' failed - got timeout
Aug 11 11:07:49 xxxxx pvestatd[517298]: VM 2105 qmp command failed - VM 2105 qmp command 'query-proxmox-support' failed - got timeout
Aug 11 11:07:49 xxxxx pvedaemon[518782]: VM 2105 qmp command failed - VM 2105 qmp command 'query-proxmox-support' failed - got timeout
Aug 11 11:07:52 xxxxx pvestatd[517298]: VM 1104 qmp command failed - VM 1104 qmp command 'query-proxmox-support' failed - got timeout
Aug 11 11:07:55 xxxxx pvestatd[517298]: VM 2126 qmp command failed - VM 2126 qmp command 'query-proxmox-support' failed - got timeout
Aug 11 11:07:55 xxxxx pvedaemon[518782]: VM 3324 qmp command failed - VM 3324 qmp command 'query-proxmox-support' failed - got timeout
Aug 11 11:07:58 xxxxx pvestatd[517298]: VM 3503 qmp command failed - VM 3503 qmp command 'query-proxmox-support' failed - unable to connect to VM 3503 qmp socket - timeout after 31 retries
Aug 11 11:08:01 xxxxx pvestatd[517298]: VM 3501 qmp command failed - VM 3501 qmp command 'query-proxmox-support' failed - got timeout
Aug 11 11:08:01 xxxxx pvedaemon[527325]: VM 3503 qmp command failed - VM 3503 qmp command 'query-proxmox-support' failed - unable to connect to VM 3503 qmp socket - timeout after 31 retries
Aug 11 11:08:03 xxxxx pvestatd[517298]: status update time (22.007 seconds)
Aug 11 11:08:03 xxxxx pvedaemon[518782]: VM 2105 qmp command failed - VM 2105 qmp command 'guest-ping' failed - got timeout
Aug 11 11:08:10 xxxxx pvestatd[517298]: VM 2126 qmp command failed - VM 2126 qmp command 'query-proxmox-support' failed - got timeout
Aug 11 11:08:11 xxxxx pvedaemon[527325]: VM 3324 qmp command failed - VM 3324 qmp command 'query-proxmox-support' failed - got timeout
Aug 11 11:08:13 xxxxx pvedaemon[518782]: VM 2105 qmp command failed - VM 2105 qmp command 'query-proxmox-support' failed - got timeout
Aug 11 11:08:13 xxxxx pvestatd[517298]: VM 1104 qmp command failed - VM 1104 qmp command 'query-proxmox-support' failed - got timeout
Aug 11 11:08:16 xxxxx pvestatd[517298]: VM 3503 qmp command failed - VM 3503 qmp command 'query-proxmox-support' failed - unable to connect to VM 3503 qmp socket - timeout after 31 retries
Aug 11 11:08:21 xxxxx pvestatd[517298]: VM 2105 qmp command failed - VM 2105 qmp command 'query-proxmox-support' failed - got timeout
Aug 11 11:08:21 xxxxx pvedaemon[518782]: VM 3503 qmp command failed - VM 3503 qmp command 'query-proxmox-support' failed - unable to connect to VM 3503 qmp socket - timeout after 31 retries
Aug 11 11:08:22 xxxxx pvedaemon[536118]: VM 2105 qmp command failed - VM 2105 qmp command 'guest-ping' failed - got timeout
Aug 11 11:08:24 xxxxx pvestatd[517298]: VM 3210 qmp command failed - VM 3210 qmp command 'query-proxmox-support' failed - got timeout
Aug 11 11:08:24 xxxxx pvedaemon[527325]: VM 3324 qmp command failed - VM 3324 qmp command 'query-proxmox-support' failed - got timeout
Aug 11 11:08:27 xxxxx pvestatd[517298]: VM 3324 qmp command failed - VM 3324 qmp command 'query-proxmox-support' failed - got timeoutThanks for help !
Last edited:
