We have a PVE cluster with three nodes and about 70 VMs, of which probably 50% are Windows guests. In normal operation, they're balanced across the three nodes.
We regularly patch the nodes, so we free one of the nodes, install updates, reboot, then free the next node etc., and this usually goes through without issues.
Today, I freed node 3, upgraded and rebooted it, then moved the VMs back. While moving them back, three Windows VMs had a migration error:
The VMs were transferred but switched off and had to be turned on manually. I've grepped the syslog for the VM IDs and found this:
The PIDs and timestamps match the three VMs with the migration problems, so I am somewhat confident that this is related.
The apt history log also confirms that TPM packages got updated today when I patched the server:
Interestingly, two of the syslog errors end in
It sounds somewhat reasonable to me that a VM migration can fail with a TPM related error if the TPM packages differ in version, even though of course it shouldn't. I just don't understand why it only happened to a few of the Windows VMs, even though they apparently all use TPM.
I'd like to understand what caused this and if there's anything I can do to avoid this happening again in the future. Needless to say, the customer wasn't very happy about the unexpected outage and unclean shutdown...
We regularly patch the nodes, so we free one of the nodes, install updates, reboot, then free the next node etc., and this usually goes through without issues.
Today, I freed node 3, upgraded and rebooted it, then moved the VMs back. While moving them back, three Windows VMs had a migration error:
Code:
2026-11-11 16:54:07 ERROR: tunnel replied 'ERR: resume failed - VM 1110120 not running' to command 'resume 1110120'
2026-11-11 16:56:48 ERROR: tunnel replied 'ERR: resume failed - VM 1230720 qmp command 'query-status' failed - client closed connection' to command 'resume 1230720'
2026-11-11 17:00:36 ERROR: tunnel replied 'ERR: resume failed - VM 43149 qmp command 'query-status' failed - client closed connection' to command 'resume 43149'The VMs were transferred but switched off and had to be turned on manually. I've grepped the syslog for the VM IDs and found this:
Code:
2026-11-11T16:54:07.684453+01:00 node3 QEMU[16937]: kvm: load of migration failed: Invalid argument: error while loading state for instance 0x0 of device 'tpm-emulator': post load hook failed for: tpm-emulator, version_id: 0, minimum_version: 0: tpm-emulator: Setting the stateblob (type 1) failed with a TPM error 0x3 a parameter is bad
2026-11-11T16:56:47.830592+01:00 node3 QEMU[32557]: kvm: load of migration failed: Invalid argument: error while loading state for instance 0x0 of device 'tpm-emulator': post load hook failed for: tpm-emulator, version_id: 0, minimum_version: 0: tpm-emulator: Setting the stateblob (type 1) failed with a TPM error 0x3 a parameter is bad
2026-11-11T17:00:36.415358+01:00 node3 QEMU[48693]: kvm: load of migration failed: Invalid argument: error while loading state for instance 0x0 of device 'tpm-emulator': post load hook failed for: tpm-emulator, version_id: 0, minimum_version: 0: tpm-emulator: Setting the stateblob (type 1) failed with a TPM error 0x95The PIDs and timestamps match the three VMs with the migration problems, so I am somewhat confident that this is related.
The apt history log also confirms that TPM packages got updated today when I patched the server:
Code:
swtpm-libs:amd64 (0.8.0+pve3, 0.10.2+pve1)
swtpm-tools:amd64 (0.8.0+pve3, 0.10.2+pve1)
swtpm:amd64 (0.8.0+pve3, 0.10.2+pve1)
libtpms0:amd64 (0.9.7+pve2, 0.10.2+pve1)Interestingly, two of the syslog errors end in
TPM error 0x3 a parameter is bad and one ends in TPM error 0x95.It sounds somewhat reasonable to me that a VM migration can fail with a TPM related error if the TPM packages differ in version, even though of course it shouldn't. I just don't understand why it only happened to a few of the Windows VMs, even though they apparently all use TPM.
I'd like to understand what caused this and if there's anything I can do to avoid this happening again in the future. Needless to say, the customer wasn't very happy about the unexpected outage and unclean shutdown...
