Hi,
Proxmox VE 9.2 running on a 2U, 4 nodes test cluster.
All 4 nodes started with an SSD as the boot drive (ext4/LVM). 3 out of 4 have a 2TB mechanical drive to experiment with ZFS replication, all with a pool called "local-zfs01". I added a mechanical drive to the 4th and started adding it to the replication configuration for the existing 3 VMs that use it (most of the other VMs are on an iSCSI SAN), as well as moving storage for one last VM that had its disk image temporarily on "local-lvm".
Forgot the reason the 4th didn't have an extra drive was due to backplane issues and sure enough, 9k+ IO errors, disk gone, ZFS pool suspended, all replication is stalled and I can't delete the latest replication jobs -- which I guess expects being able to reclaim disk space on the target ZFS pool.
I hotswaped the drive in and out and attempted an zpool clear as suggested
but no dice
Clearing the drive with parted for a zpool replace fails.
Quite a few zfs processes just hanging about since around 11:00 am CET today:
that won't even respond to a
The drive is detected by the host but that's about it.
None of the VMs currently on ZFS with replication are critical (hence my less than subtle attempts to pruge the cluster from the "failed" mechanical drive) but I'd really like to get out of limbo.
Any ideas?
Update 2026-10-27:
With a cooler head I worked my way backwards to track and kill some processes to remediate the replication stall, but the node with the "SUSPENDED" ZFS pool ended up becoming unresponsive from the GUI (HA standby, connection errors) as I moved the drive to a USB-dock; ZFS is fighting that hard for a drive it cannot access huh... Lessons learned.
Anyway, since HA was gone for the last "Important" VM I had running on that node, I used
which worked OK, except keeping contrack-state was a mistake (the VM is an ELK stack for our staging environment though, the less downtime the better). Anyway, the last console output I got was:
The VM is up and running on the target, I can log in Kibana and index health is green, but the VM status in the Proxmox control panel still has the paper plane icon, "ERROR: received interrupt - delayed" happened as I am writing this.
I was able to put the misbehaving node in maintenance using ha-manager.
Right now I have 2 questions:
- Is the misbehaving node going to try and recover HA on the ELK VM when I put it out of maintenance? Failback was on when HA was set up for that VM.
- How do I clear the migration status on that ELK VM? Another migration to a healthy node?
Update #2:
I've found no trace of the misbehaving node nor the ELK VM in the /etc/pve/ha files so I should be in the clear.
The busted ZFS pool is now neither listed through
Update #3: https://forum.proxmox.com/threads/vm-is-locked-migrate.9358/ is still relevant,
Proxmox VE 9.2 running on a 2U, 4 nodes test cluster.
All 4 nodes started with an SSD as the boot drive (ext4/LVM). 3 out of 4 have a 2TB mechanical drive to experiment with ZFS replication, all with a pool called "local-zfs01". I added a mechanical drive to the 4th and started adding it to the replication configuration for the existing 3 VMs that use it (most of the other VMs are on an iSCSI SAN), as well as moving storage for one last VM that had its disk image temporarily on "local-lvm".
Forgot the reason the 4th didn't have an extra drive was due to backplane issues and sure enough, 9k+ IO errors, disk gone, ZFS pool suspended, all replication is stalled and I can't delete the latest replication jobs -- which I guess expects being able to reclaim disk space on the target ZFS pool.
I hotswaped the drive in and out and attempted an zpool clear as suggested
Code:
# zpool status
pool: local-zfs01
state: SUSPENDED
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-HC
config:
NAME STATE READ WRITE CKSUM
local-zfs01 UNAVAIL 0 0 0 insufficient replicas
ata-WDC_WD60EFPX-68C5ZN0_WD-WX12D842S0L5 UNAVAIL 0 0 0
errors: 9536 data errors, use '-v' for a list
Code:
# zpool clear local-zfs01
cannot clear errors for local-zfs01: I/O errorClearing the drive with parted for a zpool replace fails.
Quite a few zfs processes just hanging about since around 11:00 am CET today:
Code:
# ps faux | grep zfs
root 3244114 0.0 0.0 158552 105968 ? Ss 10:50 0:01 | \_ /usr/bin/perl /usr/sbin/pvesm import local-zfs01:vm-101-disk-0 zfs - -with-snapshots 1 -snapshot __replicate_101-2_1793008235__ -allow-rename 0
root 3244116 0.0 0.0 14720 6572 ? S 10:50 0:03 | \_ zfs recv -F -x encryption -- local-zfs01/vm-101-disk-0
root 3244454 0.0 0.0 158472 105804 ? Ss 10:51 0:02 | \_ /usr/bin/perl /usr/sbin/pvesm import local-zfs01:vm-102-disk-0 zfs - -with-snapshots 1 -snapshot __replicate_102-2_1793008266__ -allow-rename 0
root 3244464 0.0 0.0 14720 6604 ? D 10:51 0:00 | \_ zfs recv -F -x encryption -- local-zfs01/vm-102-disk-0
root 3720668 0.0 0.0 6528 2380 pts/0 S+ 16:39 0:00 \_ grep zfs
root 3570382 0.0 0.1 240656 157912 ? Ss 11:34 0:01 \_ task UPID:pio-host06:00367ACE:138E2C22:6ADF2CA8:zfsremove:local-zfs01:julien.pinget@safespace:
root 3570435 0.0 0.0 14480 6264 ? D 11:34 0:00 \_ zpool destroy local-zfs01
root 3245406 0.0 0.0 14972 6628 ? D 10:53 0:00 zfs create -V 104857600k local-zfs01/vm-124-disk-0kill -9The drive is detected by the host but that's about it.
None of the VMs currently on ZFS with replication are critical (hence my less than subtle attempts to pruge the cluster from the "failed" mechanical drive) but I'd really like to get out of limbo.
Any ideas?
Update 2026-10-27:
With a cooler head I worked my way backwards to track and kill some processes to remediate the replication stall, but the node with the "SUSPENDED" ZFS pool ended up becoming unresponsive from the GUI (HA standby, connection errors) as I moved the drive to a USB-dock; ZFS is fighting that hard for a drive it cannot access huh... Lessons learned.
Anyway, since HA was gone for the last "Important" VM I had running on that node, I used
Code:
qm migrate <vmid> <target host> --migration_type insecure --online 1 --targetstorage (our SAN ID) --with-conntrack-state 1
Code:
2026-10-27 15:32:05 average migration speed: 103.4 MiB/s - downtime 307 ms
2026-10-27 15:32:05 migration completed, transferred 34.6 GiB VM-state
2026-10-27 15:32:05 migration status: completed
2026-10-27 15:32:05 stopping migration dbus-vmstate helpers
2026-10-27 15:32:05 migrated 0 conntrack state entries
2026-10-27 15:32:08 flushing conntrack state for guest on source node
2026-10-27 15:56:07 ERROR: received interrupt - delayedI was able to put the misbehaving node in maintenance using ha-manager.
Right now I have 2 questions:
- Is the misbehaving node going to try and recover HA on the ELK VM when I put it out of maintenance? Failback was on when HA was set up for that VM.
- How do I clear the migration status on that ELK VM? Another migration to a healthy node?
Update #2:
I've found no trace of the misbehaving node nor the ELK VM in the /etc/pve/ha files so I should be in the clear.
The busted ZFS pool is now neither listed through
zpool status nor found through zpool import but still present with a ? in the Proxmox UI, and the failed replication jobs are stuck pending removal. What's the best course of action to completely purge the last remnants?Update #3: https://forum.proxmox.com/threads/vm-is-locked-migrate.9358/ is still relevant,
qm unlock solved my ELK VM quandry
Last edited: