ZFS pool full, io-error, cannot delete VM, nothing works

[ggroke]

Hello,

I have a Proxmox 7.4, with 3 VMs, one of them being an off-site backup.

Due to a fail in my backup engine, it dumped several Tb on one VM, therefore filling all space available. All VMs crashed.

The GUI no longer works, only shell. I tried to kill that VM in order to free some space, but it doesn't work.

root@pve3:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 1.28T 0B 151K /rpool
rpool/ROOT 2.91G 0B 140K /rpool/ROOT
rpool/ROOT/pve-1 2.91G 0B 2.91G /
rpool/data 1.27T 0B 140K /rpool/data
rpool/data/vm-100-disk-0 81.4K 0B 81.4K -
rpool/data/vm-100-disk-1 5.27G 0B 5.27G -
rpool/data/vm-101-disk-0 2.71G 0B 2.71G -
rpool/data/vm-101-disk-1 607G 0B 607G -
rpool/data/vm-102-disk-0 3.69G 0B 3.69G -
rpool/data/vm-102-disk-1 685G 0B 685G -

root@pve3:~# qm delete 102
ipcc_send_rec[1] failed: Connection refused
ipcc_send_rec[2] failed: Connection refused
ipcc_send_rec[3] failed: Connection refused
Unable to load access control list: Connection refused

What can I do? The 101 and 102 VMs are not important, I can delete them no problem, but I need to keep the VM 100.

Best regards

 

[leesteken]

Your Proxmox is probably not working because some services could not be started, due to lack of disk space. Try using the ZFS command line tools to delete the virtual disk(s) of VM 101 or 102. For example: zfs destroy rpool/data/vm-102-disk-0. Do make sure that I did not make a typo or used the wrong VM or disk in this example! I expect that (maybe after a reboot) you can manage your VMs from the Proxmox GUI again. You might have to edit /etc/pve/qemu-server/102.conf manually to remove the virtual disk you deleted.

 

[ggroke]

Yes, that worked. after deleting that disk, the system came up in working condition.

But still, I’m intrigued, I had allocated enough space to all VMs and left some spare, how come it used all enough space?

 

[leesteken]

But still, I’m intrigued, I had allocated enough space to all VMs and left some spare, how come it used all enough space?

Virtual disk-1 or both VM 101 and 102 used almost half of the entire space each. What do their configuration files look like? What was the space usage inside the VMs? Did you make sure trim/discard was working all the way?

 

[LnxBil]

In addition to @leesteken questions: What about snapshots?

You should set refreservation to e.g. 1 G for you root dataset to avoid this problem in the future. I created a bug report for that some years ago.