ZFS pool full, io-error, cannot delete VM, nothing works

ggroke

New Member
Jul 1, 2022
19
9
3
Hello,

I have a Proxmox 7.4, with 3 VMs, one of them being an off-site backup.

Due to a fail in my backup engine, it dumped several Tb on one VM, therefore filling all space available. All VMs crashed.

The GUI no longer works, only shell. I tried to kill that VM in order to free some space, but it doesn't work.

root@pve3:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 1.28T 0B 151K /rpool
rpool/ROOT 2.91G 0B 140K /rpool/ROOT
rpool/ROOT/pve-1 2.91G 0B 2.91G /
rpool/data 1.27T 0B 140K /rpool/data
rpool/data/vm-100-disk-0 81.4K 0B 81.4K -
rpool/data/vm-100-disk-1 5.27G 0B 5.27G -
rpool/data/vm-101-disk-0 2.71G 0B 2.71G -
rpool/data/vm-101-disk-1 607G 0B 607G -
rpool/data/vm-102-disk-0 3.69G 0B 3.69G -
rpool/data/vm-102-disk-1 685G 0B 685G -

root@pve3:~# qm delete 102
ipcc_send_rec[1] failed: Connection refused
ipcc_send_rec[2] failed: Connection refused
ipcc_send_rec[3] failed: Connection refused
Unable to load access control list: Connection refused

What can I do? The 101 and 102 VMs are not important, I can delete them no problem, but I need to keep the VM 100.

Best regards
 
Your Proxmox is probably not working because some services could not be started, due to lack of disk space. Try using the ZFS command line tools to delete the virtual disk(s) of VM 101 or 102. For example: zfs destroy rpool/data/vm-102-disk-0. Do make sure that I did not make a typo or used the wrong VM or disk in this example! I expect that (maybe after a reboot) you can manage your VMs from the Proxmox GUI again. You might have to edit /etc/pve/qemu-server/102.conf manually to remove the virtual disk you deleted.
 
  • Like
Reactions: LnxBil
Yes, that worked. after deleting that disk, the system came up in working condition.

But still, I’m intrigued, I had allocated enough space to all VMs and left some spare, how come it used all enough space?
 
But still, I’m intrigued, I had allocated enough space to all VMs and left some spare, how come it used all enough space?
Virtual disk-1 or both VM 101 and 102 used almost half of the entire space each. What do their configuration files look like? What was the space usage inside the VMs? Did you make sure trim/discard was working all the way?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!