I had an event. PVE server went down. The zpool was full.
(5 disk raidz2, configured by the PVE installer.)
It's been failing on backups. That leaves behind fleecing files. They filled the drive of this machine.
I think the bump over the line was data being added to a vm's dataset.
Machine stopped responding. Had to recover from rescue iso. (I didn't do that bit, but I'm told the rescue iso is not great.)
We were able to delete the fleecing datasets, and that gave us enough room to work again. So strangely, the fleecing garbage files kinda helped with recovery in that they were the obvious candidate to delete.
It's alive again.
...
Sigh. Drama. Too much excitement.
So ... this whole thing happens to PBS servers too. There's an easy fix.
On PBS, the root is in rpool/ROOT, so you can just do this.
root is in the base of rpool on standard-install PVE, not a specific dataset where I can apply a reservation.
Is there an easy fix for this on a PVE server?
----
Edit ... I'm an idiot. Yes, its right there on PVE as well. Same command.
I'm going to leave this up for comment and ridicule.
(5 disk raidz2, configured by the PVE installer.)
It's been failing on backups. That leaves behind fleecing files. They filled the drive of this machine.
I think the bump over the line was data being added to a vm's dataset.
Machine stopped responding. Had to recover from rescue iso. (I didn't do that bit, but I'm told the rescue iso is not great.)
We were able to delete the fleecing datasets, and that gave us enough room to work again. So strangely, the fleecing garbage files kinda helped with recovery in that they were the obvious candidate to delete.
It's alive again.
...
Sigh. Drama. Too much excitement.
So ... this whole thing happens to PBS servers too. There's an easy fix.
On PBS, the root is in rpool/ROOT, so you can just do this.
root is in the base of rpool on standard-install PVE, not a specific dataset where I can apply a reservation.
Is there an easy fix for this on a PVE server?
----
Edit ... I'm an idiot. Yes, its right there on PVE as well. Same command.
I'm going to leave this up for comment and ridicule.
Last edited: