Web UI broken, SSH still works – rpool/ROOT/pve-1 full while rpool still shows free space

[Pascal.ch]

Hi everyone,

I am looking for some help to better understand and fix a storage issue on my Proxmox VE host.

My setup is roughly the following:

• 1 Proxmox VE host
• ZFS-based installation
• main pool: rpool
• system root dataset: rpool/ROOT/pve-1
• another storage/pool: vmdata
• some VM disks are stored on rpool/data

For example, I found:

• rpool/data/vm-100-disk-0 around 23.4G
• rpool/data/vm-200-disk-0 around 34.9G

From zpool list, I currently see something like:

• rpool total around 63.5G, allocated 61.5G, free 1.96G
• vmdata still has around 60.2G free

My problem is that the node is still reachable over SSH, but the Proxmox Web UI is no longer working properly.

What I observed:

• pveproxy, pvedaemon, and pve-cluster are active
• port 8006 is listening
• but the Web UI is basically unusable

From df -h, / is mounted on rpool/ROOT/pve-1, and that dataset shows as 100% full with 0 available.

In the logs I also saw errors like:

• commit transaction failed: database or disk is full
• rollback transaction failed: cannot rollback - no transaction is active

What I do not really understand is this:

• rpool/ROOT/pve-1 seems full at around 3.3G
• but rpool still shows around 1.96G free

So I am trying to understand:

• why this 3.3G root dataset filled up like that
• why it seems to be the thing blocking the system
• and why it shows 0 available while the pool still has a bit of free space left

Proxmox support suggested that I could try moving the disk of VM 100 from local-zfs to vmdata in order to free space, and also remove ISO files from local storage if there are any.

My issue is that I am not fully sure how to do that safely, and I do not want to break my Windows VMs or make the situation worse.

So the reason for this post is:

• to better understand what is happening technically
• to understand why rpool/ROOT/pve-1 is full
• and to get advice on the safest way to recover from this

This post was written with the help of ChatGPT to make the issue easier to describe clearly.

Thanks a lot in advance for any help or guidance.
Pascal

 

[wbk]

Hi Pascal.ch,

I'm not fluent in ZFS, so other members will have to chime in for support on that.

How fluent are you in Linux?

What you could do for a small improvement, is check the amount taken by /var/log and /var/cache:

# du -hs /var/log/*
# du -hs /var/cache/*

If there's a lot of data in cache, it may be that apt clean can make a difference (that deletes cached downloaded .debs)

Logs can be safely removed, although of course they come in handy as well. In a similar situation, I recently got rid of a few GB of log by vacuuming the journal:

# journalctl --vacuum-time=4d # keep the last 4 days

On ext4 if there's a great disparity between the expected size and the reported size, it could mean the situation calls for fsck.ext4, but ZFS is supposed to take care of that by itself so that's a worry less.

Once you got some free space but the web UI does not recover, you could try restarting PVE: pveproxy restart

This post was written with the help of ChatGPT to make the issue easier to describe clearly.

Such a pity ChatGPT did not advice to paste the output of df -h , lsblk and zpool list (in "CODE" block, use the </> button in the editor) to give insight in the situation. Please do!