Yesterday afternoon I noticed that my Proxmox server was acting up in some strange ways. After a bit of digging I found that the Local drive was at 100% full. Multiple reboots later and there's been no change. There are no files stored in that drive and as far as I know, nothing but the PVE itself should even be using it. The issue seems to have occurred yesterday at 3:00 AM. When checking the drive's storage history it used 4.3GB at 3:00 and immediately after that went all the way up to 100GB/100GB. The only thing I can think of that could've caused this is that I have a backup job that runs at 03:00 on Sundays. What's weird about that is that my backup job goes to an external USB drive, so the local /dev/mapper/pve-root shouldn't have been touched. Unfortunately I can't even use the shell to get in as it gives me an error "Connection failed (Error 500: closing file '/var/tmp/pve-reserved-ports.tmp.4026430' failed - No space left on device)". Any suggestions?
[SOLVED] PVE Root Full
- Thread starter SpiderManAPV
- Start date
I would guess that that your external drive wasn't mounted and so everything written to that mountpoint was stored on root instead of your external drive.
If you got physical access to the server use your keyboard and navigate to the mountpoint of your external drive. If its not mounted delete some of the backups. If it is mounted meanwhile, unmount the external drive and look if there is still something in that directory.
If you got physical access to the server use your keyboard and navigate to the mountpoint of your external drive. If its not mounted delete some of the backups. If it is mounted meanwhile, unmount the external drive and look if there is still something in that directory.
That appears to be the solution. I couldn't actually get it working in Proxmox proper, but by livebooting Ubuntu and navigating the drive I found that the mount point had a directory inside with a partially completed backup. I deleted that file and everything is back to normal. Thanks for your assistance!
Someone knows if there is a way to ensure that a mountpoint is really mounted before a backup is allowed to start? I thought about some kind of quota or owner without write permissions for the empty folder itself that is used as a mountpoint but I think that isn't working if backups are run as the root user. Maybe something with hook-scripts?