[SOLVED] PBS disk full

[Andres.Urzagasti]

Hello!
I have a small installation with PBS running for almost a year, with 4 disks in ZFS RAIDZ-1.
At one point I forgot to clean up backups and the disk space ran out, making it impossible for me to boot up the computer.
So I had to re-install another PBS in other disks, and then proceeded to re-import the "rpool" pool with the original disks.

PBS setup ->
2x2T sata zfs raid1 (boot, actual pbs system)
4 x 4T sata raidz1 (old /rpool/datastore dataset mounted ok)


Zpool list:
root@pbs1211:/rpool/nfs/dump# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
rpool 21.8T 21.7T 178G - - 68% 99% 1.00x ONLINE -


I managed to re-import the Datastore configuration and it is now accessible through the web interface of the new installation.
However, I am unable to make space or remove old items through the shell. I tried to remove some files that were not part of the datastore, with commands

# cat /dev/null > /file/to/delete
# rm /file/to/delete

root@pbs1211:/rpool/nfs/dump# rm vzdump-qemu-702-2021_07_03-19_00_03.vma.lzo
rm: cannot remove 'vzdump-qemu-702-2021_07_03-19_00_03.vma.lzo': No space left on device

root@pbs1211:/rpool/nfs/dump# cat /dev/null > vzdump-qemu-702-2021_07_03-19_00_03.vma.lzo
-bash: vzdump-qemu-702-2021_07_03-19_00_03.vma.lzo: No space left on device


But I haven't had any success with them either.

Any ideas or help?
Thanks !!

 

[janssensm]

Somehow I would expect that there would be 2 pools. But only if they had different names or you imported with a different name.
Your output lists just 1 pool.
You can import a pool under a new name, other than rpool, from man zpool-import:

zpool import [-Dflmt] [-F [-nTX]] [-c cachefile|-d dir|device] [-o mntopts] [-o property=value]… [-R root] [-s] pool|id [newpool]
Imports a specific pool. A pool can be identified by its name or the numeric identifier. If newpool is specified, the pool is imported using the name
newpool. Otherwise, it is imported with the same name as its exported name.

From your info I would think you somehow have all 6 drives in 1 pool, rpool, you could check with zpool status
If all 6 drives are indeed inside 1 rpool maybe the pool config is now messed up, as the original was raidz1 and the new base install was raid1.

 

[Dunuin]

And for the future I would recommend to set a quota for that pool. Also keep in mind that a ZFS pool always should have 20% of free space left for best performance and to reduce fragmentation (and you can't defrag a ZFS pool so this is important. Only way to "defrag" it is to move everything off the pool and transfer it back so you would need another free 12TB space for that). So I would set a quota for the root of your pool of 80%. If you then accidentally run out of space (you hit the quota) you can just temporarily increase the quota from 80 to 90% and your pool will work again.

So something like this: zfs set quota=9.6T rpool

 

[Andres.Urzagasti]

Hi

Somehow I would expect that there would be 2 pools. But only if they had different names or you imported with a different name.
Your output lists just 1 pool.
You can import a pool under a new name, other than rpool, from man zpool-import:

From your info I would think you somehow have all 6 drives in 1 pool, rpool, you could check with zpool status
If all 6 drives are indeed inside 1 rpool maybe the pool config is now messed up, as the original was raidz1 and the new base install was raid1.

Thanks Janssensm for view and reply my thread.
This is correct. I have only only one pool (rpool) At this time, no need to mount a second pool, only the original rpool with the 4 disks.
The other 2 drives are apart, in a LVM config (old Dell-Raid card). In this LVM are installed actually the PBS with no pool. See attached screenshot.
The original raidz1 are imported OK and working OK, but I cannot delete any file or folder.

I have mounted without problems the original rpool, and actually expose this datastore to the other Proxmox, and can recover any backup. But will be interesting to delete some space...

 

[Andres.Urzagasti]

And for the future I would recommend to set a quota for that pool. Also keep in mind that a ZFS pool always should have 20% of free space left for best performance and to reduce fragmentation (and you can't defrag a ZFS pool so this is important. Only way to "defrag" it is to move everything off the pool and transfer it back so you would need another free 12TB space for that). So I would set a quota for the root of your pool of 80%. If you then accidentally run out of space (you hit the quota) you can just temporarily increase the quota from 80 to 90% and your pool will work again.

So something like this: zfs set quota=9.6T rpool

Thanks Dunuin for your recommendation.
In my case, would the only way be to copy the files to another larger space with some zfs utility? Or maybe add 2 more disks to the rpool and extend it?

The datastore appears to be OK... are mounted in this PBS and list OK the previous backups (see attached screenshot)

 

[janssensm]

Thanks for the info, now it's more clear. My question came also as I wondering how your rpool could be 21.8T with 4x4TB, but they are 6TB in your picture.

I think your current layout is already better then the original setup, I think that it's better to separate the datastore disks from the main install, for multiple reasons.

In the datastore content view you have a high count of backups for each vm.
It's a choice of course to keep that many, but with the pruning schedule you could tune when they are not needed anymore and not keep each daily backup for example. But then you also need to schedule garbage collection, if you didn't already, else you won't get your space back.
Did you already run it manually?
If not keep in mind that if you run it and it says they some content is removed from disk, you need to wait 24h before you really get that space back.

If you really need all these backups, then all you can do is expand I think. I would always prefer mirror vdevs over raidz1.
More detail in the pve docs[1] chapter 3.8.3, so if you would expand, maybe you should first consider if raidz1 is the option you want.
To speed up some processes on the backup server you could also add fast special devices to the pool, see [2].

And you could move backups to another datastore (local or remote) by setting op a remote and then use
proxmox-backup-manager pull <remote> <remote-store> <local-store> [OPTIONS]

[1] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_zfs
[2] https://pbs.proxmox.com/docs/sysadmin.html#zfs-administration
[3] https://forum.proxmox.com/threads/is-it-possible-to-merge-datastore-content.106498/#post-458239

 

[Dunuin]

Btw...once a ZFS pool is full, it will switch into read-only mode. I guess thats why you can't delete stuff. ZFS is a copy-on-write filesystems, which means you need to make sure that it will never happen that a pool gets to 100%. After 80% it will become slow and start to fragment faster, after 90% it will switch into panic mode become even more fragmented and incredible slow. And when the space is completely used up it will become read only.
Not sure if you can extend a pool thats already 100% full. Might be possible that you need to copy the complete pool to a bigger one. But not sure if replication will work. For that you would need to create a snapshot I and think would be problematic too.

 

[Andres.Urzagasti]

Thanks Janssensm and Dunuin for your directions. Probably my english redaction are very basic to explain better (and google translator for some things... does not help really). Please note also that I live in Argentina where, due to financial situation, majority of small enterprises need to reuse all available resources (like old computers, slow hard disks... )

With this "zfs readonly" server my intention is mantain the backup files for future restoration needs.
Now i´m testing if the saved backups can be restored without problems.

As you seen, there are high counts of backups, mainly ocassionated by no-time dedicated to the server, and appears to be that the garbage collection was not set correctly in the previous server.

Next tasks will be test to move (or copy) the backups with pull option.

For clarify the 4x4T, 21.8T <--> 6T.... sorry... the correct info is 4x6T. Attached only the disk info for /dev/sda (sdb, sdc and sdd are simillar)

Thanks again for your assistance. Will look with the pull option to another server in the next days.

 

[Andres.Urzagasti]

Hello everyone,
Update on this post (note that my previous nickname was reconquista)

I managed to sync a replica to another new server. I was able to recover almost all the files but no luck with VMs larger than 800gb.
Anyway, we have recovered part of the files. We will try to improve the supervision of the available space.

Thanks for your assistance.