Loosing space on PVE VM Store

[muekno]

My ZFS volume has 965 GGB, summery says 831 GB Use that makes about 134 GB free, Adding the space of the vMs shown under VM disks makes 818 GB, that looks OK.
Now I want to make a snapshot of a 600 GB VM, but it stops after 3,6 GB with disk full. Last week it was still possible to make snapshot and as I remember space and used conditions have been quit the same.

felling using space sind some days, in the beginning I hat 4 other VMs (about 80 GB large )on the some volume and had always (since about 5 months) about 350 GB free) . after a sudden lost of space over night I move the 4 VMs of about 80 GB to an other store.

My VMs are quite stable in using space, none of them uses relay more space over time, most are static (i.e. a reverse proxy server, configuration backup server, mail relay etc. the only really growing server is the 600 GB VM, but not as much as the space is lost.
The calculated Used space ist the brutto capacity of the VMs, but the netto capacity is less. So i-e. the 600 GB VM has about 160 GB free space.

So whats going on since about 3 weeks with my disk space, before everything was ok for months

Regards

Rainer

 

[itNGO]

"zpool status" output please and some fact about your layout may help....

 

[muekno]

zpool status
  pool: VM-Store
 state: ONLINE
  scan: scrub repaired 0B in 00:07:54 with 0 errors on Sun Sep 29 16:42:44 2024
config:

        NAME                               STATE     READ WRITE CKSUM
        VM-Store                           ONLINE       0     0     0
          nvme-CT1000P3PSSD8_241247B1DB16  ONLINE       0     0     0

errors: No known data errors

  pool: VM-USB-Store
 state: ONLINE
config:

        NAME                                STATE     READ WRITE CKSUM
        VM-USB-Store                        ONLINE       0     0     0
          ata-TOSHIBA_MQ01ABD100_X3J7P1BKT  ONLINE       0     0     0

errors: No known data errors
root@pve-rh:~#

The PVE host is an ASUS NUC with a a 128 GB SSD where thy system is on, with LVM and a 1TB NVE SSD with ZFS. I addded an external 1 TB USB drive as additional storage with ZFS too.

What other information do need.

Regards
Rainer

P.S. The system was running on a HP DL 380 since March, on a 2x164 HD 2x 1TB HD without any space problems.
The drives have been configured as RAID 1 for the 164 drives and separate for the 1TB drives in the Smart Array controller. The 1TB drives had been mirrored via ZFS. In case of high power costs I move to the configuration now end August to begin of September by backup and restore via the an PBS. The PBS was on a similar Fujitsu Server which is move to a less power consuming hardware too. I had now space problem till near end of september.

 

[leesteken]

Did you enable Discard and setup trim inside your VM: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_hard_disk_discard ? Did you make sure there are no snapshots left in your VMs?

PS: I don't see any RAID1 in your zpool status output. I do see drives that will perform poorly with ZFS because of QLC and USB/laptop-class.

 

[LnxBil]

Did you enable Discard and setup trim inside your VM: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_hard_disk_discard ?

and keep in mind that (un)checking this setting does not alter already created disks. You need to look for the attribute refreservation in each zfs dataset.

 

[muekno]

Did you enable Discard and setup trim inside your VM: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_hard_disk_discard ? Did you make sure there are no snapshots left in your VMs?

PS: I don't see any RAID1 in your zpool status output. I do see drives that will perform poorly with ZFS because of QLC and USB/laptop-class.

Thank you for quick answer, I think I did not enable Discard if it is not default. Most VMs until 1 came from the PBS backuped from the HP Server and restored to NUC.

Yes there is no mirroring any more. I know that is not the best choice, but I couldn't find actuell hardware that is good for my budget and in lower power consuming.
The main Disk for the VMs is the 1 TB NVME SSD, the USB disk is just for an archive VM the content is static and will not change but may grow from new things to archive, so performance is absolutely not the point.

On the other site I have a seperte PBS with 2 enterprise grade 1TB SSDs, one attached via SATA connector and one via USB 3.

Yes I know thats not very good, all my servers before since 40 years had RAID 1, but now I had to decide between a lot of money for the enterprise ready servers and less redundance. But I think with new hardware and a daily backup to my PBS I am on a good site. I do not think that both the PVE and the PBS USV backed up will die the same time. So if the PBS fails I can repair or build a new one and I think I will not need a perhaps lost backup in less then one day. I the PVE fails I can repair or build a new one and restore all VMs, a downtime up to 2 days will be acceptable for me.
After giving up my company when I retired, I am still working as a one man show IT consultant so the actuell situation is OK for me.

I will check for that "Disable" thing.

And no there are no snapshots leaving.

Regards
Rainer

 

[muekno]

My guest are all Linux Kernel 6+, HD controller are most vitrIO, SCSI, three are virtIO SCSI single, and one is VMware PVSCSI, but where can I see the state of discard and trim and where can I change that without risk, or what can I to change it.
The guest filesystem is ext4

Regards
Rainer

 

[muekno]

I just saw my ZFS storage is not thin provisioned, that declares a lot. I must have overseen when moveing from the HP server to NUC. Means did forgot to check when createing the VM stores,
What will be if I check that now. will it work for existing VMs.
Or what will be if I backup delete and restore them to and from my PBS

Regards
Rainer

and what about that post

LnxBil
Distinguished Member
Feb 29, 2024
Add bookmark
#10
JensF said:
To get your disk space back you have to move your data to another storage...
No, just unset the refreservation:

Code:
zfs get refreservation -r rpool

for all entries with a setting, unset it:

Code:
zfs set refreservation=none <dataset>
Like Reactions:JensF
 Like ReplyReport

can I use "zfs get refreservation -r rpool" whithout risk

 

[carles89]

will it work for existing VMs

Unfortunately you will need to backup and recover your VMs to be able to benefit from thin provisioning. Another option is to move disks to a non-zfs storage and then move them back to your ZFS pool.

 

[LnxBil]

Unfortunately you will need to backup and recover your VMs to be able to benefit from thin provisioning. Another option is to move disks to a non-zfs storage and then move them back to your ZFS pool.

No, this is not true. Just reset your refreservation.

can I use "zfs get refreservation -r rpool" whithout risk

Yes, all get commands are normally noninvasive.

 

[muekno]

Thank your for answering. Sorry for the delay to respond but I had a remote support at a customer.
But I do not yet understand what the command really does, it does not set the thin provisioning checkbox on the pool.
What happens if I set it editing the pool.
For testing I did the command on the not so critical boot of the archive machine as I have these file on my laptop and on PBS backup
Look in at the summery of that pool where noting as the 600 GB VM ist says 743 GB used, while the VM use only 374GB of the 600GB

Thank you
Regards
Rainer

 

[carles89]

No, this is not true. Just reset your refreservation.

Thanks for pointing that out, I wasn't aware of this option and I thought it was the same as with volblocksize.

 

[LnxBil]

Thanks for pointing that out, I wasn't aware of this option and I thought it was the same as with volblocksize.

Yes, it's not so simple. Some are just changeable, others are changeable yet only apply to new data (compression) and some are fixes as volblocksize and volsize.

 

[muekno]

so I have to check the thin option and then move VM by VM to another storage and remove to the original storage hopeing that the VM will now thin