installation or boot drive full even with only 1 VM

[epicurean]

Newbie alert.
I have proxmox 7 installed in a 200GB Intel SSD, and pair of 900 Samsung SSDs as ZFS mirror. I have only 1 windows 10 VM installed but the 200GB drive is 100 full. I have no idea why.
I do have another 2 proxmox 7 machines with 3-4 VMs in the same cluster but they are similarly configured (as far as I can remember) with the VM datastores in the ZFS mirrors created.
Any help appreciated on how to fix this

 

[Dunuin]

Whats your df -h, vgs and lvs output?

If your system disk gets full its most of the time because of writing to a unmounted mountpoint.

 

[epicurean]

the df -h output is:
root@fd2:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 16G 0 16G 0% /dev
tmpfs 3.2G 298M 2.9G 10% /run
/dev/mapper/pve-root 47G 47G 20K 100% /
tmpfs 16G 57M 16G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
lenovo900 758G 128K 758G 1% /lenovo900
/dev/fuse 128M 36K 128M 1% /etc/pve
//192.168.1.64/aei_storage 11T 1.5T 8.7T 15% /mnt/pve/aei_storage
192.168.1.172:/Seagate/AEI_Backup 17T 1.8T 16T 11% /mnt/pve/aei_backup
tmpfs 3.2G 0 3.2G 0% /run/user/0

vgs is:
VG #PV #LV #SN Attr VSize VFree
pve 1 3 0 wz--n- <185.81g 16.00g

lvs is:
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
data pve twi-a-tz-- <113.25g 0.00 1.40
root pve -wi-ao---- 46.25g
swap pve -wi-ao---- 8.00g

 

[Dunuin]

Looks like most of your 200GB SSD is used by some VMs/LXCs. If you say you only got one VM on that node I guess you put that VM on your system disks LVMthin and not on your ZFS pool.:

 

[epicurean]

I am almost certain I put the lone Win 10 VM on the lenovo900 mirror.
how do I keep my VM , free up the space and ensure the vm datastore is in the lenovo zfs mirror?
I am curious what setttings I did wrong to let this happen, so i can avoid it in future

 

[Dunuin]

Maybe I'm reading the output wrong because you didn't posted it in code tags to preserve indentation.
Your pve-root also looks full. You could run something like du -a --exclude=/lenovo900 --exclude=/mnt/pve/aei_storage --exclude=/mnt/pve/aei_backup / | sort -n -r | head -n 20 to find the biggest folders/files.

 

[epicurean]

I get this error msg with the command you posted:

du: cannot access '/var/lib/lxcfs/cgroup': Input/output error
sort: cannot create temporary file in '/tmp': No space left on device

 

[Dunuin]

You should try to delete some unimportant data (like compressed logs in /var/log) and try it again.

 

[epicurean]

NOw I get this error, I did delete some of the compressed logs

du: cannot access '/var/lib/lxcfs/cgroup': Input/output error

 

[epicurean]

I gave up , and did a fresh proxmox 7 install. and restored from one of the backups. During the restore, it would only allow me to restore the VM into either the LVMthin (which I did) or the NFS storage (eg aei_backup). How do I move the VM to the ZFS mirror I recreated (eg lenovo900)?

much thanks again.
Still dunno why the VM filled my boot disk to 100%