[SOLVED] How can I increase local storage? Fills up quickly then all VMs get "io-error"

[diericx]

I currently have a setup with 3 VMs and 1 container. In total, they take up 30+250+1000+100=1380gb out of the total 1.8tb available on my local-zfs. For some reason, my local (host) only has 2gb (1.78gb usable) assigned to it! It uses about 1.78gb on boot and I'm running into an issue where some process is creating files and filling it up within seconds. Upon reboot, it has space but then fills up equally fast.

Is there any way to increase the space on local (storage)? Or any advice on debugging this?



This is the output of `df -h` on the host

root@host:~# df -h
Filesystem                    Size  Used Avail Use% Mounted on
udev                          7.8G     0  7.8G   0% /dev
tmpfs                         1.6G  9.1M  1.6G   1% /run
rpool/ROOT/pve-1              1.8G  1.8G  1.8M 100% /
tmpfs                         7.9G   37M  7.8G   1% /dev/shm
tmpfs                         5.0M     0  5.0M   0% /run/lock
tmpfs                         7.9G     0  7.9G   0% /sys/fs/cgroup
rpool                         1.9M  128K  1.8M   7% /rpool
rpool/data                    1.9M  128K  1.8M   7% /rpool/data
rpool/ROOT                    1.9M  128K  1.8M   7% /rpool/ROOT
rpool/data/subvol-102-disk-0  834M  832M  1.8M 100% /rpool/data/subvol-102-disk-0
/dev/fuse                      30M   20K   30M   1% /etc/pve
tmpfs                         1.6G     0  1.6G   0% /run/user/0

And here is the output of lsblk

root@host:~# lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT,LABEL
NAME        FSTYPE      SIZE MOUNTPOINT LABEL
sda                     9.1T
└─sda1      ext4        9.1T
zd0                    16.5G
zd16                   16.5G
zd32                    1.5G
zd48                   1000G
├─zd48p1                  1M
└─zd48p2    ext4       1000G
zd64                    100G
├─zd64p1                  1M
└─zd64p2    ext4        100G
zd80                    250G
├─zd80p1                  1M
└─zd80p2    ext4        250G
zd96                    8.5G
zd112                  16.5G
nvme0n1     zfs_member  1.9T            rpool
├─nvme0n1p1 zfs_member 1007K            rpool
├─nvme0n1p2 vfat        512M            rpool
└─nvme0n1p3 zfs_member  1.9T            rpool

I'm assuming zd48, zd64 and zd80 are my VMs and that the container says it's root disk is local-zfs:subvol-102-disk-0 meaning it shouldn't be taking any space on local (storage)...

so where is the local (storage) partition if I want to increase it?
Is it zd32? Why only 1.5G? What are all of the 16.5G partitions??
What could be sucking up all the space so quickly on boot?

Thanks in advance for any help or advice

 

[diericx]

Bump, I haven't been able to use my server for days I don't know what to do

 

[Stoiko Ivanov]

seems you have a root on ZFS installation:
* check the output of `zfs list`

 

[diericx]

Thank you for helping!!

root@host:~# zfs list
NAME                                        USED  AVAIL     REFER  MOUNTPOINT
rpool                                      1.80T   841M      104K  /rpool
rpool/ROOT                                  980M   841M       96K  /rpool/ROOT
rpool/ROOT/pve-1                            979M   841M      979M  /
rpool/data                                 1.80T   841M      104K  /rpool/data
rpool/data/subvol-102-disk-0                861M   841M      832M  /rpool/data/subvol-102-disk-0
rpool/data/vm-100-disk-0                    269G   841M      198G  -
rpool/data/vm-100-state-AddedOmbi          7.23G   841M     7.23G  -
rpool/data/vm-100-state-PRSJ               1.08G   841M     1.08G  -
rpool/data/vm-100-state-SoloPlex           1.76G   841M     1.76G  -
rpool/data/vm-101-disk-0                   1.52T   841M      978G  -
rpool/data/vm-101-state-abouttoinstallegg  3.57G   841M     3.57G  -
rpool/data/vm-102-state-vanilla             362M   841M      362M  -
rpool/data/vm-103-disk-0                   5.85G   841M     5.85G  -

 

[diericx]

A coworker of mine showed me how to get the VMS to boot. I removed the CD rom from each one's hardware, they ran for about 30 seconds then got into the same IO-Error.

Now the local (host) storage total size SHRUNK... for no apparent reason. What is going on??

 

[dcsapak]

i guess you have snapshots and your storage is simply full:

rpool 1.80T 841M 104K /rpool

there are only 841M left on your pool

 

[diericx]

@dcspak I did notice that, but spent more time trying to figure out why that had happened. Your tip about zfs list helped me get there.

I'm not familiar with ZFS, so after looking at the output of zfs list and watching my local storage shrink, my best guess was that the volume associated with local storage wasn't allocating space (or "reserving space"). I also spent time trying to figure out what all those other volumes were in the zfs list(-100-state-AddedOmbi, etc.).

I realized those were backups, so I removed some backups I didn't need and was able to get some free space. I then read this article which helped me give the root volume a quota and reservation!

After removing some of the backups I realized what had initially kicked off this issue. I had accidentally created a snapshot of one of my VMs that had a bunch of media (movies and shows) in it, so the snapshot was huge... But I believe the underlying issue was that the `rpool/ROOT` volume didn't have any reservation, meaning it would shrink to allow other volumes to grow and thus suffocate.

By running this command I believe I have prevented future suffocation of my root volume:
zfs set reservation=5GB rpool/ROOT

 

[vikozo]

maybe you have old pve-kernel it /root
i had this problem too.