Disk space problems (io-error)

[morenz]

Hi everyone,
I have a 3-VM setup in a PVE 3.3.2 environment.

Host disk size is about 885 GB. We have split it in three, one per VM, with overbooking (600GB + 300GB + 100GB).
All partitions are raw format.
ATM disk usage is:
600 GB: 85%
300GB: 25%
100GB: 5%

All went well since last week I made a dump of a database that was about 50 GB in size from 600 GB machine to 300 GB one, and it suddenly went down, and 2 of 3 machines had io-error status.

I noticed that host space was 100% used.
I managed to free up about 150 GB of data and restarted, all up and running. For only 10 days.
Today I have the DB machine (600 GB) that has an io-error status, and if I put it online it goes well for about 10-15 minutes, then falls back in io-error status.
If I take a look at the VM's syslog, it reports errors writing on I/O, but I think it's related to the disk being seen as full...

Today I have freed up another 30 GB from the 600 GB, but it didn't help.
Host disk space is still at 100%, while it should be less than 80%

The machine that has to be "freed" is a Windows 2008 Server VM (300 GB)

How can I do?
Thanks
Moreno.-

 

[dcsapak]

can you post the output of

df -h

on the host?

 

[morenz]

Here it is...
that confirms my suspects... freed up disk space in VMs seems not to be released in the host....

root@ns337433:/var/log# df -h
File system  Dim. Usati Dispon. Uso% Montato su
udev  10M  0  10M  0% /dev
tmpfs  6,3G  340K  6,3G  1% /run
/dev/sda1  20G  1,3G  18G  7% /
tmpfs  5,0M  0  5,0M  0% /run/lock
tmpfs  13G  28M  13G  1% /run/shm
/dev/mapper/pve-data  869G  869G  0 100% /var/lib/vz
/dev/fuse  30M  16K  30M  1% /etc/pve

Thanks
Moreno

 

[wbumiller]

In order for guest space to be freed on the host every storage layer needs to support (and use) the discard operation. This means the guest OS has to issue a discard/trim and the file system, volume manager etc. need to propagate it to the disk, which needs to be a type which supports discards (and has to be enabled, too, via the disk's settings in the pve gui). Note that not all disk types support discard operations (eg. virtio-block does not, while virtio-scsi does).