Critical bug - create vm disk on filesystem

[czechsys]

Hi,

i did this:

1] vg pve
2] lv local-lvm-data (lvm-thick), size 100GB, ext4, mounted AS directory -> VM images are files. Used 50%.
3] in webgui, created new disk for one VM with 100GB size, RAW
==BUG
there is 50% free of the 100GB, but it allowed to create 100GB without any warning. I know, this image is not 100GB yet.
==BUG
4] in VM started copy data to this 100GB disk ===> after 50GB copied VM crashed. PVE webgui started to misbehavior, etc etc...

So, main question. Feature or Bug, that you can create VM image bigger than free space on the filesystem even when all images are RAW? What it will do if we overrun NFS storage with same process?

No warning about overriding space? No warning, that VM configured size > free filesystem space? No warning, that all VM configured size > free space?

proxmox-ve: 5.3-1 (running kernel: 4.15.18-9-pve)
pve-manager: 5.3-6 (running version: 5.3-6/37b3c8df)
pve-kernel-4.15: 5.2-12
pve-kernel-4.15.18-9-pve: 4.15.18-30
pve-kernel-4.13.8-3-pve: 4.13.8-30
pve-kernel-4.13.4-1-pve: 4.13.4-26
pve-kernel-4.10.17-3-pve: 4.10.17-23
pve-kernel-4.10.17-2-pve: 4.10.17-20
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-43
libpve-guest-common-perl: 2.0-18
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-34
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.0.2+pve1-5
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
openvswitch-switch: 2.7.0-3
proxmox-widget-toolkit: 1.0-22
pve-cluster: 5.0-31
pve-container: 2.0-31
pve-docs: 5.3-1
pve-edk2-firmware: 1.20181023-1
pve-firewall: 3.0-16
pve-firmware: 2.0-6
pve-ha-manager: 2.0-5
pve-i18n: 1.0-9
pve-libspice-server1: 0.14.1-1
pve-qemu-kvm: 2.12.1-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-43
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.12-pve1~bpo1

 

[dcsapak]

this is normal and simply the nature of thin provisioned storage systems (such as directory with sparse files)
if you are overprovisioning your storage, you should have a monitoring system in place to prevent it from getting full

 

[czechsys]

But i am using THICK method. There is no way to create RAW image with max filesize from webgui. And even discard/fstrim etc doesn't resize file to smaller size (for example, on nfs/ext4 etc).
And there is NO monitoring/checks from PVE, how much space is allocated in config vs real size of filesystem.

 

[dcsapak]

please look at the difference in output of 'ls -l' and 'du'

only the metadata is the full blown space, but the file is really not fully allocated

 

[czechsys]

That's not the problem. Problem is, there isn't any warning (or ability to block) when overprovisioning disk space in config occurs. None user will check monitoring for free space (he will look for free space in gui) and specially, don't expect everybody will write script to check config to calculate configured/available storage size (and result will be still missing in the webgui).

When allocating against LVM directly, its blocked by LVM (or PVE too?). But allocating against filesystem need to be checked, thats huge dataloss/damage hole. I am speaking especialy for THICK storages/images.

 

[dcsapak]

no that is normal thin provisioning storage behavior, it is the same with zfs/thinlvm/etc.

if you are overprovisioning (or let users overprovision) you have to monitor the resources so that you can act before they run out

 

[czechsys]

But i DON'T WANT to overprovision. I want be warned/stopped before overprovision in configuration.

So explain me (use case the first post):
1] created 100GB disk image on ~50GB free storage (100GB size)
2] copied data on this disk image (needed to copy 75 GB)
3] transfer rate ~300MBps => 2.7 minutes, before 50 GB used, 66sec in 80% trigger warning, 33sec in 90% trigger warning

Tell me something about human acting before disk space run out in such time.

 

[sb-jw]

Tell me something about human acting before disk space run out in such time.

Normally you have much more space then only 100GB. If you really need to check this you have a monitoring system, which will warn you. New monitoring systems also have a predicted alarm level.

All these features gave you the ability to act before you running out of space.
No one want an storage system which is not capable of thin provisioning. It's save money for you.

If you do not want to have an thin provisioning storage, then don't use it. Use an normal LVM Pool which not supporting that.

 

[czechsys]

Normally you have much more space then only 100GB. If you really need to check this you have a monitoring system, which will warn you. New monitoring systems also have a predicted alarm level.

All these features gave you the ability to act before you running out of space.
No one want an storage system which is not capable of thin provisioning. It's save money for you.

If you do not want to have an thin provisioning storage, then don't use it. Use an normal LVM Pool which not supporting that.

Information, what you have configured is more important, than you think. Tell me, how much vCPU you have configured on node X. You need see this information. Tell me, how much vRAM you configured on node Y. No info again. Tell me, how much disk space you configured in VMs for specific datastore for whole cluster - you know, you can have multiple datastores and not every will be iscsi/local LVM pool? Try samba, nfs, cephfs, etc etc etc. Try recovery of the VM on some datastore. Monitoring is monitoring, not info about whats configured.