Thick Provisioning of Disks Takes Forever

[cshill]

Good Morning/Evening,

I researched how I can get rid of thin provisioning by going into the storage.cfg and adding the line ' preallocation full '.

dir: Datastore2
        path /mnt/pve/Datastore2
        content rootdir,backup,iso,snippets,vztmpl,images
        is_mountpoint 1
        nodes proxmoxtest1
        preallocation full

The new disks are in fact allocating the storage space fully, however, it takes a ridiculous amount of time for a new VM to be created. Am I doing something incorrectly when utilizing the directory storage and qcow file format or does preallocation work more effectively under ZFS or BTRFS? What options do I have to make this faster?

Below are the pveversions

proxmox-ve: 8.3.0 (running kernel: 6.8.12-5-pve)
pve-manager: 8.3.2 (running version: 8.3.2/3e76eec21c4a14a7)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.12-5
proxmox-kernel-6.8.12-5-pve-signed: 6.8.12-5
proxmox-kernel-6.8.12-4-pve-signed: 6.8.12-4
proxmox-kernel-6.8.12-3-pve-signed: 6.8.12-3
proxmox-kernel-6.8.12-2-pve-signed: 6.8.12-2
proxmox-kernel-6.8.4-2-pve-signed: 6.8.4-2
ceph-fuse: 18.2.4-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
dnsmasq: 2.89-1
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.4
libpve-access-control: 8.2.0
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.10
libpve-cluster-perl: 8.0.10
libpve-common-perl: 8.2.9
libpve-guest-common-perl: 5.1.6
libpve-http-server-perl: 5.1.2
libpve-network-perl: 0.10.0
libpve-rs-perl: 0.9.1
libpve-storage-perl: 8.3.3
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.5.0-1
proxmox-backup-client: 3.3.2-1
proxmox-backup-file-restore: 3.3.2-2
proxmox-firewall: 0.6.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.3.1
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.3
pve-cluster: 8.0.10
pve-container: 5.2.3
pve-docs: 8.3.1
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.2
pve-firewall: 5.1.0
pve-firmware: 3.14-2
pve-ha-manager: 4.0.6
pve-i18n: 3.3.2
pve-qemu-kvm: 9.0.2-4
pve-xtermjs: 5.3.0-3
qemu-server: 8.3.3
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.6-pve1

 

[Kingneutron]

Why do you not want thin provisioning, and what are you using for backing storage? Spinning HD is going to take a while

 

[cshill]

Why do you not want thin provisioning, and what are you using for backing storage? Spinning HD is going to take a while

Thin provisioning has the potential of heavy fragmentation of data and my boss does not want thin provisioning. I am using SATA SSD's for the OS and then I put additional storage onto the HDD for each VM.

First Test: I was on an older computer and using a single HDD to generate 1TB of disk space for the OS of a Windows 11 machine. That took about 2.5 hrs for the 1TB to fully allocate disk space prior to me being able to even start loading Windows 11.

Second Test: I started up a Windows 11 VM on a PowerEdge R540 with 6 cores, 32 GB of ram, 850 GB of SSD storage. That took 35 minutes to complete the 850 GB allocation prior to loading the windows OS. I also need to add 4TB worth of HDD disk space to these VMs for employees. I couldn't add the 2nd disk until the main disk was complete. Now the 2nd one is sitting at 24 minutes so far and am expecting it to take a couple hours if only 1TB on the older system took 2.5 hrs.

This is is the concern that with the pre allocation option is that it's insanely slow.

 

[Kingneutron]

The only thing I can recommend is keep an eye on ' top ' or ' htop ' and renice -15 the preallocation process. That and get the fastest + high TBW rating SSDs that you can afford.

Most people I know of just stick with thin provisioning, and defrag / restore if it gets too slow

Can you go with templates or bash scripting to create your VMs?

https://github.com/kneutron/ansitest/blob/master/proxmox/newvm-proxmox.sh

 

[cshill]

The only thing I can recommend is keep an eye on ' top ' or ' htop ' and renice -15 the preallocation process. That and get the fastest + high TBW rating SSDs that you can afford.

Most people I know of just stick with thin provisioning, and defrag / restore if it gets too slow

Can you go with templates or bash scripting to create your VMs?

https://github.com/kneutron/ansitest/blob/master/proxmox/newvm-proxmox.sh

I tested the template out and can use that but will need to resize every disk to match the specs desired. The thin provision I just worry will slow down too much and these VMs will be heavily tested on. I fear they will slow down very quickly. What I don't get though is why it's so slow on proxmox for thick provisioning when I use the pre allocation method for directory storage. I tested on ZFS and people say it's thin provisioning but I saw it take away 1TB, same test.

 

[waltar]

The only thing I can recommend is keep an eye on ' top ' or ' htop ' and renice -15 the preallocation process. That and get the fastest + high TBW rating SSDs that you can afford.

Pre-allocation for a ssd makes no sense as a ssd works inside like a cow filesystem and permanently remap the cells to addresses while rewriting unfull cell clusters. This is as fragmentation but as all cells have same short access time it's irrelevant.
Thick provisioning even makes no sense on cow filesystems (like zfs and btrfs) as every block change is written elsewhere and so fragmentation is the normal operation mode.
I think the theory of thick allocation is nice but has no real effect on file-storage with ext4 and xfs as changed blocks are rewritten inside.
Otherwise thick allocation may have effect on block storage but as shown in your storage.cfg you use "dir" which is file- and not block storage.

 

[cshill]

Pre-allocation for a ssd makes no sense as a ssd works inside like a cow filesystem and permanently remap the cells to addresses while rewriting unfull cell clusters. This is as fragmentation but as all cells have same short access time it's irrelevant.
Thick provisioning even makes no sense on cow filesystems (like zfs and btrfs) as every block change is written elsewhere and so fragmentation is the normal operation mode.
I think the theory of thick allocation is nice but has no real effect on file-storage with ext4 and xfs as changed blocks are rewritten inside.
Otherwise thick allocation may have effect on block storage but as shown in your storage.cfg you use "dir" which is file- and not block storage.

I appreciate this response as it's very insightful and by default I keep the system in thin provisioning, however, my boss specifically asks for everything to be thick. This is probably due to the amount of requests that come in for more VMs and we'll have to have a separate list going to properly understand what is available for storage as we may have already used up the 20TB or so of storage but we only see 2TB in use. The concern is that we have programmers that just max out disk space one week and need more then other times they use none. The disk space is volatile for the VMs and by having the system set to thick it can tell multiple different techs that there's no space to take and they have to tell the programmers too bad, order another server.

 

[waltar]

On an external block storage appliance (the old EMC days) it could make sense to use thick volumes for predicted performance but even there from the side of the operating they would like to use thin provisioning for costs. And using thick provisioning isn't bad as new allocations didn't be done on the fly at all but with pve you have to deal with snapshot capabilities at your own then. Each has it's pros and cons and works, sometimes easier othertimes with some additional (shell) tricks.