ubuntu 20 VM - SSD emulation is recognised as HDD

D

dkvrhst

New Member
Apr 7, 2023
12
0
1
I was wondering why my RAW disks were overflowing and noticed that TRIM does not work with my ubuntu 20 VMs. The cause seems to be that an HDD is recognised instead of an SSD. Proxmox has discard and SSD emulation via VirtIO SCSI enabled, yet lshw -class disk within the VM says it is a 5400 rpm HDD "capabilities: 5400rpm gpt-1.00 partitioned partitioned:gpt". Attempts to re-detect do not lead to success, does anyone have another hint? ubuntu kernel is 5.4 and latest Proxmox.
 
Last edited:
But not with ubuntu 20: After each VM restart, 3 TiB can be trimmed, but these are not released, so that one of my RAW disks is now 13 TiB instead of the planned 11 TiB and I am slowly running out of space, cause its still growing. Only 6.8 TiB are occupied within ubuntu.

Is there any other reason why my RAW images keep growing?
 
Last edited:
dkvrhst said:
But not with ubuntu 20: After each VM restart, 3 TiB can be trimmed, but these are not released, so that one of my RAW disks is now 13 TiB instead of the planned 11 TiB and I am slowly running out of space, cause its still growing. Only 6.8 TiB are occupied within ubuntu.
Click to expand...
Does your storage support thin provisioning? What storage type are you using for your raw-format? Directory does not appear to support trimming unless you use qcow2, for example.
 
Storage Type is zfspool:
rpool/data/vm-100-disk-0 => 13.0 TB
.
.
.
 
Last edited:
df -h inside ubuntu VM says 10 TB with 7 TB used and 2.5 TB available while zfs list for the zfspool says 13.0 TB used

qm config for the same VM:
agent: 1,fstrim_cloned_disks=1
boot: order=ide2;scsi0;net0
cores: 16
ide2: none, media=cdrom
memory: 98304 name: *removed*
net: virtio=*removed*,bridge=vmbrß,firewall=1
numa: 0
onboot: 1
ostype: 126
scsi0: local-zfs:vm-100-disk-0,discard=on,size=10T,ssd=1
scsihw: virtio-scsi-pci
smbios1: *removed*
sockets: 1
vmgenid: *removed*

pveversion -v:
proxmox-ve: 7.4-1 (running kernel: 5.15.102-1-pve)
pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a)
pve-kernel-5.15: 7.3-3
pve-kernel-5.13: 7.1-9
pve-kernel-5.11: 7.0-10
pve-kernel-5.15.102-1-pve: 5.15.102-1
pve-kernel-5.15.85-1-pve: 5.15.85-1
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-1-pve: 5.11.22-2
ceph-fuse: 15.2.13-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-4
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-1
libpve-rs-perl: 0.7.5
libpve-storage-perl: 7.4-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.1-1
proxmox-backup-file-restore: 2.4.1-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.1-1
proxmox-widget-toolkit: 3.6.5
pve-cluster: 7.3-3
pve-container: 4.4-3
pve-docs: 7.4-2
pve-edk2-firmware: 3.20221111-2
pve-firewall: 4.3-1
pve-firmware: 3.6-4
pve-ha-manager: 3.6.0
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-1
qemu-server: 7.4-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.9-pve1
 
What does zpool list -v and zfs list -o space return? Bigger zvol size also could be because of padding overhead.
 
  • Like
Reactions: leesteken
zpool status -v
pool: rpool
state: ONLINE
scan: scrub repaired 0B in 03:42:07 with 0 errors on Sun Mar 12 04:06:08 2023
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
nvme-eui.*removed*-part3 ONLINE 0 0 0
nvme-eui.*removed*-part3 ONLINE 0 0 0
nvme-eui.*removed*-part3 ONLINE 0 0 0
nvme-eui.*removed*-part3 ONLINE 0 0 0
nvme-eui.*removed*-part3 ONLINE 0 0 0
nvme-eui.*removed*-part3 ONLINE 0 0 0

errors: No known data errors

zfs list -o space
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
rpool 792G 13.1T 0B 208K 0B 13.1T
rpool/ROOT 792G 11.9G 0B 192K 0B 11.9G
rpool/ROOT/pve-1 792G 11.9G 0B 11.9G 0B 0B
rpool/data 792G 13.0T 0B 192K 0B 13.0T
rpool/data/vm-100-disk-0 792G 13.0T 0B 13.0T 0B 0B
rpool/data/vm-103-disk-0 792G 19.4G 0B 19.4G 0B 0B
 
Would be way easier to red those tables, if you would put the output in CODE-blocks, so format won't be lost.

From what I see discard and snapshots isn't the problem here, as there is no "usedrefreserv" and no "usedsnap". But you are using a raidz2 and you probably didn't increase the volblocksize, so everything you write to a zvol will be way bigger because of padding overhead. Thats why your 7TB used inside the VM will consume 13TB of the pools capacity.
 
With a ashift=12 and 6 disks in raidz2 you will get:
33% of raw capacity when using the default 8K volblocksize.
66% of raw capacity when using 16K volblocksize or higher.

So way to go would be to go to your ZFS storage, increase the "Block size" field from 8K to 16K, backup every VM, destroy those VMs and then restore them from backup (as the volblocksize can only be set when creating a zvol).

Without that everyhting you write to a VMs virtual disk will consume double the size, as for each block of data there will also be a padding block.
 
Last edited:
dkvrhst said:
My volblocksize is actually 16K?
Click to expand...
You can check what volblocksize is used with zfs get volblocksize and zpool get ashift. If it is the default ashift=12 and volblocksize=8K then every VM will be 200% in size.
 
Dunuin said:
You can check what volblocksize is used with zfs get volblocksize and zpool get ashift. If it is the default ashift=12 and volblocksize=8K then every VM will be 200% in size.
Click to expand...
ashift is 12 and block size is 16K
 
You must log in or register to reply here.
Top Bottom