How to do 4Kn VirtIO disks

[Uxorious]

I just set up a new Proxmox 7.0 on a 4Kn native array.
Everything is running, and I can create and run VMs fine.
But the VM disks I create are treated as 512B drives.
This works within the VM - but from the Proxmox host, I can't do any mounting or inspection of the drives since the sector size is not right.

Is there some way to get Proxmox to create the VM drive images as 4Kn?

 

[ryba84]

How do You try to mount, because for me it's working this same with any logical block size used. Try to do:

fdisk -l /dev/your-pool/your-path/to-zvol

and then try

mount -o loop,offset=512×first-part-sector /dev/your-pool/your-path/to-zvol /your/mountpoint

 

[Uxorious]

So in Proxmox itself, /dev/sda reports as 4096/4096.
If I create a VM and install Ubuntu in the Thin-LVM storage VM (where the Thin-LVM is on the 4Kn drive), then Ubunty will report 512B sectors.
Now from Proxmox again, if I do "fdisk -l /dev/mapper/RAID-vm-blahblah" it shows me nothing since it can't parse a 512 sector MBR/GPT layout when it's expecting 4K sectors. I can pass --sector-size to fdisk, and then it works.
But things like "kpartx -a /dev/mapper/RAID-vm-blahblah" does not find anything either, so mounting partitions on the virtual disk doesn't work trivially.
I *can* use "losetup" to mount it as a loopback device, and then fdisk/kpartx etc works on the loop device.
But the whole thing is a pain.

And really, the reason I wanted 4Kn sectors was to make sure all data is aligned correctly.
If the VM ignores this and creates 512B devices, then I still have to double-check that everything is done right.

 

[fabian]

https://bugzilla.proxmox.com/show_bug.cgi?id=3282

 

[Uxorious]

Thanks for the related link.
I hadn't thought of the bootability issue if I create the drive with 4Kn sectors.
But I think I agree with Bug 3200 - shouldn't the drive by default be created with the underlying block size.
Then if for some reason (boot would be a good one) 512e is required for some volumes, we should be able to override it.

The filesystem errors sound scary though ... maybe it's still to early to put 4Kn and Proxmox live.

 

[fabian]

well, you could give the setup a spin for some non-critical or non-production VM and post the results to the bug tracker

 

[Uxorious]

Yeah I will try something to see if it would at least work, and if I get any obvious errors without too much effort.

One question: Where do I specify the global options?

 

[fabian]

the value like described in the bug entry - you need to add the 'args' string, either directly to the VM config file, or to the commandline shown by qm showcmd VMID if you just want to start it for a one-off test

 

[Uxorious]

I did a bit of testing and updated the Bugzilla.
TRIM is definitely throwing errors when the device is set to 4K/4K sectors.

 

[Dunuin]

You can add this line to the config file of your VM:
args: -global scsi-hd.physical_block_size=4k

That way virtio SCSI is using 4K and not 512B blocksize.

 

[Uxorious]

You can add this line to the config file of your VM:
args: -global scsi-hd.physical_block_size=4k

That way virtio SCSI is using 4K and not 512B blocksize.

Yeah that's basically what's being done in the Bugzilla.
But it only changes the physical sector size, so blocks are still reported 512/4096 instead of 4Kn.

 

[Uxorious]

I'm trying to track down where things are going wrong.
So far I've been experimenting with an Ubuntu guest within Proxmox.
But it looks like the problem is coming from below - virtio/qemu or something like that.
So I need to debug down into the Proxmox side.

I will install a Proxmox in a VM so I can recover quicker if I make a mistake.

But can somebody give me a quick rundown of what I need to install to be able to iterate on building the kernel and qemu?
For the kernel, I at least got it building - but how do I then make a change to some module and build again (without doing a clean rebuild)?

Typing "make" in the pve-qemu directory gives me an "apt unmet dependencies" error.

 

[fabian]

you need to install the build dependencies (usually, apt build-dep . does the trick, but some repos have a different source layout or generate debian/control as first build step)