iSCSI mount of device with 4k blocs

[jloms]

Bonsoir,

I'm testing proxmox/CEPH for the migration of a libvirtd/Kvm configuration.

The problem I get, is I use some 'data' disks on guests on my libvird that I can't 'mount on my proxmox platform ...

Theses disks are iSCSI LUNs from equallogic disks, formatted with 4 k sectors and used whith ZFS.

For a while I would need to mount theses LUNS (iSCSI not ZFS over iSCSI) on my proxmox members ... but I cant specify the 4 k blocksize .... I'm sure I can mount these devices on the guests, but I think it would be more simple and robust to have them in the storage list of the members of the cluster.

Is there a way to specify the block size of a iSCSI device, or the VirtIO disk added to a VM, like I do in libvirtd in the quemu/guest.xml ?

Thanks

 

[aaron]

Theses disks are iSCSI LUNs from equallogic disks, formatted with 4 k sectors and used whith ZFS.

I am a bit confused, are the iSCSI LUNs used in a ZFS pool?

Is there a way to specify the block size of a iSCSI device, or the VirtIO disk added to a VM, like I do in libvirtd in the quemu/guest.xml ?

I personally haven't used that in libvirtd. Can you show an example of that config?

 

[jloms]

Sorry but last week has been a bit hard and I was a bit confused ...

iSCSI mount from equallogic is Ok, and the raw device (or LUN) come as it has ben formated (Phy & logical sector of 4096).

The problem is when you give this raw device to a guest VM ... within the VM sector is 512 .... and I can't use anymore this raw device as a vdev of a zpool local to my guest.

What I find looking deeper in the forum is to add this in my wmid.conf:
args: -global virtio-blk-device.physical_block_size=4096 -global virtio-blk-device.logical_block_size=4096

That is working ... but all my virtio disks are 4096 sectors ...

Then I change my system disk to SCSI, because it's a 512 bytes sectors.

I can work this way, but in libvirtd, the qemu/vmguest.xml you lets you specify sector size fore each disk device, that can't be done in proxmox ?

I don't find a list of args that can be passed to qemu, and syntax doesn't seeem to be exactly what I found in qemu doc.

Any solution ?

 

[jloms]

In libvirtd/qemu in xml you have :

<disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/eql/seafile1'/> <blockio logical_block_size='4096' physical_block_size='4096'/> <target dev='vdb' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </disk>

And you find this on the qemu process args:
/usr/libexec/qemu-kvm -name oceanfiles ... -drive file=/dev/eql/seafile1,format=raw,if=none,id=drive-virtio-disk1,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk1,[B]id=virtio-disk1,logical_block_size=4096,physical_block_size=4096 [/B] ....

 

[jloms]

Many tests later ... I think this is not possible.

The block_size parameters (logical & physical) must be added to the virtioX: definition .... but there are not implemented in proxmox. may be on a next release .....

 

[jloms]

I've done some modifications to code, to achieve my goal, changing sector sizes of some of the virtio disks of a guest. My tests seem ok, but I don't know enough about the code to be sure.

My version : pve-manager/6.2-10/a20769ed (running kernel: 5.4.44-2-pve)


1) I introduce a new boolean param in PVE/QemuServer/Drive.pm:

+++ QemuServer/Drive.pm    2020-08-05 22:16:20.352838899 +0200
@@ -145,7 +145,12 @@
     verbose_description => "Mark this locally-managed volume as available on all nodes.\n\nWARNING: This option does not share the volume automatically, it assumes it is shared already!",
     optional => 1,
     default => 0,
-    }
+    },
+    block_size_4k => {
+    type => 'boolean',
+    description => 'Logical and Physical block_size to 4096',
+    optional=> 1,
+    },
 );
 
 my %iothread_fmt = ( iothread => {

2) In QemuServer.pm I add the logical/physical block_size to the -device argument of qemu :

+++ QemuServer.pm    2020-08-05 23:30:27.612148298 +0200
@@ -1319,6 +1319,7 @@
     my $pciaddr = print_pci_addr("$drive->{interface}$drive->{index}", $bridges, $arch, $machine_type);
     $device = "virtio-blk-pci,drive=drive-$drive->{interface}$drive->{index},id=$drive->{interface}$drive->{index}$pciaddr";
     $device .= ",iothread=iothread-$drive->{interface}$drive->{index}" if $drive->{iothread};
+    $device .= ",logical_block_size=4096,physical_block_size=4096" if $drive->{block_size_4k};
     } elsif ($drive->{interface} eq 'scsi') {
 
     my ($maxdev, $controller, $controller_prefix) = scsihw_infos($conf, $drive);
@@ -4623,7 +4624,8 @@
             safe_string_ne($drive->{iothread}, $old_drive->{iothread}) ||
             safe_string_ne($drive->{queues}, $old_drive->{queues}) ||
             safe_string_ne($drive->{cache}, $old_drive->{cache}) ||
-            safe_string_ne($drive->{ssd}, $old_drive->{ssd})) {
+            safe_string_ne($drive->{ssd}, $old_drive->{ssd}) ||
+            safe_string_ne($drive->{block_size_4k}, $old_drive->{block_size_4k})) {
             die "skip\n";
             }

Not sure of the need to add the new param to not hotplug list ....

3) My test VM:

qm config 100 agent: 1 boot: cdn bootdisk: scsi0 cores: 2 memory: 4096 net0: virtio=CE:FB:59:EB:4A:14,bridge=vmbr2,firewall=1 numa: 0 ostype: l26 parent: vm100s1 rng0: source=/dev/urandom scsi0: Pool1:vm-100-disk-0,size=32G scsihw: virtio-scsi-pci serial0: socket smbios1: uuid=f3bde295-76c8-4b48-b31c-736922406aa7 sockets: 1 virtio0: Pool1:vm-100-disk-2,size=4G,block_size_4k=1 virtio1: prox-4k:0.0.0.scsi-36090a0a8403bf0d843f265a0cd0400c3,size=51210M,block_size_4k=1 virtio2: Pool1:vm-100-disk-1,size=2G vmgenid: 01b4b712-01a2-4374-bf8d-2d3ad9655bf0

 

[Kyle]

I was able to use qm showcmd 102 > ~/102-start-script.sh, i then modified the relevant -device per post #4 and appended ,logical_block_size=4096,physical_block_size=4096

@aaron It would be good to see this make the config on a per device basis rather than global basis.

 

[Kyle]

@aaron just wondering if you spotted my reply #7. Thx

would it be possible to add this to the tweaks backlog/queue?