iSCSI mount of device with 4k blocs

jloms

New Member
Jul 28, 2020
10
4
3
Bonsoir,

I'm testing proxmox/CEPH for the migration of a libvirtd/Kvm configuration.

The problem I get, is I use some 'data' disks on guests on my libvird that I can't 'mount on my proxmox platform ...

Theses disks are iSCSI LUNs from equallogic disks, formatted with 4 k sectors and used whith ZFS.

For a while I would need to mount theses LUNS (iSCSI not ZFS over iSCSI) on my proxmox members ... but I cant specify the 4 k blocksize .... I'm sure I can mount these devices on the guests, but I think it would be more simple and robust to have them in the storage list of the members of the cluster.

Is there a way to specify the block size of a iSCSI device, or the VirtIO disk added to a VM, like I do in libvirtd in the quemu/guest.xml ?

Thanks
 

aaron

Proxmox Staff Member
Staff member
Jun 3, 2019
3,009
493
88
Theses disks are iSCSI LUNs from equallogic disks, formatted with 4 k sectors and used whith ZFS.

I am a bit confused, are the iSCSI LUNs used in a ZFS pool?

Is there a way to specify the block size of a iSCSI device, or the VirtIO disk added to a VM, like I do in libvirtd in the quemu/guest.xml ?
I personally haven't used that in libvirtd. Can you show an example of that config?
 

jloms

New Member
Jul 28, 2020
10
4
3
Sorry but last week has been a bit hard and I was a bit confused ...

iSCSI mount from equallogic is Ok, and the raw device (or LUN) come as it has ben formated (Phy & logical sector of 4096).

The problem is when you give this raw device to a guest VM ... within the VM sector is 512 .... and I can't use anymore this raw device as a vdev of a zpool local to my guest.

What I find looking deeper in the forum is to add this in my wmid.conf:
args: -global virtio-blk-device.physical_block_size=4096 -global virtio-blk-device.logical_block_size=4096

That is working ... but all my virtio disks are 4096 sectors ...

Then I change my system disk to SCSI, because it's a 512 bytes sectors.

I can work this way, but in libvirtd, the qemu/vmguest.xml you lets you specify sector size fore each disk device, that can't be done in proxmox ?

I don't find a list of args that can be passed to qemu, and syntax doesn't seeem to be exactly what I found in qemu doc.

Any solution ?
 

jloms

New Member
Jul 28, 2020
10
4
3
In libvirtd/qemu in xml you have :

<disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/eql/seafile1'/> <blockio logical_block_size='4096' physical_block_size='4096'/> <target dev='vdb' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </disk>

And you find this on the qemu process args:
/usr/libexec/qemu-kvm -name oceanfiles ... -drive file=/dev/eql/seafile1,format=raw,if=none,id=drive-virtio-disk1,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk1,[B]id=virtio-disk1,logical_block_size=4096,physical_block_size=4096 [/B] ....
 
  • Like
Reactions: Kyle

jloms

New Member
Jul 28, 2020
10
4
3
Many tests later ... I think this is not possible.

The block_size parameters (logical & physical) must be added to the virtioX: definition .... but there are not implemented in proxmox. may be on a next release .....
 

jloms

New Member
Jul 28, 2020
10
4
3
I've done some modifications to code, to achieve my goal, changing sector sizes of some of the virtio disks of a guest. My tests seem ok, but I don't know enough about the code to be sure.

My version : pve-manager/6.2-10/a20769ed (running kernel: 5.4.44-2-pve)


1) I introduce a new boolean param in PVE/QemuServer/Drive.pm:

Perl:
+++ QemuServer/Drive.pm    2020-08-05 22:16:20.352838899 +0200
@@ -145,7 +145,12 @@
     verbose_description => "Mark this locally-managed volume as available on all nodes.\n\nWARNING: This option does not share the volume automatically, it assumes it is shared already!",
     optional => 1,
     default => 0,
-    }
+    },
+    block_size_4k => {
+    type => 'boolean',
+    description => 'Logical and Physical block_size to 4096',
+    optional=> 1,
+    },
 );
 
 my %iothread_fmt = ( iothread => {

2) In QemuServer.pm I add the logical/physical block_size to the -device argument of qemu :

Perl:
+++ QemuServer.pm    2020-08-05 23:30:27.612148298 +0200
@@ -1319,6 +1319,7 @@
     my $pciaddr = print_pci_addr("$drive->{interface}$drive->{index}", $bridges, $arch, $machine_type);
     $device = "virtio-blk-pci,drive=drive-$drive->{interface}$drive->{index},id=$drive->{interface}$drive->{index}$pciaddr";
     $device .= ",iothread=iothread-$drive->{interface}$drive->{index}" if $drive->{iothread};
+    $device .= ",logical_block_size=4096,physical_block_size=4096" if $drive->{block_size_4k};
     } elsif ($drive->{interface} eq 'scsi') {
 
     my ($maxdev, $controller, $controller_prefix) = scsihw_infos($conf, $drive);
@@ -4623,7 +4624,8 @@
             safe_string_ne($drive->{iothread}, $old_drive->{iothread}) ||
             safe_string_ne($drive->{queues}, $old_drive->{queues}) ||
             safe_string_ne($drive->{cache}, $old_drive->{cache}) ||
-            safe_string_ne($drive->{ssd}, $old_drive->{ssd})) {
+            safe_string_ne($drive->{ssd}, $old_drive->{ssd}) ||
+            safe_string_ne($drive->{block_size_4k}, $old_drive->{block_size_4k})) {
             die "skip\n";
             }

Not sure of the need to add the new param to not hotplug list ....

3) My test VM:

qm config 100 agent: 1 boot: cdn bootdisk: scsi0 cores: 2 memory: 4096 net0: virtio=CE:FB:59:EB:4A:14,bridge=vmbr2,firewall=1 numa: 0 ostype: l26 parent: vm100s1 rng0: source=/dev/urandom scsi0: Pool1:vm-100-disk-0,size=32G scsihw: virtio-scsi-pci serial0: socket smbios1: uuid=f3bde295-76c8-4b48-b31c-736922406aa7 sockets: 1 virtio0: Pool1:vm-100-disk-2,size=4G,block_size_4k=1 virtio1: prox-4k:0.0.0.scsi-36090a0a8403bf0d843f265a0cd0400c3,size=51210M,block_size_4k=1 virtio2: Pool1:vm-100-disk-1,size=2G vmgenid: 01b4b712-01a2-4374-bf8d-2d3ad9655bf0
 
  • Like
Reactions: Kyle and Secundo°

Kyle

Member
Oct 18, 2017
8
4
23
40
I was able to use qm showcmd 102 > ~/102-start-script.sh, i then modified the relevant -device per post #4 and appended ,logical_block_size=4096,physical_block_size=4096

@aaron It would be good to see this make the config on a per device basis rather than global basis.
 
Last edited:

Kyle

Member
Oct 18, 2017
8
4
23
40
@aaron just wondering if you spotted my reply #7. Thx

would it be possible to add this to the tweaks backlog/queue?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!