Help setting iser transport for ZFS on ISCSI

[cjaffree]

Hi all,

I'm new to Proxmox having been running VMWare since maybe 3.5 or something. Decided to switch over because I wanted to take the cheaper path to Infiniband and then found the support in VMWare not quite there.

My setup now is Proxmox server with a dual-port Mellanox ConnectX-3 card. My storage server is Solaris 11 with Comstar. I've been able to install open-iscsi, discover targets, configure target for iser transport and log into target. Problem is, when I login using iscsiadm, the login uses the iser transport as expected. However, when using qm start <vmid> to bring up a vm that uses a zfs on iscsi disk, I show the transport for the new login session that (kvm apparently) establishes is using tcp as the transport. I should mention too that zfs on iscsi has already been setup, ssh is working and all that jazz. I ca create a VM on pm using the zfs on iscsi storage as the backing and the volumes gets properly created on the storage server.

Is kvm simply not able to support iser and so this is a lost cause?


pve version:

root@pmox1:~# pveversion -v
proxmox-ve: 6.2-2 (running kernel: 5.4.65-1-pve)
pve-manager: 6.2-12 (running version: 6.2-12/b287dd27)
pve-kernel-5.4: 6.2-7
pve-kernel-helper: 6.2-7
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-9
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
openvswitch-switch: 2.12.0-1
proxmox-backup-client: 0.9.1-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.3-1
pve-cluster: 6.2-1
pve-container: 3.2-2
pve-docs: 6.2-6
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-1
pve-qemu-kvm: 5.1.0-3
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-15
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve2

Login sessions on storage server:

root@stor1:~# itadm list-target -v
TARGET NAME                                                  STATE    SESSIONS
iqn.1986-03.com.sun:02:13e4a574-4bd0-4349-ae71-f9a283902702  online   3     
        alias:                  storage-bond-target
        auth:                   none (defaults)
        targetchapuser:         -
        targetchapsecret:       unset
        tpg-tags:               ib-tpg2 = 2
        Initiator:              iqn.1993-08.org.debian:01:17d981965af
        Connections:            3
          ISID:                 00023d000005
            CID:                        0
              Target Address:           1.0.0.1:3260
              Host Address:             1.0.0.10:55493
              Transport:                iser
          ISID:                 00023d000006
            CID:                        0
              Target Address:           1.0.0.1:3260
              Host Address:             1.0.0.10:49075
              Transport:                iser
          ISID:                 805b7a060000
            CID:                        0
              Target Address:           1.0.0.1:3260
              Host Address:             1.0.0.10:44928
              Transport:                socket

 

[wolfgang]

Hi,

Is kvm simply not able to support iser and so this is a lost cause?

I never test this but you could and report back.
At the moment iscsi is hardcoded and no option for iser is possible.

qemu uses libiscsi and libiscsi should able to use iser.

qm showcmd <VMID>

take the output and replace iscsi again iser.

the option starts with -drive 'file=iscsi:...

then execute it manually.

 

[cjaffree]

Hmmmm, I see the support in libiscsi. Looks like it's definitely there for 1.18.0 and I've got 1.18.0-2 installed.

When trying to start manually, I get an error:

root@pmox1:~# qemu-system-x86_64 -id 100 -name vmm1 -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=803139d1-d189-4553-892a-5205a0c5f832' -smp '2,sockets=2,cores=1,maxcpus=2' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc unix:/var/run/qemu-server/100.vnc,password -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep -m 2048 -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'vmgenid,guid=be6a2aac-d288-4bed-9e2d-8ad2aaa7fd65' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'VGA,id=vga,bus=pci.0,addr=0x2' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:17d981965af' -drive 'file=/var/lib/vz/template/iso/CentOS-7-x86_64-Minimal-2003.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=iser://1.0.0.1/iqn.1986-03.com.sun:02:13e4a574-4bd0-4349-ae71-f9a283902702/0,if=none,id=drive-scsi0,discard=on,format=raw,cache=none,aio=native,detect-zeroes=unmap' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=7A:0E:57:F9:93:66,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -machine 'type=pc+pve0'
qemu-system-x86_64: -drive file=iser://1.0.0.1/iqn.1986-03.com.sun:02:13e4a574-4bd0-4349-ae71-f9a283902702/0,if=none,id=drive-scsi0,discard=on,format=raw,cache=none,aio=native,detect-zeroes=unmap: iSCSI: Failed to connect to LUN : Failed to start full connect Couldn't connect transport

root@pmox1:~# iscsiadm -m session
iser: [1] 1.0.0.1:3260,2 iqn.1986-03.com.sun:02:13e4a574-4bd0-4349-ae71-f9a283902702 (non-flash)

 

[cjaffree]

More detail:

root@pmox1:~# LIBISCSI_DEBUG=10 qemu-system-x86_64 -id 100 -name vmm1 -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=803139d1-d189-4553-892a-5205a0c5f832' -smp '2,sockets=2,cores=1,maxcpus=2' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc unix:/var/run/qemu-server/100.vnc,password -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep -m 2048 -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'vmgenid,guid=be6a2aac-d288-4bed-9e2d-8ad2aaa7fd65' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'VGA,id=vga,bus=pci.0,addr=0x2' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:17d981965af' -drive 'file=/var/lib/vz/template/iso/CentOS-7-x86_64-Minimal-2003.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=iser://1.0.0.1/iqn.1986-03.com.sun:02:13e4a574-4bd0-4349-ae71-f9a283902702/0,if=none,id=drive-scsi0,discard=on,format=raw,cache=none,aio=native,detect-zeroes=unmap' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=7A:0E:57:F9:93:66,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -machine 'type=pc+pve0'
libiscsi:5 small allocation size is 256 byte
libiscsi:2 connecting to portal 1.0.0.1 [iqn.1986-03.com.sun:02:13e4a574-4bd0-4349-ae71-f9a283902702]
libiscsi:1 Failed creating Event Channel
 [iqn.1986-03.com.sun:02:13e4a574-4bd0-4349-ae71-f9a283902702]
libiscsi:1 Couldn't connect transport [iqn.1986-03.com.sun:02:13e4a574-4bd0-4349-ae71-f9a283902702]
libiscsi:1 Failed to start full connect Couldn't connect transport [iqn.1986-03.com.sun:02:13e4a574-4bd0-4349-ae71-f9a283902702]
libiscsi:1 1 memory blocks lost at iscsi_destroy_context() after 3 malloc(s), 0 realloc(s), 2 free(s) and 0 reused small allocations [iqn.1986-03.com.sun:02:13e4a574-4bd0-4349-ae71-f9a283902702]
qemu-system-x86_64: -drive file=iser://1.0.0.1/iqn.1986-03.com.sun:02:13e4a574-4bd0-4349-ae71-f9a283902702/0,if=none,id=drive-scsi0,discard=on,format=raw,cache=none,aio=native,detect-zeroes=unmap: iSCSI: Failed to connect to LUN : Failed to start full connect Couldn't connect transport