Hi All,
first of all, i want to say thanks for great Virtual Environment.
I have some troubles with using IET Target on Linux for ZFS over iSCSI,
for move operation from one iscsi storage to another.
Below: versions, config and error results.
If needs more info, I'll post it. Thx
Tested on Proxmox VE 4.2/4.3
Proxmox 4.3 HA Cluster Config:
nodes: noc16, noc17, noc25
Cluster status & nodes:
storage config:
servers with IET Target ZFS over iSCSI on Debian 8.6 (tested on 8.3/4/5)
two storage servers: zfsnoc3 & zfsnoc4 with the same config
1st zfsnoc3: ZFS & IET iSCSI config:
------------------------------------
2nd zfsnoc4: ZFS & IET iSCSI config:
Create 1st KVM VM via WEB-GUI on node: noc16:
Move Disk Image from iscsi3 storage(zfsnoc3) to iscsi4 storage(zfsnoc4),
via WEB-GUI or CLI:
ZFS Storage iscsi3 (zfsnoc3)
now see iscsi4 (zfsnoc4) storage and config:
We can see that on iscsi4 was created zvol: vm-100-disk-1, but operation didn't complete and vm config didn't
change, because done with error:
Move from one iSCSI Storage to another, don't work on IET Target.
After future experminet, Proxmox crash my /etc/iet/ietd.conf
on first or second ZFS Storage servers, for example, on zfsnoc4
was:
after incomplete move may be rewrite to target definition from zfsnoc3,
as:
???
What's wrong?
Proxmox don't supported MOVE from one ZFSoveriSCSI target to another on IET Target?
As I wrote above, this was tested on Proxmox 4.2/4.3 with IET Target,
but seems on Proxmox 4.1 also didn't work.
Resume of my test labs, which i tried:
- FreeBSD & any based NAS on release higher 10.x iSCSI target
don't supported by Proxmox
- IET Target (any Linux'es) don't have FULL support by Proxmox
Only Solaris-based iSCSI Comstar Target have Full Proxmox support.
Is it so, what OS can you recommend:
- OpenIndiana (it's my preffer)
- OmniOS
?
Thanks for great jobs and good VE.
Best reagrds
--
lavr
first of all, i want to say thanks for great Virtual Environment.
I have some troubles with using IET Target on Linux for ZFS over iSCSI,
for move operation from one iscsi storage to another.
Below: versions, config and error results.
If needs more info, I'll post it. Thx
Tested on Proxmox VE 4.2/4.3
Proxmox 4.3 HA Cluster Config:
nodes: noc16, noc17, noc25
Code:
root@noc16:~# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
159.93.11.16 noc16.local noc16 pvelocalhost
#
159.93.11.17 noc17.local noc17
159.93.11.25 noc25.local noc25
#
159.93.11.18 zfsnoc1.local zfsnoc1
159.93.11.28 zfsnoc2.local zfsnoc2
#
159.93.11.38 zfsnoc3.jinr.ru zfsnoc3
159.93.11.48 zfsnoc4.jinr.ru zfsnoc4
#
10.93.11.16 ibnoc16.local ibnoc16
10.93.11.17 ibnoc17.local ibnoc17
10.93.11.25 ibnoc25.local ibnoc25
#
10.93.11.18 ibzfsnoc1.local ibzfsnoc1
10.93.11.28 ibzfsnoc2.local ibzfsnoc2
Code:
root@noc16:~# pvecm nodes
Membership information
----------------------
Nodeid Votes Name
1 1 noc16 (local)
2 1 noc17
3 1 noc25storage config:
Code:
root@noc16:~# cat /etc/pve/storage.cfg
zfs: iscsi3
pool raid50/iscsi
blocksize 4k
iscsiprovider iet
target iqn.2016-01.ru.jinr.zfsnoc3:noc3iscsi
portal 159.93.11.38
nowritecache 0
sparse 0
content images
zfs: iscsi4
pool raid50/iscsi
blocksize 4k
iscsiprovider iet
target iqn.2016-01.ru.jinr.zfsnoc4:noc4iscsi
portal 159.93.11.48
nowritecache 0
sparse 0
content imagesservers with IET Target ZFS over iSCSI on Debian 8.6 (tested on 8.3/4/5)
two storage servers: zfsnoc3 & zfsnoc4 with the same config
1st zfsnoc3: ZFS & IET iSCSI config:
------------------------------------
Code:
root@zfsnoc3:~# cat /etc/debian_version
8.6
root@zfsnoc3:~# dpkg -l | grep zfs
ii debian-zfs 7~jessie amd64 Nat
ive ZFS filesystem metapackage for Debian.
ii libzfs2 0.6.5.7-8-jessie amd64 Nat
ive ZFS filesystem library for Linux
ii zfs-dkms 0.6.5.7-8-jessie all Nat
ive ZFS filesystem kernel modules for Linux
ii zfsonlinux 8 all arc
hive.zfsonlinux.org trust package
ii zfsutils 0.6.5.7-8-jessie amd64 com
mand-line tools to manage ZFS filesystems
root@zfsnoc3:~# cat /etc/debian_version
8.6
root@zfsnoc3:~# dpkg -l | grep iscsi
ii iscsitarget 1.4.20.3+svn502-1 amd64 iSC
SI Enterprise Target userland tools
ii iscsitarget-dkms 1.4.20.3+svn502-1 all iSC
SI Enterprise Target kernel module source - dkms version
Code:
root@zfsnoc3:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
raid10 648K 3.51T 96K /raid10
raid10/backup 96K 3.51T 96K /raid10/backup
raid10/iscsi 96K 3.51T 96K /raid10/iscsi
raid10/nfs 96K 3.51T 96K /raid10/nfs
raid50 3.71G 10.5T 153K /raid50
raid50/backup 153K 10.5T 153K /raid50/backup
raid50/iscsi 153K 10.5T 153K /raid50/iscsi
raid50/nfs 3.71G 10.5T 3.71G /raid50/nfs
Code:
root@zfsnoc3:~# cat /etc/iet/ietd.conf
Target iqn.2016-01.ru.jinr.zfsnoc3:noc3iscsi
Code:
root@zfsnoc3:~# cat /proc/net/iet/volume
tid:1 name:iqn.2016-01.ru.jinr.zfsnoc3:noc3iscsi
root@zfsnoc3:~# cat /proc/net/iet/session
tid:1 name:iqn.2016-01.ru.jinr.zfsnoc3:noc3iscsi
Code:
root@zfsnoc4:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
raid10 648K 3.51T 96K /raid10
raid10/backup 96K 3.51T 96K /raid10/backup
raid10/iscsi 96K 3.51T 96K /raid10/iscsi
raid10/nfs 96K 3.51T 96K /raid10/nfs
raid50 1.22M 10.5T 153K /raid50
raid50/backup 153K 10.5T 153K /raid50/backup
raid50/iscsi 153K 10.5T 153K /raid50/iscsi
raid50/nfs 153K 10.5T 153K /raid50/nfs
Code:
root@zfsnoc4:~# cat /etc/iet/ietd.conf
Target iqn.2016-01.ru.jinr.zfsnoc4:noc4iscsi
Code:
root@zfsnoc4:~# cat /proc/net/iet/volume
tid:1 name:iqn.2016-01.ru.jinr.zfsnoc4:noc4iscsi
root@zfsnoc4:~# cat /proc/net/iet/session
tid:1 name:iqn.2016-01.ru.jinr.zfsnoc4:noc4iscsiCreate 1st KVM VM via WEB-GUI on node: noc16:
Code:
root@noc16:~# qm list
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
100 testvm stopped 512 32.00 0
root@noc16:~# qm config 100
bootdisk: virtio0
cores: 1
ide2: nfs3:iso/debian-8.6.0-amd64-netinst.iso,media=cdrom
memory: 512
name: testvm
net0: virtio=CA:F5:DE:85:BA:5A,bridge=vmbr0
numa: 0
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=ab96042a-8b19-4695-9cc9-3457c61991d5
sockets: 1
virtio0: iscsi3:vm-100-disk-1,size=32Gvia WEB-GUI or CLI:
Code:
root@noc16:~# qm move_disk 100 virtio0 iscsi4 -delete 1
create full clone of drive virtio0 (iscsi3:vm-100-disk-1)
storage migration failed: /dev/raid50/iscsi/vm-100-disk-1: LUN exists at /usr/sh
are/perl5/PVE/Storage/LunCmd/Iet.pm line 362.
Code:
root@zfsnoc3:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
raid10 648K 3.51T 96K /raid10
raid10/backup 96K 3.51T 96K /raid10/backup
raid10/iscsi 96K 3.51T 96K /raid10/iscsi
raid10/nfs 96K 3.51T 96K /raid10/nfs
raid50 37.7G 10.5T 153K /raid50
raid50/backup 153K 10.5T 153K /raid50/backup
raid50/iscsi 34.0G 10.5T 153K /raid50/iscsi
raid50/iscsi/vm-100-disk-1 34.0G 10.5T 102K -
raid50/nfs 3.71G 10.5T 3.71G /raid50/nfs
Code:
root@zfsnoc3:~# cat /etc/iet/ietd.conf
Target iqn.2016-01.ru.jinr.zfsnoc3:noc3iscsi
Lun 0 Path=/dev/raid50/iscsi/vm-100-disk-1,Type=blockio
root@zfsnoc3:~# cat /proc/net/iet/volume
tid:1 name:iqn.2016-01.ru.jinr.zfsnoc3:noc3iscsi
lun:0 state:0 iotype:blockio iomode:wt blocks:67108864 blocksize:512 pat
h:/dev/raid50/iscsi/vm-100-disk-1
root@zfsnoc3:~# cat /proc/net/iet/session
tid:1 name:iqn.2016-01.ru.jinr.zfsnoc3:noc3iscsinow see iscsi4 (zfsnoc4) storage and config:
Code:
root@zfsnoc4:~# cat /etc/iet/ietd.conf
Target iqn.2016-01.ru.jinr.zfsnoc4:noc4iscsi
root@zfsnoc4:~# cat /proc/net/iet/volume
tid:1 name:iqn.2016-01.ru.jinr.zfsnoc4:noc4iscsi
root@zfsnoc4:~# cat /proc/net/iet/session
tid:1 name:iqn.2016-01.ru.jinr.zfsnoc4:noc4iscsi
root@zfsnoc4:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
raid10 648K 3.51T 96K /raid10
raid10/backup 96K 3.51T 96K /raid10/backup
raid10/iscsi 96K 3.51T 96K /raid10/iscsi
raid10/nfs 96K 3.51T 96K /raid10/nfs
raid50 34.0G 10.5T 153K /raid50
raid50/backup 153K 10.5T 153K /raid50/backup
raid50/iscsi 34.0G 10.5T 153K /raid50/iscsi
raid50/iscsi/vm-100-disk-1 34.0G 10.5T 102K -
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
raid50/nfs 153K 10.5T 153K /raid50/nfschange, because done with error:
Code:
root@noc16:~# qm config 100
bootdisk: virtio0
cores: 1
ide2: nfs3:iso/debian-8.6.0-amd64-netinst.iso,media=cdrom
memory: 512
name: testvm
net0: virtio=CA:F5:DE:85:BA:5A,bridge=vmbr0
numa: 0
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=ab96042a-8b19-4695-9cc9-3457c61991d5
sockets: 1
virtio0: iscsi3:vm-100-disk-1,size=32GMove from one iSCSI Storage to another, don't work on IET Target.
After future experminet, Proxmox crash my /etc/iet/ietd.conf
on first or second ZFS Storage servers, for example, on zfsnoc4
was:
Code:
Target iqn.2016-01.ru.jinr.zfsnoc4:noc4iscsias:
Code:
Target iqn.2016-01.ru.jinr.zfsnoc3:noc3iscsiWhat's wrong?
Proxmox don't supported MOVE from one ZFSoveriSCSI target to another on IET Target?
As I wrote above, this was tested on Proxmox 4.2/4.3 with IET Target,
but seems on Proxmox 4.1 also didn't work.
Resume of my test labs, which i tried:
- FreeBSD & any based NAS on release higher 10.x iSCSI target
don't supported by Proxmox
- IET Target (any Linux'es) don't have FULL support by Proxmox
Only Solaris-based iSCSI Comstar Target have Full Proxmox support.
Is it so, what OS can you recommend:
- OpenIndiana (it's my preffer)
- OmniOS
?
Thanks for great jobs and good VE.
Best reagrds
--
lavr