Getting iSCSI storage working with proxmox

[Ced]

Maybe I'm just dense, but I've looked everywhere to my idea.
I have a windows server, with Microsoft iSCSI software target installed, and it works fine between windows clients.
Now, I tried hooking it up to proxmox, which is all fun and games until I try actually using it.
What I did:
Datacenter => Storage tab => Add => iSCSI target => Fill in all data => add => Works => Add => LVM group => Fill in all stuff and select the appropriate storage volume => Add.
After that, the storage appears, but I can't add anything, I'll just get a 'can't save file' error. Additionally, in the terminal/syslog output I see this:

Jun  9 19:21:31 Cobalt kernel: sd 8:0:0:0: [sdb] Assuming drive cache: write through
Jun  9 19:21:51 Cobalt kernel: sd 8:0:0:0: [sdb] Got wrong page
Jun  9 19:21:51 Cobalt kernel: sd 8:0:0:0: [sdb] Assuming drive cache: write through
Jun  9 19:22:11 Cobalt kernel: sd 8:0:0:0: [sdb] Got wrong page
Jun  9 19:22:11 Cobalt kernel: sd 8:0:0:0: [sdb] Assuming drive cache: write through
Jun  9 19:22:31 Cobalt kernel: sd 8:0:0:0: [sdb] Got wrong page
Jun  9 19:22:31 Cobalt kernel: sd 8:0:0:0: [sdb] Assuming drive cache: write through
Jun  9 19:22:51 Cobalt kernel: sd 8:0:0:0: [sdb] Got wrong page
Jun  9 19:22:51 Cobalt kernel: sd 8:0:0:0: [sdb] Assuming drive cache: write through
Jun  9 19:23:11 Cobalt kernel: sd 8:0:0:0: [sdb] Got wrong page
Jun  9 19:23:11 Cobalt kernel: sd 8:0:0:0: [sdb] Assuming drive cache: write through
Jun  9 19:23:31 Cobalt kernel: sd 8:0:0:0: [sdb] Got wrong page
Jun  9 19:23:31 Cobalt kernel: sd 8:0:0:0: [sdb] Assuming drive cache: write through
Jun  9 19:23:51 Cobalt kernel: sd 8:0:0:0: [sdb] Got wrong page
Jun  9 19:23:51 Cobalt kernel: sd 8:0:0:0: [sdb] Assuming drive cache: write through
Jun  9 19:24:11 Cobalt kernel: sd 8:0:0:0: [sdb] Got wrong page
Jun  9 19:24:11 Cobalt kernel: sd 8:0:0:0: [sdb] Assuming drive cache: write through

And that just goes on forever.

Now, I tried removing and re-adding all the storage-related things (except local storage), but now I get stuck when I try add a LVM group to the iscsi target:
"create storage failed: device '/dev/disk/by-id/scsi-360003ff047cc3c448c88ea8d6431979d' is already used by volume group 'LVMISOSTOR' (500)"
And it just stops with that. I didn't use the terminal for anything so far but doing an apt-get update/upgrade, so maybe it's a bug in a new package?



This is a completely fresh install, I've used proxmox 1.9 for at least a year before this, but I wanted to redo the whole thing and get going with iscsi, which led to this
I'd really appreciate any comments on this, I'm completely stuck.

 

[Ced]

Got nothing better to do, gonna reinstall the server and try see if it's the apt-get upgrade to blame for it.
[8:22PM]
WELL, I reinstalled. Got the same 'create storage failed' error, so figured it must've been part of the server.
Created a new target and virtual disk and tried again, now I'm getting this:
"create storage failed: vgcreate ISOstorage /dev/disk/by-id/scsi-360003ff047cc3c448c88ea8d6431979d error: Unable to add physical volume '/dev/disk/by-id/scsi-360003ff047cc3c448c88ea8d6431979d' to volume group 'ISOstorage'. (500)"
Additionally, the "Got wrong page" and "Assuming drive cache: write-through" messages haven't gone away either.
I haven't ran apt-get upgrade yet, so it's not a new package.. perhaps a protocol mismatch with microsoft iscsi?

 

[udo]

Got nothing better to do, gonna reinstall the server and try see if it's the apt-get upgrade to blame for it.
[8:22PM]
WELL, I reinstalled. Got the same 'create storage failed' error, so figured it must've been part of the server.
Created a new target and virtual disk and tried again, now I'm getting this:
"create storage failed: vgcreate ISOstorage /dev/disk/by-id/scsi-360003ff047cc3c448c88ea8d6431979d error: Unable to add physical volume '/dev/disk/by-id/scsi-360003ff047cc3c448c88ea8d6431979d' to volume group 'ISOstorage'. (500)"
Additionally, the "Got wrong page" and "Assuming drive cache: write-through" messages haven't gone away either.
I haven't ran apt-get upgrade yet, so it's not a new package.. perhaps a protocol mismatch with microsoft iscsi?

Hi,
I assume on the iscsi-disk is allready an lvm.
Look (and post) with following commands:

iscsiadm -m session
fdisk -l
pvs
vgs
lvs

Udo

 

[Ced]

iscsiadm -m session
fdisk -l
pvs
vgs
lvs

root@Cobalt:~# iscsiadm -m session
tcp: [1] 192.168.5.11:3260,1 iqn.1991-05.com.microsoft:celebi-isostor-target
root@Cobalt:~# fdisk -l

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0002fb31

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          66      523264   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2              66      121602   976237568   8e  Linux LVM

Disk /dev/dm-0: 103.1 GB, 103079215104 bytes
255 heads, 63 sectors/track, 12532 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/dm-0 doesn't contain a valid partition table

Disk /dev/dm-1: 7516 MB, 7516192768 bytes
255 heads, 63 sectors/track, 913 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/dm-1 doesn't contain a valid partition table

Disk /dev/dm-2: 871.9 GB, 871895138304 bytes
255 heads, 63 sectors/track, 106001 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/dm-2 doesn't contain a valid partition table

Disk /dev/sdb: 104.9 GB, 104857600000 bytes
255 heads, 63 sectors/track, 12748 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sdb doesn't contain a valid partition table
root@Cobalt:~# pvs
  PV         VG   Fmt  Attr PSize   PFree
  /dev/sda2  pve  lvm2 a--  931.01g 16.00g
  /dev/sdb        lvm2 a--   97.66g 97.66g
root@Cobalt:~# vgs
  VG   #PV #LV #SN Attr   VSize   VFree
  pve    1   3   0 wz--n- 931.01g 16.00g
root@Cobalt:~# lvs
  LV   VG   Attr     LSize   Pool Origin Data%  Move Log Copy%  Convert
  data pve  -wi-ao-- 812.02g
  root pve  -wi-ao--  96.00g
  swap pve  -wi-ao--   7.00g

In the mean time, I'm also trying to setup a freenas iscsi to see if it's a protocol related error.

 

[udo]

root@Cobalt:~# iscsiadm -m session
tcp: [1] 192.168.5.11:3260,1 iqn.1991-05.com.microsoft:celebi-isostor-target
root@Cobalt:~# fdisk -l

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0002fb31

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          66      523264   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2              66      121602   976237568   8e  Linux LVM

Disk /dev/dm-0: 103.1 GB, 103079215104 bytes
255 heads, 63 sectors/track, 12532 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/dm-0 doesn't contain a valid partition table

Disk /dev/dm-1: 7516 MB, 7516192768 bytes
255 heads, 63 sectors/track, 913 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/dm-1 doesn't contain a valid partition table

Disk /dev/dm-2: 871.9 GB, 871895138304 bytes
255 heads, 63 sectors/track, 106001 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/dm-2 doesn't contain a valid partition table

Disk /dev/sdb: 104.9 GB, 104857600000 bytes
255 heads, 63 sectors/track, 12748 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sdb doesn't contain a valid partition table
root@Cobalt:~# pvs
  PV         VG   Fmt  Attr PSize   PFree
  /dev/sda2  pve  lvm2 a--  931.01g 16.00g
  /dev/sdb        lvm2 a--   97.66g 97.66g
root@Cobalt:~# vgs
  VG   #PV #LV #SN Attr   VSize   VFree
  pve    1   3   0 wz--n- 931.01g 16.00g
root@Cobalt:~# lvs
  LV   VG   Attr     LSize   Pool Origin Data%  Move Log Copy%  Convert
  data pve  -wi-ao-- 812.02g
  root pve  -wi-ao--  96.00g
  swap pve  -wi-ao--   7.00g

In the mean time, I'm also trying to setup a freenas iscsi to see if it's a protocol related error.

Hi,
I'm right, that your iscsi-disk is 100GB in size?
If you don't expand the disk over 2TB use "fdisk /dev/sdb" to create an partition on the disk (type 8e), otherwise use parted with an gpt-partition.
Create a volumegroup:

pvcreate /dev/sdb1
vgcreate -s 4M iscsi_win /dev/sdb1

after that, you can define the VG inside the storage-gui as excisting lvm-group.

Udo

 

[Ced]

Seems strange that this can't be done from the GUI?
..Anyway, I've done that now, but I dont' get what you mean by this:

after that, you can define the VG inside the storage-gui as excisting lvm-group.

If you meant that I should do Add LVM group => fill in the stuff and select the lun just like the last time => add, I get this:
"create storage failed: command '/sbin/pvs --separator : --noheadings --units k --unbuffered --nosuffix --options pv_name,pv_size,vg_name,pv_uuid /dev/disk/by-id/scsi-360003ff047cc3c44b3e273631df44f3d' failed: exit code 5 (500)"
Or am I doing something wrong?

 

[udo]

Seems strange that this can't be done from the GUI?
..Anyway, I've done that now, but I dont' get what you mean by this:

If you meant that I should do Add LVM group => fill in the stuff and select the lun just like the last time => add, I get this:
"create storage failed: command '/sbin/pvs --separator : --noheadings --units k --unbuffered --nosuffix --options pv_name,pv_size,vg_name,pv_uuid /dev/disk/by-id/scsi-360003ff047cc3c44b3e273631df44f3d' failed: exit code 5 (500)"
Or am I doing something wrong?

Hi,
if you create an lvm-group (partition, pvcreate, vgcreate) you can add this group with
DataCenter -> Storage -> Add -> LVM-Group
use a name
Existing Volumegroup
Select Volumegroup (the name which you have used with vgcreate)
Select enable and in if the storage are accesible also from other nodes select shared too.

Udo

 

[Ced]

Hi,
if you create an lvm-group (partition, pvcreate, vgcreate) you can add this group with
DataCenter -> Storage -> Add -> LVM-Group
use a name
Existing Volumegroup
Select Volumegroup (the name which you have used with vgcreate)
Select enable and in if the storage are accesible also from other nodes select shared too.

Udo

Thanks! That worked.

...However, when I try upload something to the storage I get "Error 500: cant upload to storage type 'lvm' at /usr/share/perl5/PVE/API2/Storage/Status.pm line 298."
I suspect I wrongly formatted it though, so I'm gonna try that first. Thanks for the help so far!

 

[Ced]

Nope! Still doesn't work
Do I really just suck at this or is something wrong? xD;
Sorry for being such a leech but I think we're almost there, whatever I'm doing wrong this time.
I did exactly as you said, is there something I missed?

 

[udo]

Thanks! That worked.

...However, when I try upload something to the storage I get "Error 500...

Upload??
You create an lvm-storage - this is like an resizable blockdevice - with no filesystem on. This is the reason, why you can't upload things to it.

But you can create an VM with an disk, which are on this storage. The disk of the vm is an logical volume on the volume group. And if you have more than one node, you can live migrate such an VM, because the VM-disks are available on all nodes.

Udo

 

[Ced]

Upload??
You create an lvm-storage - this is like an resizable blockdevice - with no filesystem on. This is the reason, why you can't upload things to it.

But you can create an VM with an disk, which are on this storage. The disk of the vm is an logical volume on the volume group. And if you have more than one node, you can live migrate such an VM, because the VM-disks are available on all nodes.

Udo

Oh! Well, thanks, that solves one big mystery.
Though, isn't the disk of a VM a file itself? (.raw, qcow2, vmdk, etc)
And next mystery, how do I make a storage for e.g ISOs with this? I have the iscsi mapped, the LVM is made, now I just wanna store stuff on it.
So should I just partition and format it?
Again I don't mean to be demanding but, aren't there other people that want to create a setup like this? You know, store files on a NAS/SAN and access it on the hypervisor? I couldn't even find option for SMB/CIFS, they seem to assume you follow this one certain setup, and if you want anything else you have to do it through the console..

Again thanks for helping so far!

 

[udo]

Oh! Well, thanks, that solves one big mystery.
Though, isn't the disk of a VM a file itself? (.raw, qcow2, vmdk, etc)
And next mystery, how do I make a storage for e.g ISOs with this? I have the iscsi mapped, the LVM is made, now I just wanna store stuff on it.

Again thanks for helping so far!

Hi,
the VM-disk is not in all cases a file - this need an underlaying filesystem. Other ways: lvm-storage, direct harddisk access.

To save ISOs you need an filesystem which is normaly only local (except an clusterfilesystem) or an nfs-mount.
If you wan't to use the iscsi-volume only for ISO-files (and only at one node) you can remove the lvm-storage (remove in the gui, vgremove and pvremove) and after that create an normal filesystem on the partition. Mount the partion with an entry in /etc/fstab (use uuid for the device to avoid further trouble).

Or ( I prefer this way) simply create an lv on the vg and mount this for ISOs - so you are able to use the remaining space for VM-disks.

To create an lv use someting like "lvcreate -n isostore -L 50G /dev/iscsi; mkfs.ext4 /dev/mapper/iscsi-isostore".

Udo