Storage iscsi move results to io error

[krs342]

Hello,

We are currently evaluating the proxmox. We do have a 9 node cluster with latest proxmox 5.1 . We do have 2 x zfs over iscsi configuration ( which are srvs23-voliscsi and srvs22-voliscsi) and these are omnios based storage devices.


When we do want to live migrate the storage of the vm from one iscsi storage to another, migration starts without problems but we see io errors on the vm.

Migration command we have started :

root@kvm03:~# qm move_disk 8000700 scsi0 srvs23-voliscsi
create full clone of drive scsi0 (srvs22-voliscsi:vm-8000700-disk-1)
drive mirror is starting for drive-scsi0
drive-scsi0: transferred: 0 bytes remaining: 1099511627776 bytes total: 1099511627776 bytes progression: 0.00 % busy: 1 ready: 0
drive-scsi0: transferred: 1071644672 bytes remaining: 1098439983104 bytes total: 1099511627776 bytes progression: 0.10 % busy: 1 ready: 0
drive-scsi0: transferred: 2122317824 bytes remaining: 1097389309952 bytes total: 1099511627776 bytes progression: 0.19 % busy: 1 ready: 0
drive-scsi0: transferred: 3238002688 bytes remaining: 1096273625088 bytes total: 1099511627776 bytes progression: 0.29 % busy: 1 ready: 0
drive-scsi0: transferred: 4371513344 bytes remaining: 1095140114432 bytes total: 1099511627776 bytes progression: 0.40 % busy: 1 ready: 0
drive-scsi0: transferred: 5335154688 bytes remaining: 1094176473088 bytes total: 1099511627776 bytes progression: 0.49 % busy: 1 ready: 0
drive-scsi0: transferred: 6282018816 bytes remaining: 1093229608960 bytes total: 1099511627776 bytes progression: 0.57 % busy: 1 ready: 0
drive-scsi0: transferred: 7297040384 bytes remaining: 1092214587392 bytes total: 1099511627776 bytes progression: 0.66 % busy: 1 ready: 0
drive-scsi0: transferred: 8395948032 bytes remaining: 1091115679744 bytes total: 1099511627776 bytes progression: 0.76 % busy: 1 ready: 0
drive-scsi0: transferred: 9068085248 bytes remaining: 1090443542528 bytes total: 1099511627776 bytes progression: 0.82 % busy: 1 ready: 0
drive-scsi0: transferred: 9753853952 bytes remaining: 1089757773824 bytes total: 1099511627776 bytes
^X^Cdrive-scsi0: Cancelling block job


IO errors on the VM (when i run the qm move command these errors are starting to show at the beginning ) :


end_request: I/O error, dev sda, sector 1074008896
I/O error on devıce sda2, logical block 270890
Aborting journal on device sda2-8.
EXT-fs (sda2): Remounting filesystem read-only

Our storage config :
zfs: srvs22-voliscsi
blocksize 8k
iscsiprovider comstar
pool s22
portal 10.0.0.215
target iqn.2010-09.org.napp-it:srvs22
content images
nowritecache 0
sparse 1

zfs: srvs23-voliscsi
blocksize 8k
iscsiprovider comstar
pool s23
portal 10.0.0.216
target iqn.2010-09.org.napp-it:srvs23
content images
nowritecache 0
sparse 1

Our vm configuration :

agent: 1
balloon: 0
bootdisk: scsi0
cores: 6
cpu: host
ide2: none,media=cdrom
keyboard: tr
memory: 32768
name: srvtest.xxx.net
net0: virtio=CE:58:19:04:6B:11,bridge=vmbr0,tag=60
net1: virtio=D2:C0:50C:15:94,bridge=vmbr0,tag=110
numa: 1
onboot: 1
ostype: l26
scsi0: srvs22-voliscsi:vm-8000700-disk-1,cache=writeback,discard=on,iothread=1,size=1T
scsihw: virtio-scsi-single
smbios1: uuid=f015f701-2249-4c98-af7d-1ce1ee0fa294
sockets: 2

And our proxmox package versions :

proxmox-ve: 5.1-32 (running kernel: 4.13.13-2-pve) pve-manager: 5.1-41 (running version: 5.1-41/0b958203) pve-kernel-4.13.4-1-pve: 4.13.4-26 pve-kernel-4.13.13-2-pve: 4.13.13-32 pve-kernel-4.13.8-3-pve: 4.13.8-30 pve-kernel-4.10.17-5-pve: 4.10.17-25 libpve-http-server-perl: 2.0-8 lvm2: 2.02.168-pve6 corosync: 2.4.2-pve3 libqb0: 1.0.1-1 pve-cluster: 5.0-19 qemu-server: 5.0-18 pve-firmware: 2.0-3 libpve-common-perl: 5.0-25 libpve-guest-common-perl: 2.0-14 libpve-access-control: 5.0-7 libpve-storage-perl: 5.0-17 pve-libspice-server1: 0.12.8-3 vncterm: 1.5-3 pve-docs: 5.1-12 pve-qemu-kvm: 2.9.1-5 pve-container: 2.0-18 pve-firewall: 3.0-5 pve-ha-manager: 2.0-4 ksm-control-daemon: 1.2-2 glusterfs-client: 3.8.8-1 lxc-pve: 2.1.1-2 lxcfs: 2.0.8-1 criu: 2.11.1-1~bpo90 novnc-pve: 0.6-4 smartmontools: 6.5+svn4324-1 zfsutils-linux: 0.7.3-pve1~bpo9 openvswitch-switch: 2.7.0-2


Live storage migration is essential for us. May I ask what are we doing wrong here?

 

[fabian]

looks like moving the disk saturates your source storage (or the network link to it) to the point that the VM runs into I/O errors

 

[krs342]

Hello Fabian,

I am sorry I forgot to mention that all servers and storage boxes are connected with 2 x10gbit LACP.

I am checking with nload for the utilization and see that there is only max 8gbit/sec utilization on both input and output on the migration process. BTW nfs works perfect on the server. This problem exists when iscsi is used.

The problem is something on the qemu side but i do not have much experience on kvm and i cannot figure out where.

Best regards

 

[krs342]

Hello,

Has anyone faced this issue before? Any tips would be so helpful.

Best regards

 

[danielb]

This thread is old. But I do have the same issue. Using a CentOS 7.6 + ZoL 0.8.1 storage box (with the LIO backend)