virtio-scsi over LVM problems

ghusson

ghusson

Renowned Member
Proxmox Subscriber
Feb 25, 2015
86
7
73
FRANCE
www.liberasys.com
Hello, here is my setup :

synology server, 2x100G LUN exposed via a common target

proxmox 4.1-1 i cluster on 3 servers

new VM :
bios: ovmf
bootdisk: scsi0
cores: 4
ide2: local:iso/debian-8.2.0-amd64-netinst.iso,media=cdrom
memory: 2048
name: vm1
net0: virtio=3A:34:66:31:36:37,bridge=vmbr0
numa: 0
ostype: l26
scsi0: lvm1-lun0-synology-target1:vm-100-disk-1,size=5G
scsihw: virtio-scsi-pci
smbios1: uuid=7491a96f-312e-40f7-b345-33b80e53f06b
sockets: 1

I try to install debian-8.2.0-amd64-netinst.iso

Bad result : the debian guest sees the whole LUN. When trying to make partitions : "Inout/Output error during write on /dev/sda ERROR!!!"

When using default BIOS (SeaBIOS), the result is good : disk size=5G, partitions are created.

EDIT :
That was OK during installation and at first start. After that :
- I installed qemu-guest-agent
- I shuted down the VM (from inside the guest)
- I enabled qemu-guest-agent on the VM
- I restarted the VM
==> disk seen was now 100G, with IO errors
I removed the agent, disabled it on quest settings, and started the VM, problem is the same.
(see log down here)
Conclusion :
- the documentation encourage iSCSI -> LVM
- my tests reveal a problem when using multiple LVs over this structure
- am I doing something bad or is there a bug somewhere ?


====================
root@debian1:/var/log# cat syslog | grep -E 'logical blocks|I/O error|Kernel command line'
Feb 23 14:44:00 debian1 kernel: [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.16.0-4-amd64 root=UUID=d636efe1-9676-411a-9ddb-e5507b71e1a1 ro quiet
Feb 23 14:44:00 debian1 kernel: [ 0.975933] sd 0:0:0:0: [sda] 10485760 512-byte logical blocks: (5.36 GB/5.00 GiB)
Feb 23 14:50:05 debian1 kernel: [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.16.0-4-amd64 root=UUID=d636efe1-9676-411a-9ddb-e5507b71e1a1 ro quiet
Feb 23 14:50:05 debian1 kernel: [ 1.060430] sd 0:0:0:0: [sda] 209715200 512-byte logical blocks: (107 GB/100 GiB)
Feb 23 14:50:05 debian1 kernel: [ 1.252148] end_request: I/O error, dev sda, sector 209715072
Feb 23 14:50:05 debian1 kernel: [ 1.252432] Buffer I/O error on device sda, logical block 26214384
Feb 23 14:50:05 debian1 kernel: [ 1.292112] end_request: I/O error, dev sda, sector 209715072
Feb 23 14:50:05 debian1 kernel: [ 1.292412] Buffer I/O error on device sda, logical block 26214384
Feb 23 14:50:05 debian1 kernel: [ 2.798884] end_request: I/O error, dev sda, sector 209715072
Feb 23 14:50:05 debian1 kernel: [ 2.799203] Buffer I/O error on device sda, logical block 26214384
Feb 23 14:50:05 debian1 kernel: [ 2.932765] end_request: I/O error, dev sda, sector 209715072
Feb 23 14:50:05 debian1 kernel: [ 2.933070] Buffer I/O error on device sda, logical block 26214384
Feb 23 14:50:05 debian1 kernel: [ 5.552190] end_request: I/O error, dev sda, sector 209715072
Feb 23 14:50:05 debian1 kernel: [ 5.552238] Buffer I/O error on device sda, logical block 26214384
Feb 23 14:53:49 debian1 kernel: [ 235.926060] end_request: I/O error, dev sda, sector 209715072
Feb 23 14:53:49 debian1 kernel: [ 235.926068] Buffer I/O error on device sda, logical block 26214384
Feb 23 14:53:49 debian1 kernel: [ 236.172244] end_request: I/O error, dev sda, sector 209715072
Feb 23 14:53:49 debian1 kernel: [ 236.172247] Buffer I/O error on device sda, logical block 26214384
Feb 23 14:53:50 debian1 kernel: [ 236.692194] end_request: I/O error, dev sda, sector 209715072
Feb 23 14:53:50 debian1 kernel: [ 236.692203] Buffer I/O error on device sda, logical block 26214384
Feb 23 14:53:50 debian1 kernel: [ 236.804195] end_request: I/O error, dev sda, sector 209715072
Feb 23 14:53:50 debian1 kernel: [ 236.804220] Buffer I/O error on device sda, logical block 26214384
Feb 23 14:56:10 debian1 kernel: [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.16.0-4-amd64 root=UUID=d636efe1-9676-411a-9ddb-e5507b71e1a1 ro quiet
Feb 23 14:56:10 debian1 kernel: [ 1.002188] sd 0:0:0:0: [sda] 209715200 512-byte logical blocks: (107 GB/100 GiB)
Feb 23 14:56:10 debian1 kernel: [ 1.200125] end_request: I/O error, dev sda, sector 209715072
Feb 23 14:56:10 debian1 kernel: [ 1.200428] Buffer I/O error on device sda, logical block 26214384
Feb 23 14:56:10 debian1 kernel: [ 1.240148] end_request: I/O error, dev sda, sector 209715072
Feb 23 14:56:10 debian1 kernel: [ 1.240509] Buffer I/O error on device sda, logical block 26214384
Feb 23 14:56:10 debian1 kernel: [ 2.644112] end_request: I/O error, dev sda, sector 209715072
Feb 23 14:56:10 debian1 kernel: [ 2.644433] Buffer I/O error on device sda, logical block 26214384
Feb 23 14:56:10 debian1 kernel: [ 2.666116] end_request: I/O error, dev sda, sector 209715072
Feb 23 14:56:10 debian1 kernel: [ 2.666436] Buffer I/O error on device sda, logical block 26214384
Feb 23 14:56:10 debian1 kernel: [ 5.276010] end_request: I/O error, dev sda, sector 209715072
Feb 23 14:56:10 debian1 kernel: [ 5.276303] Buffer I/O error on device sda, logical block 26214384


====================
 
Last edited:
Now : I changed the drive to scsi-blk. Problem gone. Installed qemu-guest-agent, still no problem.

root@proxmox1:~# cat /etc/pve/qemu-server/100.conf
agent: 1
bios: seabios
boot: cdn
bootdisk: virtio0
cores: 4
ide2: none,media=cdrom
memory: 2048
name: vm1
net0: virtio=3A:34:66:31:36:37,bridge=vmbr0
numa: 0
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=7491a96f-312e-40f7-b345-33b80e53f06b
sockets: 1
virtio0: lvm1-lun0-synology-target1:vm-100-disk-1,size=5G

Conclusion : fear the scsi-virtio ?
 
ghusson said:
spirit : virtio = disk type SCSI and hardware type = VIRTIO, you confirm ?
My test debian VM uses default partitioning : no UEFI leads to msdos partiton, no LVM usage.
Click to expand...

virtio-blk = virtio disk in proxmox
virtio-scsi = scsi disk + virtio scsi controller in proxmox

can you send me the kvm command line when you have running with virtio-scsi ?

#ps -aux|grep kvm|grep <vmid>
 
To be completly clear, do you agree with :
virtio-blk = virtio disk in proxmox Hardware tab + virtio controller in proxmox Options tab
virtio-scsi = scsi disk in proxmox Hardware tab + virtio controller in proxmox Options tab

NEW VM : 110 (SCSI / VIRTIO / SeaBIOS) : sees 100G LUN (bad)
root@proxmox1:~# ps aux | grep kvm | grep 110
root 2191 6.6 0.4 978760 37904 ? Sl 17:04 0:00 /usr/bin/kvm -id 110 -chardev socket,id=qmp,path=/var/run/qemu-server/110.qmp,server,nowait -mon chardev=qmp,mode=control -vnc unix:/var/run/qemu-server/110.vnc,x509,password -pidfile /var/run/qemu-server/110.pid -daemonize -smbios type=1,uuid=de2958d7-bc36-46ab-93d4-7e43129d755b -name plop -smp 1,sockets=1,cores=1,maxcpus=1 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000 -vga cirrus -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 512 -k fr -device pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e -device pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=uhci.0,port=1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:fe2d16479fac -drive file=/var/lib/vz/template/iso/debian-8.2.0-amd64-netinst.iso,if=none,id=drive-ide2,media=cdrom,aio=threads -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -device virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5 -drive file=/dev/vg2/vm-110-disk-1,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on -device scsi-block,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100 -netdev type=tap,id=net0,ifname=tap110i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=36:64:61:64:39:61,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300

root@proxmox1:~# cat /etc/pve/local/
lrm_status lxc/ openvz/ priv/ pve-ssl.key pve-ssl.pem qemu-server/
root@proxmox1:~# cat /etc/pve/local/qemu-server/110.conf
bootdisk: scsi0
cores: 1
ide2: local:iso/debian-8.2.0-amd64-netinst.iso,media=cdrom
memory: 512
name: plop
net0: virtio=36:64:61:64:39:61,bridge=vmbr0
numa: 0
ostype: l26
scsi0: lvm2-lun0-synology-target1:vm-110-disk-1,size=5G
scsihw: virtio-scsi-pci
smbios1: uuid=de2958d7-bc36-46ab-93d4-7e43129d755b
sockets: 1

NEW VM : 110 (VIRTIO / VIRTIO / SeaBIOS) : sees 5G LV (OK)
root 2623 3.5 0.5 986952 42308 ? Sl 17:08 0:00 /usr/bin/kvm -id 111 -chardev socket,id=qmp,path=/var/run/qemu-server/111.qmp,server,nowait -mon chardev=qmp,mode=control -vnc unix:/var/run/qemu-server/111.vnc,x509,password -pidfile /var/run/qemu-server/111.pid -daemonize -smbios type=1,uuid=06d44985-a3c1-4355-9964-d4d3643c593e -name plop2 -smp 1,sockets=1,cores=1,maxcpus=1 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000 -vga cirrus -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 512 -k fr -device pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f -device pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=uhci.0,port=1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:fe2d16479fac -drive file=/dev/vg2/vm-111-disk-1,if=none,id=drive-virtio0,format=raw,cache=none,aio=native,detect-zeroes=on -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100 -drive file=/var/lib/vz/template/iso/debian-8.2.0-amd64-netinst.iso,if=none,id=drive-ide2,media=cdrom,aio=threads -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -netdev type=tap,id=net0,ifname=tap111i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=66:66:64:33:35:39,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300

root@proxmox1:~# cat /etc/pve/local/qemu-server/111.conf
bootdisk: virtio0
cores: 1
ide2: local:iso/debian-8.2.0-amd64-netinst.iso,media=cdrom
memory: 512
name: plop2
net0: virtio=66:66:64:33:35:39,bridge=vmbr0
numa: 0
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=06d44985-a3c1-4355-9964-d4d3643c593e
sockets: 1
virtio0: lvm2-lun0-synology-target1:vm-111-disk-1,size=5G


==> remark that this time, the problem appears with SeaBIOS at first time. This was only at reboot with precedent test. So the BIOS probably has no influence over this problem.

Do you see anything shocking in my configuration or in the kvm options ?
 
tried but it not as simple as that. In cluster mode, it seems that LVs are spawned when PVE starts the VM :

root@proxmox3:~# /usr/bin/kvm -id 100 -chardev socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait -mon chardev=qmp,mode=control -vnc unix:/var/run/qemu-server/100.vnc,x509,password -pidfile /var/run/qemu-server/100.pid -daemonize -smbios type=1,uuid=a6d5cfe1-57e3-4ecd-9ec2-82faef8f9dec -name debiantmpl -smp 2,sockets=1,cores=2,maxcpus=2 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000 -vga cirrus -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 1024 -k fr -device pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f -device pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=uhci.0,port=1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:89685dbb46da -device virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5 -drive file=/dev/vg1-lun0-t1-s1/vm-100-disk-1,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on -device scsi-generic,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100 -drive file=/var/lib/vz/template/iso/debian-8.2.0-amd64-netinst.iso,if=none,id=drive-ide2,media=cdrom,aio=threads -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -netdev type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=3A:32:62:36:30:32,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300
kvm: -drive file=/dev/vg1-lun0-t1-s1/vm-100-disk-1,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on: Could not open '/dev/vg1-lun0-t1-s1/vm-100-disk-1': No such file or directory

Unless you have a quick solution for making the test, I will not pass many time on this bug. It is not mandatory for me to use virtio-scsi, neither OVMF. I was testing new features. They don't work at now. I will retest them at next version.
 
ghusson said:
tried but it not as simple as that. In cluster mode, it seems that LVs are spawned when PVE starts the VM :

root@proxmox3:~# /usr/bin/kvm -id 100 -chardev socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait -mon chardev=qmp,mode=control -vnc unix:/var/run/qemu-server/100.vnc,x509,password -pidfile /var/run/qemu-server/100.pid -daemonize -smbios type=1,uuid=a6d5cfe1-57e3-4ecd-9ec2-82faef8f9dec -name debiantmpl -smp 2,sockets=1,cores=2,maxcpus=2 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000 -vga cirrus -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 1024 -k fr -device pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f -device pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=uhci.0,port=1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:89685dbb46da -device virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5 -drive file=/dev/vg1-lun0-t1-s1/vm-100-disk-1,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on -device scsi-generic,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100 -drive file=/var/lib/vz/template/iso/debian-8.2.0-amd64-netinst.iso,if=none,id=drive-ide2,media=cdrom,aio=threads -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -netdev type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=3A:32:62:36:30:32,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300
kvm: -drive file=/dev/vg1-lun0-t1-s1/vm-100-disk-1,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on: Could not open '/dev/vg1-lun0-t1-s1/vm-100-disk-1': No such file or directory

Unless you have a quick solution for making the test, I will not pass many time on this bug. It is not mandatory for me to use virtio-scsi, neither OVMF. I was testing new features. They don't work at now. I will retest them at next version.
Click to expand...

Ok, thanks. I'll try to provide a patch this week. I'll keep you in touch.
Thanks for reporting it !
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back