[SOLVED] Storage Issues on Proxmox 4

A

adamb

Famous Member
Proxmox Subscriber
Mar 1, 2012
1,329
77
113
I am running into a very odd issue with a new iSCSI storage solution we purchased. I will do my best to describe the issue I am seeing as it is a very odd one.

Storage Hardware
Nimble CS300 iSCSI 10G

Servers
HP DL 380 Gen9

I currently have two hosts configured and connected to the Nimble CS300. One of the hosts is a updated Proxmox 3.4 install and the other is a updated Proxmox 4 install.

If I setup a fresh VM on the Proxmox 4 node and try to install any OS it will fail due to not being able to create the filesystem properly. It looks as if its an issue with creating the journal.

I was originally thinking this was a issue with the storage, but after testing with Proxmox 3.4 I don't feel that is the case.

I can move the VM config over to the proxmox 3.4 host, and boom it installs with no issues. This leads me to believe that this is an issue with proxmox 4, but im very stumped as to what the issue could be.

OS Install's I have tried, all of them have the same issues. Move over to the Proxmox 3.4 host and the install no problem.
CentOS6
CentOS7
Ubuntu
Windows 7

Providing some SS of a CentOS6 install when it fails.

Failed_CMD.png
Failed_GUI.png
 
Man this is odd.

After the install fails I move to ctrl+alt+f2 and attempt to create and mount a ext4 filesystem but it fails like the below SS.

ext4_creation.png
ext4_mount.png

As another test I decided to test creating a ext3 and xfs filesystem and there are no issues at all. Below SS is only of the xfs creation and mounting, but ext3 also had no issues.

xfs_creation_mount.png
 
This is interesting. On the Proxmox 4 host, I can install with no issues if I change the cache settings of the disk.

Works
writethrough
writeback
writeback (unsafe)

Doesn't Work
Default (no cache)
Direct Sync
No cache

So the question is, what changed between Proxmox 3.4 and Proxmox 4 for the disk cache types which arn't working for me.
 
dietmar said:
On local storage? Or over iSCSI? If iSCSI, are there ane error logs in /var/log/syslog (on the host)?
Click to expand...

Storage is iSCSI. No errors that I notice in the logs on the Proxmox 4 host.

Code: 
root@ccsmiscrit1:/var/log# cat syslog.1 | grep iscsi
Nov 19 08:25:58 ccsmiscrit1 iscsid: iSCSI logger with pid=2569 started!
Nov 19 08:25:58 ccsmiscrit1 open-iscsi[2548]: Starting iSCSI initiator service: iscsid.
Nov 19 08:25:58 ccsmiscrit1 open-iscsi[2548]: Setting up iSCSI targets:
Nov 19 08:25:58 ccsmiscrit1 iscsid: iSCSI daemon with pid=2570 started!
Nov 19 08:25:58 ccsmiscrit1 open-iscsi[2548]: Logging in to [iface: default, target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] (multiple)
Nov 19 08:25:58 ccsmiscrit1 open-iscsi[2548]: Logging in to [iface: default, target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] (multiple)
Nov 19 08:25:58 ccsmiscrit1 open-iscsi[2548]: Login to [iface: default, target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] successful.
Nov 19 08:25:58 ccsmiscrit1 open-iscsi[2548]: Login to [iface: default, target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] successful.
Nov 19 08:25:58 ccsmiscrit1 open-iscsi[2548]: .
Nov 19 08:25:58 ccsmiscrit1 open-iscsi[2548]: Mounting network filesystems:.
Nov 19 08:25:58 ccsmiscrit1 open-iscsi[2548]: Enabling network swap devices:.
Nov 19 08:25:58 ccsmiscrit1 iscsid: Connection1:0 to [target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] through [iface: default] is operational now
Nov 19 08:25:58 ccsmiscrit1 iscsid: Connection2:0 to [target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] through [iface: default] is operational now
Nov 19 08:25:58 ccsmiscrit1 kernel: [    1.901787] iscsi: registered transport (be2iscsi)
Nov 19 08:25:58 ccsmiscrit1 kernel: [    1.901789] In beiscsi_module_init, tt=ffffffffc01ad100
Nov 19 08:25:58 ccsmiscrit1 kernel: [    9.789019] scsi host1: Avago Technologies 10Gbe open-iscsi Initiator Driver
Nov 19 08:25:58 ccsmiscrit1 kernel: [   15.768743] scsi host2: Avago Technologies 10Gbe open-iscsi Initiator Driver
Nov 19 08:25:58 ccsmiscrit1 kernel: [   39.834311] iscsi: registered transport (tcp)
Nov 19 08:25:58 ccsmiscrit1 kernel: [   39.862388] iscsi: registered transport (iser)
Nov 19 10:58:39 ccsmiscrit1 iscsid: iSCSI logger with pid=2578 started!
Nov 19 10:58:39 ccsmiscrit1 open-iscsi[2557]: Starting iSCSI initiator service: iscsid.
Nov 19 10:58:39 ccsmiscrit1 open-iscsi[2557]: Setting up iSCSI targets:
Nov 19 10:58:39 ccsmiscrit1 iscsid: iSCSI daemon with pid=2579 started!
Nov 19 10:58:39 ccsmiscrit1 open-iscsi[2557]: Logging in to [iface: default, target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] (multiple)
Nov 19 10:58:39 ccsmiscrit1 open-iscsi[2557]: Logging in to [iface: default, target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] (multiple)
Nov 19 10:58:39 ccsmiscrit1 open-iscsi[2557]: Login to [iface: default, target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] successful.
Nov 19 10:58:39 ccsmiscrit1 open-iscsi[2557]: Login to [iface: default, target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] successful.
Nov 19 10:58:39 ccsmiscrit1 open-iscsi[2557]: .
Nov 19 10:58:39 ccsmiscrit1 open-iscsi[2557]: Mounting network filesystems:.
Nov 19 10:58:39 ccsmiscrit1 open-iscsi[2557]: Enabling network swap devices:.
Nov 19 10:58:39 ccsmiscrit1 iscsid: Connection1:0 to [target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] through [iface: default] is operational now
Nov 19 10:58:39 ccsmiscrit1 iscsid: Connection2:0 to [target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] through [iface: default] is operational now
Nov 19 10:58:39 ccsmiscrit1 kernel: [    1.899563] iscsi: registered transport (be2iscsi)
Nov 19 10:58:39 ccsmiscrit1 kernel: [    1.899565] In beiscsi_module_init, tt=ffffffffc020c100
Nov 19 10:58:39 ccsmiscrit1 kernel: [    9.778110] scsi host1: Avago Technologies 10Gbe open-iscsi Initiator Driver
Nov 19 10:58:39 ccsmiscrit1 kernel: [   15.760453] scsi host2: Avago Technologies 10Gbe open-iscsi Initiator Driver
Nov 19 10:58:39 ccsmiscrit1 kernel: [   39.217240] iscsi: registered transport (tcp)
Nov 19 10:58:39 ccsmiscrit1 kernel: [   39.269808] iscsi: registered transport (iser)
Nov 19 13:01:44 ccsmiscrit1 iscsid: iSCSI logger with pid=2571 started!
Nov 19 13:01:44 ccsmiscrit1 open-iscsi[2546]: Starting iSCSI initiator service: iscsid.
Nov 19 13:01:44 ccsmiscrit1 open-iscsi[2546]: Setting up iSCSI targets:
Nov 19 13:01:44 ccsmiscrit1 iscsid: iSCSI daemon with pid=2572 started!
Nov 19 13:01:44 ccsmiscrit1 open-iscsi[2546]: Logging in to [iface: default, target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] (multiple)
Nov 19 13:01:44 ccsmiscrit1 open-iscsi[2546]: Logging in to [iface: default, target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] (multiple)
Nov 19 13:01:44 ccsmiscrit1 open-iscsi[2546]: Login to [iface: default, target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] successful.
Nov 19 13:01:44 ccsmiscrit1 open-iscsi[2546]: Login to [iface: default, target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] successful.
Nov 19 13:01:44 ccsmiscrit1 open-iscsi[2546]: .
Nov 19 13:01:44 ccsmiscrit1 open-iscsi[2546]: Mounting network filesystems:.
Nov 19 13:01:44 ccsmiscrit1 open-iscsi[2546]: Enabling network swap devices:.
Nov 19 13:01:44 ccsmiscrit1 iscsid: Connection2:0 to [target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] through [iface: default] is operational now
Nov 19 13:01:44 ccsmiscrit1 iscsid: Connection1:0 to [target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] through [iface: default] is operational now
Nov 19 13:01:44 ccsmiscrit1 kernel: [    1.955785] iscsi: registered transport (be2iscsi)
Nov 19 13:01:44 ccsmiscrit1 kernel: [    1.955787] In beiscsi_module_init, tt=ffffffffc038a100
Nov 19 13:01:44 ccsmiscrit1 kernel: [    9.840920] scsi host1: Avago Technologies 10Gbe open-iscsi Initiator Driver
Nov 19 13:01:44 ccsmiscrit1 kernel: [   15.815605] scsi host2: Avago Technologies 10Gbe open-iscsi Initiator Driver
Nov 19 13:01:44 ccsmiscrit1 kernel: [   39.631879] iscsi: registered transport (tcp)
Nov 19 13:01:44 ccsmiscrit1 kernel: [   39.691978] iscsi: registered transport (iser)
Nov 20 05:28:26 ccsmiscrit1 iscsid: iSCSI logger with pid=2570 started!
Nov 20 05:28:26 ccsmiscrit1 open-iscsi[2548]: Starting iSCSI initiator service: iscsid.
Nov 20 05:28:26 ccsmiscrit1 open-iscsi[2548]: Setting up iSCSI targets:
Nov 20 05:28:26 ccsmiscrit1 iscsid: iSCSI daemon with pid=2571 started!
Nov 20 05:28:26 ccsmiscrit1 open-iscsi[2548]: Logging in to [iface: default, target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] (multiple)
Nov 20 05:28:26 ccsmiscrit1 open-iscsi[2548]: Logging in to [iface: default, target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] (multiple)
Nov 20 05:28:26 ccsmiscrit1 open-iscsi[2548]: Login to [iface: default, target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] successful.
Nov 20 05:28:26 ccsmiscrit1 open-iscsi[2548]: Login to [iface: default, target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] successful.
Nov 20 05:28:26 ccsmiscrit1 open-iscsi[2548]: .
Nov 20 05:28:26 ccsmiscrit1 open-iscsi[2548]: Mounting network filesystems:.
Nov 20 05:28:26 ccsmiscrit1 open-iscsi[2548]: Enabling network swap devices:.
Nov 20 05:28:26 ccsmiscrit1 kernel: [    1.956989] iscsi: registered transport (be2iscsi)
Nov 20 05:28:26 ccsmiscrit1 kernel: [    1.956991] In beiscsi_module_init, tt=ffffffffc019b100
Nov 20 05:28:26 ccsmiscrit1 kernel: [    9.834592] scsi host1: Avago Technologies 10Gbe open-iscsi Initiator Driver
Nov 20 05:28:26 ccsmiscrit1 kernel: [   15.814053] scsi host2: Avago Technologies 10Gbe open-iscsi Initiator Driver
Nov 20 05:28:26 ccsmiscrit1 kernel: [   36.815254] iscsi: registered transport (tcp)
Nov 20 05:28:26 ccsmiscrit1 kernel: [   36.823337] iscsi: registered transport (iser)
Nov 20 05:28:26 ccsmiscrit1 iscsid: Connection2:0 to [target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] through [iface: default] is operational now
Nov 20 05:28:26 ccsmiscrit1 iscsid: Connection1:0 to [target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] through [iface: default] is operational now
Nov 20 06:15:21 ccsmiscrit1 iscsid: iSCSI logger with pid=2570 started!
Nov 20 06:15:21 ccsmiscrit1 open-iscsi[2549]: Starting iSCSI initiator service: iscsid.
Nov 20 06:15:21 ccsmiscrit1 open-iscsi[2549]: Setting up iSCSI targets:
Nov 20 06:15:21 ccsmiscrit1 iscsid: iSCSI daemon with pid=2571 started!
Nov 20 06:15:21 ccsmiscrit1 open-iscsi[2549]: Logging in to [iface: default, target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] (multiple)
Nov 20 06:15:21 ccsmiscrit1 open-iscsi[2549]: Logging in to [iface: default, target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] (multiple)
Nov 20 06:15:21 ccsmiscrit1 open-iscsi[2549]: Login to [iface: default, target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] successful.
Nov 20 06:15:21 ccsmiscrit1 open-iscsi[2549]: Login to [iface: default, target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] successful.
Nov 20 06:15:21 ccsmiscrit1 open-iscsi[2549]: .
Nov 20 06:15:21 ccsmiscrit1 open-iscsi[2549]: Mounting network filesystems:.
Nov 20 06:15:21 ccsmiscrit1 open-iscsi[2549]: Enabling network swap devices:.
Nov 20 06:15:21 ccsmiscrit1 kernel: [    1.951208] iscsi: registered transport (be2iscsi)
Nov 20 06:15:21 ccsmiscrit1 kernel: [    1.951210] In beiscsi_module_init, tt=ffffffffc0374100
Nov 20 06:15:21 ccsmiscrit1 kernel: [    9.818703] scsi host1: Avago Technologies 10Gbe open-iscsi Initiator Driver
Nov 20 06:15:21 ccsmiscrit1 kernel: [   15.798342] scsi host2: Avago Technologies 10Gbe open-iscsi Initiator Driver
Nov 20 06:15:21 ccsmiscrit1 kernel: [   39.355237] iscsi: registered transport (tcp)
Nov 20 06:15:21 ccsmiscrit1 kernel: [   39.425370] iscsi: registered transport (iser)
Nov 20 06:15:21 ccsmiscrit1 iscsid: Connection2:0 to [target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] through [iface: default] is operational now
Nov 20 06:15:21 ccsmiscrit1 iscsid: Connection1:0 to [target: iqn.2007-11.com.nimblestorage:missioncrit-v294ae2cc043d7114.00000004.2d7793d8, portal: 10.210.45.1,3260] through [iface: default] is operational now
 
I just tried removing multipath from the equation but that didn't make a difference.

I also tested creating a ext4 filesystem on the logical volume directly on the host. No problems there. Compared iscsi settings between my prox3 and prox4 nodes, nothing.

Everything runs perfectly on Proxmox 3.4.
 
Last edited:
I was reviewing my notes from when I first hit this issue back with the beta of Proxmox 4 and found that the issue only exists on the first disk created from the iSCSI LUN. As crazy as it sounds, every new disk after the first one doesn't have this issue. For now I simple install with writethrough for the first disk then set it back to cache=none. After that all disks can be left as cache=none. What an odd issue!
 
This seems specific to my storage type, but I don't know were to even begin looking.

It seems we would be aok using cache=writeback so long our guests filesystems are mounted with barriers and we ensure they are working 100%. In theory if the application is written properly, both cache modes should be just as safe. Does that sound correct?

This is the first time for us getting into iSCSI, we have never had issues with SAS. On our HP MSA 2040 config's we have no issue with cache=none, but those are all SAS.
 
Last edited:
Have been working closely with Proxmox Dev's and Nimble engineers and today I finally made a breakthrough in this issue.

It seems this issue was related to writing zero's to the disk. Nimble confirmed that they don't actually write zero's.

So after analyzing the kvm guest command string to start the VM I noticed this option.

detect-zeroes=on

If I set detect-zeroes=off, or remove it all together the issues are 100% resolved. Enable it again and they are reproducible.

Only downside is I don't see an option to disable this from the VM config file.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom