bad performance on cluster

[copymaster]

Hi

After a lot of configuring and asking questions my VMs on the Cluster (3 servers) increase.
i use ISCSI/LVM on a netapp and every server of the node has 2 network cards. One connected to the LAN and the other one connected to another subnet with netapp ISCSI.

So every server has one connection to the netapp and one to the LAN.

All KVM machines are residing on the ISCSI/LVM LUN on the netapp. But i recognize performance problems with these machines. the vm's seems to "freeze" or are responding VERY slow. i used the e1000 network card driver and the vms all are windows 2003

the LOG of the nodes are telling almost the same:

Jan 17 10:26:51 Donald kernel: scsi5 : iSCSI Initiator over TCP/IP
Jan 17 10:26:52 Donald kernel: scsi 5:0:0:0: Direct-Access     NETAPP   LUN              0.2  PQ: 0 ANSI: 4
Jan 17 10:26:52 Donald kernel: sd 5:0:0:0: [sdb] 2147518464 512-byte hardware sectors (1099529 MB)
Jan 17 10:26:52 Donald kernel: sd 5:0:0:0: [sdb] Write Protect is off
Jan 17 10:26:52 Donald kernel: sd 5:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 17 10:26:52 Donald kernel: sd 5:0:0:0: [sdb] 2147518464 512-byte hardware sectors (1099529 MB)
Jan 17 10:26:52 Donald kernel: sd 5:0:0:0: [sdb] Write Protect is off
Jan 17 10:26:52 Donald kernel: sd 5:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 17 10:26:52 Donald kernel: sdb: unknown partition table
Jan 17 10:26:52 Donald kernel: sd 5:0:0:0: [sdb] Attached SCSI disk
Jan 17 10:26:52 Donald kernel: sd 5:0:0:0: Attached scsi generic sg2 type 0
Jan 17 10:26:53 Donald kernel: sd 5:0:0:0: [sdb] 2147518464 512-byte hardware sectors (1099529 MB)
Jan 17 10:26:53 Donald kernel: sd 5:0:0:0: [sdb] Write Protect is off
Jan 17 10:26:53 Donald kernel: sd 5:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 17 10:27:51 Donald kernel: sd 5:0:0:0: [sdb] 2147518464 512-byte hardware sectors (1099529 MB)
Jan 17 10:27:51 Donald kernel: sd 5:0:0:0: [sdb] Write Protect is off
Jan 17 10:27:51 Donald kernel: sd 5:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Jan 17 10:28:51 Donald kernel: sd 5:0:0:0: [sdb] 2147518464 512-byte hardware sectors (1099529 MB)
Jan 17 10:28:51 Donald kernel: sd 5:0:0:0: [sdb] Write Protect is off
Jan 17 10:28:51 Donald kernel: sd 5:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA

Does this have anything to do with my problem?

 

[thoohm]

Hi copymaster,
those messages about DPO and FUA are quite common and no problem at all. But what seems weird is that write cache is disabled. If no cache can be written to disk that might be one performance problem at least. Also that message about the unknown partition table.
What format do you use for your partitions? Do you have the appropriate filesystem drivers installed?

Greets
Thomas

 

[copymaster]

well, i have a netapp and created a lun with 1TB

Then i added that iscsi Target through the webinterface of proxmox VE
after that i used this ISCSI Target as base for a lvm-Volume and used that for the Virtual machines.

As Dietmar told me it is not necessary to format the iscsi disk.

I created the LUN on the netapp as base for "LINUX". on the netapp there's a config option when you create a llun. YOu can say "LINUX" , "windows" or "vmware" on creation.

I don't know if i can enable writecache on a ISCSI LUN ???

 

[udo]

Hi,
the error message about "unknown partition table" is normal, because you use the full iSCSI-Disk for lvm - so you don't have a partition-table.
What is the Output of IO-Delay in the Webfrontend (home) when the client struggle?

You can also test the speed of your iSCSI-Device: create a logical volume on it, make a filesystem (ext3) on this lv and mount it eg. to /mnt.
Then run pveperf /mnt
Look at IO Delay and speed when you write to the lv (perhaps "time (dd if=/dev/zero of=/mnt/bigfile bs=1024k count=16384; sync)" )

At the end just unmount /mnt and lvremove the lv.

Udo

 

[xrisse]

Look at IO Delay and speed when you write to the lv (perhaps "time (dd if=/dev/zero of=/mnt/bigfile bs=1024k count=16384; sync)" )

Hi,

for more realistic results from dd, you schould use oflags!

"dd if=/dev/zero of=/mnt/bigfile bs=1M count=16384 oflag=direct,sync"

and if you have a second console, you can type "kill -USR1 $(pidof dd)" while dd is running. It will display the current state of dd on the first console ;-)