slow performance since upgrade to 4.x

[El Muchacho]

hi folkz ...

... I've upgraded from 3.4 to 4.1 today.
Upgrade-procedure went smooth and without any errors.

But the performance of the three running windows-vms is very poor now.
f.e.:
- boot time grows from < 2min to > 10min ...
- high cpu load on kvm for all machines

these vms run fine before upgrading from 3.4 to 4.1 !
and the machines are set-up by using the best practice hints ...

Anything changed i haven't seen?

regards

 

[markusd]

Hi,
which hardware?
Have you tried latest kernel from "pve-test" or an older one?
Are you using pci-passthrough?
Then you could search for related posts here..

Markus

 

[El Muchacho]

It's an intel-server E3-1230 V2 with an 3ware 9750-4i RAID controller.

lspci -nn | grep 3w
02:00.0 RAID bus controller [0104]: 3ware Inc 9750 SAS2/SATA-II RAID PCIe [13c1:1010] (rev 05)

lsmod | grep 3w
3w_sas                 28672  2

shouldn't this be 3w_9xxx ?
antother machine uses this one wih an 3ware 9650 on Debian Wheezy ...

more details:

pveversion -v
proxmox-ve: 4.1-33 (running kernel: 4.2.6-1-pve)
pve-manager: 4.1-5 (running version: 4.1-5/f910ef5c)
pve-kernel-4.2.6-1-pve: 4.2.6-33
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 0.17.2-1
pve-cluster: 4.0-30
qemu-server: 4.0-46
pve-firmware: 1.1-7
libpve-common-perl: 4.0-43
libpve-access-control: 4.0-11
libpve-storage-perl: 4.0-38
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.4-21
pve-container: 1.0-37
pve-firewall: 2.0-15
pve-ha-manager: 1.0-18
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-5
lxcfs: 0.13-pve3
cgmanager: 0.39-pve1
criu: 1.6.0-1
fence-agents-pve: not correctly installed

pveperf
CPU BOGOMIPS:      52678.24
REGEX/SECOND:      1770192
HD SIZE:           94.37 GB (/dev/dm-0)
BUFFERED READS:    293.08 MB/sec
AVERAGE SEEK TIME: 14.31 ms
FSYNCS/SECOND:     21.50
DNS EXT:           1101.14 ms
DNS INT:           1019.03 ms (mydomain.lan)

very low fsyncs ... i think ...

and I also found some interesting info in kern.log:

before the upgrade:

Jan 15 23:01:17 pmatz kernel: 3w-sas: scsi0: AEN: INFO (0x04:0x0029): Verify started:unit=0, subunit=1.
Jan 15 23:01:17 pmatz kernel: 3w-sas: scsi0: AEN: INFO (0x04:0x0029): Verify started:unit=0, subunit=0.
Jan 16 07:41:55 pmatz kernel: 3w-sas: scsi0: AEN: INFO (0x04:0x002B): Verify completed:unit=0, subunit=1.
Jan 16 08:30:51 pmatz kernel: 3w-sas: scsi0: AEN: INFO (0x04:0x002B): Verify completed:unit=0, subunit=0.

and now after the upgrade to actual version:

Jan 19 11:48:18 pmatz kernel: [  1.156632] LSI 3ware SAS/SATA-RAID Controller device driver for Linux v3.26.02.000.
Jan 19 11:48:18 pmatz kernel: [  1.994424] scsi host0: 3w-sas
Jan 19 11:48:18 pmatz kernel: [  2.106419] 3w-sas: scsi0: Found an LSI 3ware 9750-4i Controller at 0xc2460000, IRQ: 17.
Jan 19 11:48:18 pmatz kernel: [  2.442411] 3w-sas: scsi0: Firmware FH9X 5.12.00.007, BIOS BE9X 5.11.00.006, Phys: 8.
Jan 19 11:48:18 pmatz kernel: [  9.914314] 3w-sas: scsi0: ERROR: (0x03:0x0101): Invalid command opcode:opcode=0x85.
Jan 19 11:48:18 pmatz kernel: [  9.935836] 3w-sas: scsi0: ERROR: (0x03:0x0101): Invalid command opcode:opcode=0x85.
Jan 19 11:48:18 pmatz kernel: [  9.957339] 3w-sas: scsi0: ERROR: (0x03:0x0101): Invalid command opcode:opcode=0x85.
Jan 19 11:48:18 pmatz kernel: [  10.000157] 3w-sas: scsi0: ERROR: (0x03:0x0101): Invalid command opcode:opcode=0x85.
Jan 19 11:48:18 pmatz kernel: [  10.021660] 3w-sas: scsi0: ERROR: (0x03:0x0101): Invalid command opcode:opcode=0x85.
Jan 19 11:48:18 pmatz kernel: [  10.043164] 3w-sas: scsi0: ERROR: (0x03:0x0101): Invalid command opcode:opcode=0x85.
Jan 19 11:48:18 pmatz kernel: [  10.074510] 3w-sas: scsi0: ERROR: (0x03:0x0101): Invalid command opcode:opcode=0x85.
Jan 19 11:48:18 pmatz kernel: [  10.096019] 3w-sas: scsi0: ERROR: (0x03:0x0101): Invalid command opcode:opcode=0x85.
Jan 19 11:48:18 pmatz kernel: [  10.117523] 3w-sas: scsi0: ERROR: (0x03:0x0101): Invalid command opcode:opcode=0x85.

any ideas ?!

 

[Lonnie]

There were many Proxmox 4 updates released today. I'm curious if you've been able to apply them and possibly reboot your server. I wonder if you'll notice some speed increases from them.

 

[El Muchacho]

updates installed ... no change ...
pveperf still shows FSYNCS < 20

 

[mmenaz]

updates installed ... no change ...
pveperf still shows FSYNCS < 20

Is your writeback raid cache working for sure? Are you on ext4 and is mounted with barries (by default is on, it's safest but much slower)? Are you testing pveperf with NO VM running? With no vm running, iotop -o is showing some strange activity by some daemon? I've no more ideas left

 

[El Muchacho]

hi ...

I'm on ext3.
here the fstab

# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext3 errors=remount-ro 0 1
/dev/pve/data /var/lib/vz ext3 defaults 0 1
#/dev/pve/data /var/lib/vz ext3 rw,relatime,data=ordered 0 0
UUID=3bcb2bfd-f519-4fe5-94b9-98c8280a4031 /boot ext3 defaults 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0

and here some info from kern.log

Jan 20 08:53:04 pmatz kernel: [    7.963428] EXT4-fs (dm-0): mounting ext3 file system using the ext4 subsystem
Jan 20 08:53:04 pmatz kernel: [   10.609397] EXT4-fs (sda1): mounting ext3 file system using the ext4 subsystem

is that OK ?

no "strange" activity in iotop

and for sure ... pveperf was done without any running vm

but something seemed to be changed in the kernel or anywhere else:

3ware 9650 with bbu runs fine in balance mode using module 3w_9xxx on proxmox 3.4 and also after the upgrade to actual 4.1
no need to switch to performance mode.
3ware 9750 without bbu runs fine in balance mode using module 3w_sas on proxmox 3.4
After upgrading to 4.1 the fsyncs went down from about 7000 to 15 in balance mode
After switching the controller to performance mode everything runs fine again.

Looks like the module checks controller for an bbu in balance mode now and switches to protection mode, if no bbu is present.

So i'll go for a BBU for this controller

 

[Ovidiu]

@El Muchacho how did you change the controller's mode, by booting into tis menu or via some command line commands? If the latter woudl you mind pointing me to some documentation or sharing how you did it?

 

[El Muchacho]

I've used the 3ware web gui for the settings ...
The lates version of the software for the 3ware 9xxx series can be grabbed here.
Just download and unzip and install ...

unzip 3DM2...
chmod u+x install.sh
./install.sh -i

answer following questions with
y
0
<ENTER>
<ENTER>

and it's done ...

web-interface is running on https://<your-server>: 888
default-login: admin
default-pass: 3ware

After login switch to "Management" --> "Controller Settings" and change "StorSave" in the Unit Policies from "Balance" to "Performance".

But don't forget: Without a BBU or a stable USV you can loose data in case of a power outage.

regards

Rico