Looking for better performance understanding

[jompsi]

Hi all,

At the moment I am testing Active Directory in a test lab, which is one physical server running proxmox.

Hardware:
Intel Core i7-4790K, 4 GHz, 8 Cores
32 GB of RAM
Proxmox is installed on a SSD
There are two Seagate 2000 GB HDD in RAID1. Here are the VMs stored

Proxmox:
pve-manager/3.4-11/6502936f (running kernel: 2.6.32-43-pve)

What I now see, is that sometimes if two VMs are running and I turn on a third, it can take up to 30 minutes till the VM is up.

The VMs all have one CPU socket with 2 cores, 4 GB RAM, a 50 GB qcow2 disk and the os is windows 7 or windows server 2012 r2.

Everything runs really slow and lacky. For example if three windows server 2012 r2 as domain controller run and one client with windows 7, with the specs from above. I am not able click in the vms normally. It is alway delayed.

What could be the reason? I know my server is not for production but I would say it isnt that bad for testing.

How would you benchmark test this?

I am looking that no more than 8 virtual cpus are used, because the hypervisor only has 8 cores. Or is this not necessary?

I just dont have the overview of how the performance is distributed to the vms and if I can give the vms more cpus then there are physical available. And how does ist look with the memory. If I have 32 GB of ram I wouldnt deploy more than 30 GB of ram to the vms.

This is a question that is in my head for a long time and now that I have experienced again bad performance in the vms with windows, most likely because of me not knowing how to do it right, I hope to get this cleared here in the forum.

I have read the three books "Mastering Proxmox", "Proxmox High Availability" and "Proxmox Cookbook" from PacktPub and searched in the internet, but I was not able to answer this question by myself.

Really hoping to get some help here and kind regards
jompsi

 

[udo]

Hi,
I guess that's an IO-Problem.

You wrote, that your VMs on an raid1. What kind of raid-1 is it?
How looks pveperf on this raid?

If you look with top (or better atop) how looks wait?
Does the same happens with an raw-vm-disk instead of qcow2?

About cpu-cores: you can overprovide your 8 cores to different VMs - but no VM can take more than 8 core!

Udo

 

[jompsi]

Hello Udo

I just checked my test lab computer and saw, that there is no raid configured for the HDDs. I just use only one of the two HDDs at the moment.

pveperf
CPU BOGOMIPS:      63850.88
REGEX/SECOND:      3134022
HD SIZE:           27.31 GB (/dev/mapper/pve-root)
BUFFERED READS:    405.56 MB/sec
AVERAGE SEEK TIME: 0.14 ms
FSYNCS/SECOND:     6542.50
DNS EXT:           69.85 ms
DNS INT:           72.59 ms (home)

and the result for the disk and partition, where the VMs are stored:

pveperf /dev/sdb9
CPU BOGOMIPS:      63850.88
REGEX/SECOND:      3173023
HD SIZE:           857.35 GB (/dev/sdb9)
BUFFERED READS:    121.63 MB/sec
AVERAGE SEEK TIME: 62.98 ms
open failed at /usr/bin/pveperf line 83.

And atop outputs the following, when two VMs are starting:

ATOP - evora                                                      2016/01/13  15:05:01                                                      ------                                                       10s elapsed
PRC | sys    9.31s  | user  10.91s  |               |               | #proc    284  | #trun      3  | #tslpi   304 |  #tslpu     0 |  #zombie    0 |               |  clones    16 |               |  #exit     16 |
CPU | sys      85%  | user    100%  | irq       0%  |               |               | idle    548%  | wait      1% |               |  steal     0% |               |  guest    66% |  curf 3.99GHz |  curscal   ?% |
  cpu | sys      34%  | user     39%  | irq       0%  |               |               | idle      0%  | cpu001 w  0% |               |  steal     0% |               |  guest    26% |  curf 3.99GHz |  curscal   ?% |
  cpu | sys      33%  | user     40%  | irq       0%  |               |               | idle      0%  | cpu003 w  0% |               |  steal     0% |               |  guest    26% |  curf 3.99GHz |  curscal   ?% |
cpu | sys       0%  | user      1%  | irq       0%  |               |               | idle     99%  | cpu000 w  0% |               |  steal     0% |               |  guest     0% |  curf 3.99GHz |  curscal   ?% |
cpu | sys       0%  | user      0%  | irq       0%  |               |               | idle     98%  | cpu004 w  2% |               |  steal     0% |               |  guest     0% |  curf 3.99GHz |  curscal   ?% |
cpu | sys       0%  | user      0%  | irq       0%  |               |               | idle     99%  | cpu002 w  0% |               |  steal     0% |               |  guest     0% |  curf 3.99GHz |  curscal   ?% |
cpu | sys       0%  | user      0%  | irq       0%  |               |               | idle    100%  | cpu006 w  0% |               |  steal     0% |               |  guest     0% |  curf 3.99GHz |  curscal   ?% |
cpu | sys       0%  | user      0%  | irq       0%  |               |               | idle    100%  | cpu005 w  0% |               |  steal     0% |               |  guest     0% |  curf 3.99GHz |  curscal   ?% |
CPL | avg1    1.04  |               | avg5    0.56  | avg15   0.43  |               |               |              |  csw    36927 |               |  intr   36130 |               |               |  numcpu     8 |
MEM | tot    31.1G  | free   29.8G  |               | cache 199.2M  | dirty   0.1M  | buff   94.6M  |              |  slab  410.3M |               |               |               |               |               |
SWP | tot    13.9G  | free   13.9G  |               |               |               |               |              |               |               |               |               |  vmcom   9.4G |  vmlim  29.4G |
LVM |     pve-root  | busy      1%  |               | read       0  | write      9  | KiB/r      0  |              |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.00 |               |  avq     2.01 |  avio 17.6 ms |
  DSK |          sdb  | busy     92%  |               | read    4356  | write      0  | KiB/r      0  |              |  KiB/w      0 |  MBr/s   0.23 |  MBw/s   0.00 |               |  avq     1.98 |  avio 2.30 ms |
DSK |          sda  | busy      1%  |               | read       0  | write      4  | KiB/r      0  |              |  KiB/w      9 |  MBr/s   0.00 |  MBw/s   0.00 |               |  avq     1.98 |  avio 39.5 ms |
NET | transport     | tcpi     271  | tcpo     254  | udpi       3  | udpo       3  | tcpao      3  |              |  tcppo      3 |  tcprs      0 |  tcpie      0 |  tcpor      0 |  udpnp      0 |  udpip      0 |
NET | network       | ipi      296  |               | ipo      257  | ipfrw      0  | deliv    289  |              |               |               |               |               |  icmpi      0 |  icmpo      0 |
NET | eth0      0%  | pcki     169  | pcko      83  |               | si   28 Kbps  | so   18 Kbps  | coll       0 |  mlti      33 |  erri       0 |               |  erro       0 |  drpi       0 |  drpo       0 |
NET | lo      ----  | pcki     176  | pcko     176  |               | si   24 Kbps  | so   24 Kbps  | coll       0 |  mlti       0 |  erri       0 |               |  erro       0 |  drpi       0 |  drpo       0 |
NET | vmbr0   ----  | pcki     162  | pcko      82  |               | si   26 Kbps  | so   18 Kbps  | coll       0 |  mlti       0 |  erri       0 |               |  erro       0 |  drpi       0 |  drpo       0 |

The three indented lines are marked as red in atop. I am waiting for 15 minutes now and they are not up yet.

What do you think?

Thanks a lot and regards
jompsi

 

[jompsi]

EDIT: I found out which HDD is built in. It is a Seagate ST2000VN000. When I bought it, clearly didnt look at performance and till then i never made bad experience. What kind of HDD should be used in virtualization envirnments? My mainboard is AsRock H97 Pro4 which supports Sata 3.

 

[1nerdyguy]

In a perfect world, you'd use all SSD for your storage, in a RAID10 Array or such. However, that wont work if you have large files.

Personally, I use SAS drives in a ZFS Mirrored VDEV setup hooked up over 10GB fiber on NFS. It works wonderfully.

 

[udo]

Hello Udo

I just checked my test lab computer and saw, that there is no raid configured for the HDDs. I just use only one of the two HDDs at the moment.

pveperf
CPU BOGOMIPS:      63850.88
REGEX/SECOND:      3134022
HD SIZE:           27.31 GB (/dev/mapper/pve-root)
BUFFERED READS:    405.56 MB/sec
AVERAGE SEEK TIME: 0.14 ms
FSYNCS/SECOND:     6542.50
DNS EXT:           69.85 ms
DNS INT:           72.59 ms (home)

and the result for the disk and partition, where the VMs are stored:

pveperf /dev/sdb9
CPU BOGOMIPS:      63850.88
REGEX/SECOND:      3173023
HD SIZE:           857.35 GB (/dev/sdb9)
BUFFERED READS:    121.63 MB/sec
AVERAGE SEEK TIME: 62.98 ms
open failed at /usr/bin/pveperf line 83.

Hi,
you don't write which kind of raid-1 do you use. Hardware?
In this case your VM-Files (qcow2) must be on a filesystem, so you can do an pveperf on the mountpoint?!

And atop outputs the following, when two VMs are starting:

ATOP - evora                                                      2016/01/13  15:05:01                                                      ------                                                       10s elapsed
PRC | sys    9.31s  | user  10.91s  |               |               | #proc    284  | #trun      3  | #tslpi   304 |  #tslpu     0 |  #zombie    0 |               |  clones    16 |               |  #exit     16 |
CPU | sys      85%  | user    100%  | irq       0%  |               |               | idle    548%  | wait      1% |               |  steal     0% |               |  guest    66% |  curf 3.99GHz |  curscal   ?% |
  cpu | sys      34%  | user     39%  | irq       0%  |               |               | idle      0%  | cpu001 w  0% |               |  steal     0% |               |  guest    26% |  curf 3.99GHz |  curscal   ?% |
  cpu | sys      33%  | user     40%  | irq       0%  |               |               | idle      0%  | cpu003 w  0% |               |  steal     0% |               |  guest    26% |  curf 3.99GHz |  curscal   ?% |
cpu | sys       0%  | user      1%  | irq       0%  |               |               | idle     99%  | cpu000 w  0% |               |  steal     0% |               |  guest     0% |  curf 3.99GHz |  curscal   ?% |
cpu | sys       0%  | user      0%  | irq       0%  |               |               | idle     98%  | cpu004 w  2% |               |  steal     0% |               |  guest     0% |  curf 3.99GHz |  curscal   ?% |
cpu | sys       0%  | user      0%  | irq       0%  |               |               | idle     99%  | cpu002 w  0% |               |  steal     0% |               |  guest     0% |  curf 3.99GHz |  curscal   ?% |
cpu | sys       0%  | user      0%  | irq       0%  |               |               | idle    100%  | cpu006 w  0% |               |  steal     0% |               |  guest     0% |  curf 3.99GHz |  curscal   ?% |
cpu | sys       0%  | user      0%  | irq       0%  |               |               | idle    100%  | cpu005 w  0% |               |  steal     0% |               |  guest     0% |  curf 3.99GHz |  curscal   ?% |
CPL | avg1    1.04  |               | avg5    0.56  | avg15   0.43  |               |               |              |  csw    36927 |               |  intr   36130 |               |               |  numcpu     8 |
MEM | tot    31.1G  | free   29.8G  |               | cache 199.2M  | dirty   0.1M  | buff   94.6M  |              |  slab  410.3M |               |               |               |               |               |
SWP | tot    13.9G  | free   13.9G  |               |               |               |               |              |               |               |               |               |  vmcom   9.4G |  vmlim  29.4G |
LVM |     pve-root  | busy      1%  |               | read       0  | write      9  | KiB/r      0  |              |  KiB/w      4 |  MBr/s   0.00 |  MBw/s   0.00 |               |  avq     2.01 |  avio 17.6 ms |
  DSK |          sdb  | busy     92%  |               | read    4356  | write      0  | KiB/r      0  |              |  KiB/w      0 |  MBr/s   0.23 |  MBw/s   0.00 |               |  avq     1.98 |  avio 2.30 ms |
DSK |          sda  | busy      1%  |               | read       0  | write      4  | KiB/r      0  |              |  KiB/w      9 |  MBr/s   0.00 |  MBw/s   0.00 |               |  avq     1.98 |  avio 39.5 ms |
NET | transport     | tcpi     271  | tcpo     254  | udpi       3  | udpo       3  | tcpao      3  |              |  tcppo      3 |  tcprs      0 |  tcpie      0 |  tcpor      0 |  udpnp      0 |  udpip      0 |
NET | network       | ipi      296  |               | ipo      257  | ipfrw      0  | deliv    289  |              |               |               |               |               |  icmpi      0 |  icmpo      0 |
NET | eth0      0%  | pcki     169  | pcko      83  |               | si   28 Kbps  | so   18 Kbps  | coll       0 |  mlti      33 |  erri       0 |               |  erro       0 |  drpi       0 |  drpo       0 |
NET | lo      ----  | pcki     176  | pcko     176  |               | si   24 Kbps  | so   24 Kbps  | coll       0 |  mlti       0 |  erri       0 |               |  erro       0 |  drpi       0 |  drpo       0 |
NET | vmbr0   ----  | pcki     162  | pcko      82  |               | si   26 Kbps  | so   18 Kbps  | coll       0 |  mlti       0 |  erri       0 |               |  erro       0 |  drpi       0 |  drpo       0 |

The three indented lines are marked as red in atop. I am waiting for 15 minutes now and they are not up yet.

What do you think?

Thanks a lot and regards
jompsi

sdb is not fast enough...

 

[udo]

EDIT: I found out which HDD is built in. It is a Seagate ST2000VN000. When I bought it, clearly didnt look at performance and till then i never made bad experience. What kind of HDD should be used in virtualization envirnments? My mainboard is AsRock H97 Pro4 which supports Sata 3.

Hi,
depends on your workload. I have made good experiences with fast an areca raid-controller. On the raid-controller I use hdds in raid-10 (sata for normal usages and sas for faster IO).
I have tried three time zfs, but the performance satesfied me not...

Udo

 

[Q-wulf]

I like to use ceph for this.

In a perfect world, you'd use all SSD for your storage, in a RAID10 Array or such

I like to use ceph for this. the more nodes, the more performance. split SSD/nvme/HDD into their own type of OSD's, then run the vDisks on the appropriate type of Pool (replicated SSD / NVME / HDD, or erasure coded HDD Pool with SSD/NVME cache), depending on your requirements (performance/size)

works at home/Lab and production/work quite well.

 

[jompsi]

Hi all

First of all, thanks for all your inputs.

@udo

you don't write which kind of raid-1 do you use. Hardware?

I was mistaken. I do not use a raid 1. So nothing is configured.

In this case your VM-Files (qcow2) must be on a filesystem, so you can do an pveperf on the mountpoint?!

I did the pveperf on sdb9. It is in my second post the second code section.

sata for normal usages and sas for faster IO

What kind of sata disk do you use? It is in my case just for a testlab. And is this local storage? If yes, how do you handel it? With ceph, lvm or like me just mount to a directory?

sdb is not fast enough...

Which value does indicate this to you and what are acceptable values? So I can see in the future by my self what the problem is

@Q-wulf

I like to use ceph for this

Do you use it on local storage? And like this: https://pve.proxmox.com/wiki/Ceph_Server or do you have another link, which you suggest?

Best regards
jompsi

 

[Q-wulf]

I use

Do you use it on local storage? And like this: https://pve.proxmox.com/wiki/Ceph_Server or do you have another link, which you suggest?

I use it at home mostly in single-node configurations. 6-8 SSD's with replicated pools with 2/1 or 3/1 scheme. Basically what ever i can get my hands on cheap (GB/currency). When i need some more Storage space (happens rarely and only for some testing purpose) i plug some HDD-Disks as a Erasure coded pool in there. and use the replicated ssd pool as Cache.

I do have a small 3-node test cluster (2 SSD+2 HDD) and a single node "storage cluster" at home with 8 SSD and 24 HDD's for all my home storage needs, that does pretty well with separated Spinners and Flash osd's. Cache-Tiering is really awesome

At work we operate multiple clusters with nvme based Cachepool's and couple hundred to thousand Spinner based OSD's per cluster.


I mostly enjoy the flexibility of ceph. great performance (when you use cache-tiers and osd separation by disk type) and also great utilisation of the available raw storage by using Erasure-coding. On top of that the great customizability with regards to failure domains is what gets me to use this in production most.

these links here:
http://docs.ceph.com/docs/master/rados/
http://docs.ceph.com/docs/master/architecture/

together with this blog:
http://www.sebastien-han.fr/blog/categories/ceph/

are pretty much all you need on top of the link you already posted.

(I admit it is a bit over the top for home use and only a couple of disks at work environment, due to cpu and ram overhead for ceph compared to e.g. software raid-10 . Scale is where ceph shines.

 

[udo]

Hi all

First of all, thanks for all your inputs.

@udo

I was mistaken. I do not use a raid 1. So nothing is configured.


I did the pveperf on sdb9. It is in my second post the second code section.

yes but sdb9 is the blockdevice and I ask for pveperf on the mountpoint to get the fsyncs info!
And why partion 9? Are the othe partion in use by other processes? Perhaps this slow down the disk?

What kind of sata disk do you use? It is in my case just for a testlab. And is this local storage? If yes, how do you handel it? With ceph, lvm or like me just mount to a directory?

I use normal enterprise HDDs - mostly HDS - like the 4TB one. The speed came from raid-10 in conjunction with the write-cache from the raid-controller.

The storage is splitted in different raid-sets (depends on usage): local-storage, drbd with lvm on top.

I use external storage too - an ceph-cluster, FC-Raids and iScsi-Raids.

Which value does indicate this to you and what are acceptable values? So I can see in the future by my self what the problem is

The disk is busy, but the output looks much to less (no real data througput and also not too much IOPS). Perhaps something wrong with the disk, or sata-settings? IDE/AHCPI...

 

[spirit]

@jompsi :

do you use virtio disks and nics ?

can you try pve-kernel-3.10 ? (if you don't use openvz)

 

[jompsi]

Hello

@udo
The disk has multiple partition:

fdisk -l /dev/sdb

Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1              63  3907029167  1953514552+   5  Extended
Partition 1 does not start on physical sector boundary.
/dev/sdb5             126   107426654    53713264+  83  Linux
Partition 5 does not start on physical sector boundary.
/dev/sdb6       107426718   214853309    53713296   83  Linux
Partition 6 does not start on physical sector boundary.
/dev/sdb7       214853373   322279964    53713296   83  Linux
Partition 7 does not start on physical sector boundary.
/dev/sdb8       322280028  2080096199   878908086   83  Linux
Partition 8 does not start on physical sector boundary.
/dev/sdb9      2080096263  3907029167   913466452+  83  Linux
Partition 9 does not start on physical sector boundary.

And this partitions are all mounted.

/dev/sdb5 on /mnt/bkp_clean_sdb5 type ext4 (rw,relatime,barrier=1,data=ordered)
/dev/sdb6 on /mnt/bkp_normal_sdb6 type ext4 (rw,relatime,barrier=1,data=ordered)
/dev/sdb7 on /mnt/bkp_temp_sdb7 type ext4 (rw,relatime,barrier=1,data=ordered)
/dev/sdb8 on /mnt/openvz_sdb8 type ext4 (rw,relatime,barrier=1,data=ordered)
/dev/sdb9 on /mnt/kvm_sdb9 type ext4 (rw,relatime,barrier=1,data=ordered)

The pveperf of the mount point is.

pveperf /mnt/kvm_sdb9
CPU BOGOMIPS:      63855.76
REGEX/SECOND:      3174180
HD SIZE:           857.35 GB (/dev/sdb9)
BUFFERED READS:    18.43 MB/sec
AVERAGE SEEK TIME: 25.43 ms
FSYNCS/SECOND:     9.30
DNS EXT:           67.21 ms
DNS INT:           59.29 ms (home)

Sorry I thought I had to do it on the partition.

@spirit
No, in this lab I am not using the virtio drivers. And I really think I use bad HDDs or/and the configuration is not ideal.

Yes, there is an OpenVZ, so I am not able to test kernel 3.10.

Thanks a lot
jompsi

 

[jompsi]

Hi all,

I have made my harddisk to a LVM like this https://pve.proxmox.com/wiki/Storage_Model#LVM_Groups_with_Local_Backing and my VMs feel faster now. At the moment I dont have time to make performance tests between directory storage and LVM, but it feels faster and I will do it when I have time again. Any tipps on how to do performance tests?

Thanks a lot and kind regards
jompsi