Dell R920 VM Performance issues

T

tomstephens89

Renowned Member
Mar 10, 2014
202
9
83
Kingsclere, United Kingdom
Hi all,

Just spun up a couple of new Proxmox boxes, these are Dell R920's with 4 x Xeon E7 4880v2's 15 cores each, so 60 cores + hyperthreading. 512GB RAM in the boxes, local storage is a 22 disk RAID 10 of spinners. No SSD's yet, but still not slow.

I have created 2 linux VM's and a windows VM so far and the VM's don't feel very responsive at all. Slow in fact.

I left Hyperthreading ENABLED, I think I might disable it and see what that does in performance terms. I am wondering if such a high core count causes CPU scheduler issues when HT is on.

NUMA is ENABLED. Disabling/Enabling makes no difference.

Does anyone have similar experience when using high core count nodes?
 
Last edited:
Dell Rxxx (4U) with 4x Gold Xeons 12cores = 48 physical, 48 ht cores, ssd array. No performance hit.
 
tomstephens89 said:
Hi all,

Just spun up a couple of new Proxmox boxes, these are Dell R920's with 4 x Xeon E7 4880v2's 15 cores each, so 60 cores + hyperthreading. 512GB RAM in the boxes, local storage is a 22 disk RAID 10 of spinners. No SSD's yet, but still not slow.

I have created 2 linux VM's and a windows VM so far and the VM's don't feel very responsive at all. Slow in fact.

I left Hyperthreading ENABLED, I think I might disable it and see what that does in performance terms. I am wondering if such a high core count causes CPU scheduler issues when HT is on.

NUMA is ENABLED. Disabling/Enabling makes no difference.

Does anyone have similar experience when using high core count nodes?
Click to expand...

How many cores are you passing through to VM? What CPU type you selected?
 
I have disabled hyperthreading on another identical box thats still in the dev lab and performance on VM's still seems sluggish.

Heres a pveperf output with an lvm disk mounted at /mnt/test

root@proxmox01:~# pveperf /mnt/test
CPU BOGOMIPS: 598857.60
REGEX/SECOND: 1511571
HD SIZE: 1967.61 GB (/dev/mapper/pve-vm--100--disk--0)
BUFFERED READS: 623.98 MB/sec
AVERAGE SEEK TIME: 34.23 ms
FSYNCS/SECOND: 1438.72
DNS EXT: 39.53 ms
DNS INT: 35.05 ms

The local disk set up is 22 1TB 7.2K SAS disks in RAID10 with 2 as hot spares. Stripe size is 128K, controller is a Dell H730P with 2GB Cache.
 
Heres a fio sequential read:

root@proxmox01:/mnt/test# fio --name=seqread --rw=read --ioengine=libaio --bs=8k --numjobs=1 --size=8G --runtime=600 --group_reporting
seqread: (g=0): rw=read, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=981MiB/s][r=126k IOPS][eta 00m:00s]
seqread: (groupid=0, jobs=1): err= 0: pid=115266: Thu Jan 30 13:27:28 2020
read: IOPS=107k, BW=836MiB/s (876MB/s)(8192MiB/9802msec)
slat (nsec): min=1498, max=17983k, avg=6553.16, stdev=40837.90
clat (nsec): min=660, max=258654, avg=805.92, stdev=590.84
lat (usec): min=2, max=17988, avg= 7.44, stdev=40.96
clat percentiles (nsec):
| 1.00th=[ 692], 5.00th=[ 700], 10.00th=[ 708], 20.00th=[ 716],
| 30.00th=[ 716], 40.00th=[ 724], 50.00th=[ 732], 60.00th=[ 740],
| 70.00th=[ 756], 80.00th=[ 772], 90.00th=[ 1012], 95.00th=[ 1176],
| 99.00th=[ 1528], 99.50th=[ 1608], 99.90th=[ 8160], 99.95th=[14912],
| 99.99th=[19072]
bw ( KiB/s): min=933312, max=1035776, per=100.00%, avg=1010940.00, stdev=26304.89, samples=16
iops : min=116664, max=129472, avg=126367.50, stdev=3288.11, samples=16
lat (nsec) : 750=67.38%, 1000=22.15%
lat (usec) : 2=10.32%, 4=0.03%, 10=0.02%, 20=0.09%, 50=0.01%
lat (usec) : 100=0.01%, 500=0.01%
cpu : usr=16.21%, sys=82.80%, ctx=169, majf=0, minf=18
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
READ: bw=836MiB/s (876MB/s), 836MiB/s-836MiB/s (876MB/s-876MB/s), io=8192MiB (8590MB), run=9802-9802msec

Disk stats (read/write):
dm-6: ios=160786/4, merge=0/0, ticks=18192/0, in_queue=18192, util=82.98%, aggrios=163842/58, aggrmerge=0/0, aggrticks=17316/0, aggrin_queue=17316, aggrutil=82.76%
dm-4: ios=163842/58, merge=0/0, ticks=17316/0, in_queue=17316, util=82.76%, aggrios=81921/29, aggrmerge=0/0, aggrticks=8616/0, aggrin_queue=8616, aggrutil=82.76%
dm-2: ios=1/0, merge=0/0, ticks=28/0, in_queue=28, util=0.08%, aggrios=33497/70, aggrmerge=130355/10, aggrticks=4324/22, aggrin_queue=152, aggrutil=82.77%
sda: ios=33497/70, merge=130355/10, ticks=4324/22, in_queue=152, util=82.77%
dm-3: ios=163842/58, merge=0/0, ticks=17204/0, in_queue=17204, util=82.76%
Click to expand...

And heres a fio sequential write:

root@proxmox01:/mnt/test# fio --name=seqread --rw=write --ioengine=libaio --bs=8k --numjobs=1 --size=8G --runtime=600 --group_reporting
seqread: (g=0): rw=write, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=941MiB/s][w=120k IOPS][eta 00m:00s]
seqread: (groupid=0, jobs=1): err= 0: pid=115508: Thu Jan 30 13:28:18 2020
write: IOPS=91.5k, BW=715MiB/s (750MB/s)(8192MiB/11454msec); 0 zone resets
slat (usec): min=5, max=514, avg= 8.44, stdev= 4.65
clat (nsec): min=724, max=53758, avg=860.79, stdev=432.11
lat (usec): min=6, max=570, avg= 9.44, stdev= 4.83
clat percentiles (nsec):
| 1.00th=[ 748], 5.00th=[ 756], 10.00th=[ 764], 20.00th=[ 772],
| 30.00th=[ 780], 40.00th=[ 788], 50.00th=[ 788], 60.00th=[ 796],
| 70.00th=[ 804], 80.00th=[ 820], 90.00th=[ 844], 95.00th=[ 1080],
| 99.00th=[ 2064], 99.50th=[ 2128], 99.90th=[ 2224], 99.95th=[ 2640],
| 99.99th=[15168]
bw ( KiB/s): min=110992, max=976752, per=100.00%, avg=757565.43, stdev=220784.79, samples=21
iops : min=13874, max=122094, avg=94695.67, stdev=27598.16, samples=21
lat (nsec) : 750=0.96%, 1000=93.96%
lat (usec) : 2=1.96%, 4=3.07%, 10=0.01%, 20=0.03%, 50=0.01%
lat (usec) : 100=0.01%
cpu : usr=11.42%, sys=88.51%, ctx=79, majf=0, minf=56
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1048576,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
WRITE: bw=715MiB/s (750MB/s), 715MiB/s-715MiB/s (750MB/s-750MB/s), io=8192MiB (8590MB), run=11454-11454msec

Disk stats (read/write):
dm-6: ios=0/9681, merge=0/0, ticks=0/6936, in_queue=6936, util=2.34%, aggrios=0/9706, aggrmerge=0/0, aggrticks=0/6696, aggrin_queue=6696, aggrutil=2.73%
dm-4: ios=0/9706, merge=0/0, ticks=0/6696, in_queue=6696, util=2.73%, aggrios=0/4853, aggrmerge=0/0, aggrticks=0/3340, aggrin_queue=3340, aggrutil=2.73%
dm-2: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=9/9658, aggrmerge=0/55, aggrticks=16/6801, aggrin_queue=2924, aggrutil=2.97%
sda: ios=9/9658, merge=0/55, ticks=16/6801, in_queue=2924, util=2.97%
dm-3: ios=0/9706, merge=0/0, ticks=0/6680, in_queue=6680, util=2.73%
Click to expand...
 
Last edited:
Random read, looks low to me, but my previous proxmox installs performed just fine and they used a SAN with spinners over a network. No performance issues with those VM's.

root@proxmox01:/mnt/test# fio --name=randread --rw=randread --ioengine=libaio --bs=8k --numjobs=1 --size=1G --runtime=600 --group_reporting
randread: (g=0): rw=randread, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=1401KiB/s][r=175 IOPS][eta 00m:00s]
randread: (groupid=0, jobs=1): err= 0: pid=116855: Thu Jan 30 13:43:13 2020
read: IOPS=152, BW=1220KiB/s (1249kB/s)(715MiB/600002msec)
slat (usec): min=73, max=158310, avg=6540.38, stdev=7871.30
clat (nsec): min=1436, max=321584, avg=6104.69, stdev=2001.03
lat (usec): min=75, max=158318, avg=6548.21, stdev=7871.65
clat percentiles (nsec):
| 1.00th=[ 3792], 5.00th=[ 4080], 10.00th=[ 4320], 20.00th=[ 6112],
| 30.00th=[ 6176], 40.00th=[ 6176], 50.00th=[ 6240], 60.00th=[ 6240],
| 70.00th=[ 6240], 80.00th=[ 6304], 90.00th=[ 6560], 95.00th=[ 7072],
| 99.00th=[10048], 99.50th=[11456], 99.90th=[29824], 99.95th=[38144],
| 99.99th=[54528]
bw ( KiB/s): min= 192, max= 1920, per=100.00%, avg=1220.04, stdev=184.90, samples=1200
iops : min= 24, max= 240, avg=152.48, stdev=23.12, samples=1200
lat (usec) : 2=0.08%, 4=3.53%, 10=95.38%, 20=0.83%, 50=0.16%
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%
cpu : usr=0.32%, sys=1.72%, ctx=91655, majf=0, minf=53
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=91515,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
READ: bw=1220KiB/s (1249kB/s), 1220KiB/s-1220KiB/s (1249kB/s-1249kB/s), io=715MiB (750MB), run=600002-600002msec

Disk stats (read/write):
dm-6: ios=91493/0, merge=0/0, ticks=591084/0, in_queue=591084, util=42.34%, aggrios=91516/1608, aggrmerge=0/0, aggrticks=588704/516, aggrin_queue=589220, aggrutil=42.79%
dm-4: ios=91516/1608, merge=0/0, ticks=588704/516, in_queue=589220, util=42.79%, aggrios=45780/804, aggrmerge=0/0, aggrticks=294552/246, aggrin_queue=294798, aggrutil=42.81%
dm-2: ios=45/0, merge=0/0, ticks=588/0, in_queue=588, util=0.04%, aggrios=92042/1978, aggrmerge=0/491, aggrticks=594022/357, aggrin_queue=458056, aggrutil=42.92%
sda: ios=92042/1978, merge=0/491, ticks=594022/357, in_queue=458056, util=42.92%
dm-3: ios=91516/1608, merge=0/0, ticks=588516/492, in_queue=589008, util=42.81%
Click to expand...

Random write:

root@proxmox01:/mnt/test# fio --name=randwrite --rw=randwrite --ioengine=libaio --bs=8k --numjobs=1 --size=4G --runtime=600 --group_reporting
randwrite: (g=0): rw=randwrite, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
randwrite: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [w(1)][87.5%][w=786MiB/s][w=101k IOPS][eta 00m:01s]
randwrite: (groupid=0, jobs=1): err= 0: pid=120022: Thu Jan 30 13:54:01 2020
write: IOPS=75.5k, BW=590MiB/s (618MB/s)(4096MiB/6947msec); 0 zone resets
slat (usec): min=5, max=534, avg=10.98, stdev= 7.75
clat (nsec): min=722, max=59669, avg=1101.73, stdev=740.19
lat (usec): min=6, max=565, avg=12.27, stdev= 8.15
clat percentiles (nsec):
| 1.00th=[ 748], 5.00th=[ 756], 10.00th=[ 764], 20.00th=[ 772],
| 30.00th=[ 780], 40.00th=[ 788], 50.00th=[ 796], 60.00th=[ 812],
| 70.00th=[ 844], 80.00th=[ 1960], 90.00th=[ 2008], 95.00th=[ 2064],
| 99.00th=[ 2160], 99.50th=[ 2192], 99.90th=[ 7456], 99.95th=[14784],
| 99.99th=[19072]
bw ( KiB/s): min=351632, max=825328, per=97.48%, avg=588513.23, stdev=178138.31, samples=13
iops : min=43954, max=103166, avg=73564.31, stdev=22267.27, samples=13
lat (nsec) : 750=2.11%, 1000=73.01%
lat (usec) : 2=12.23%, 4=12.54%, 10=0.01%, 20=0.09%, 50=0.01%
lat (usec) : 100=0.01%
cpu : usr=8.51%, sys=91.43%, ctx=14, majf=0, minf=84
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,524288,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
WRITE: bw=590MiB/s (618MB/s), 590MiB/s-590MiB/s (618MB/s-618MB/s), io=4096MiB (4295MB), run=6947-6947msec

Disk stats (read/write):
dm-6: ios=0/56, merge=0/0, ticks=0/0, in_queue=0, util=0.12%, aggrios=0/69, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.51%
dm-4: ios=0/69, merge=0/0, ticks=0/0, in_queue=0, util=0.51%, aggrios=0/34, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.51%
dm-2: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=8/23, aggrmerge=0/51, aggrticks=4/3, aggrin_queue=0, aggrutil=0.85%
sda: ios=8/23, merge=0/51, ticks=4/3, in_queue=0, util=0.85%
dm-3: ios=0/69, merge=0/0, ticks=0/0, in_queue=0, util=0.51%
Click to expand...
 
You must log in or register to reply here.
Top Bottom