Dell R920 VM Performance issues

tomstephens89

Renowned Member
Mar 10, 2014
177
6
83
Kingsclere, United Kingdom
Hi all,

Just spun up a couple of new Proxmox boxes, these are Dell R920's with 4 x Xeon E7 4880v2's 15 cores each, so 60 cores + hyperthreading. 512GB RAM in the boxes, local storage is a 22 disk RAID 10 of spinners. No SSD's yet, but still not slow.

I have created 2 linux VM's and a windows VM so far and the VM's don't feel very responsive at all. Slow in fact.

I left Hyperthreading ENABLED, I think I might disable it and see what that does in performance terms. I am wondering if such a high core count causes CPU scheduler issues when HT is on.

NUMA is ENABLED. Disabling/Enabling makes no difference.

Does anyone have similar experience when using high core count nodes?
 
Last edited:
Dell Rxxx (4U) with 4x Gold Xeons 12cores = 48 physical, 48 ht cores, ssd array. No performance hit.
 
Hi all,

Just spun up a couple of new Proxmox boxes, these are Dell R920's with 4 x Xeon E7 4880v2's 15 cores each, so 60 cores + hyperthreading. 512GB RAM in the boxes, local storage is a 22 disk RAID 10 of spinners. No SSD's yet, but still not slow.

I have created 2 linux VM's and a windows VM so far and the VM's don't feel very responsive at all. Slow in fact.

I left Hyperthreading ENABLED, I think I might disable it and see what that does in performance terms. I am wondering if such a high core count causes CPU scheduler issues when HT is on.

NUMA is ENABLED. Disabling/Enabling makes no difference.

Does anyone have similar experience when using high core count nodes?

How many cores are you passing through to VM? What CPU type you selected?
 
I have disabled hyperthreading on another identical box thats still in the dev lab and performance on VM's still seems sluggish.

Heres a pveperf output with an lvm disk mounted at /mnt/test

root@proxmox01:~# pveperf /mnt/test
CPU BOGOMIPS: 598857.60
REGEX/SECOND: 1511571
HD SIZE: 1967.61 GB (/dev/mapper/pve-vm--100--disk--0)
BUFFERED READS: 623.98 MB/sec
AVERAGE SEEK TIME: 34.23 ms
FSYNCS/SECOND: 1438.72
DNS EXT: 39.53 ms
DNS INT: 35.05 ms

The local disk set up is 22 1TB 7.2K SAS disks in RAID10 with 2 as hot spares. Stripe size is 128K, controller is a Dell H730P with 2GB Cache.
 
Heres a fio sequential read:

root@proxmox01:/mnt/test# fio --name=seqread --rw=read --ioengine=libaio --bs=8k --numjobs=1 --size=8G --runtime=600 --group_reporting
seqread: (g=0): rw=read, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=981MiB/s][r=126k IOPS][eta 00m:00s]
seqread: (groupid=0, jobs=1): err= 0: pid=115266: Thu Jan 30 13:27:28 2020
read: IOPS=107k, BW=836MiB/s (876MB/s)(8192MiB/9802msec)
slat (nsec): min=1498, max=17983k, avg=6553.16, stdev=40837.90
clat (nsec): min=660, max=258654, avg=805.92, stdev=590.84
lat (usec): min=2, max=17988, avg= 7.44, stdev=40.96
clat percentiles (nsec):
| 1.00th=[ 692], 5.00th=[ 700], 10.00th=[ 708], 20.00th=[ 716],
| 30.00th=[ 716], 40.00th=[ 724], 50.00th=[ 732], 60.00th=[ 740],
| 70.00th=[ 756], 80.00th=[ 772], 90.00th=[ 1012], 95.00th=[ 1176],
| 99.00th=[ 1528], 99.50th=[ 1608], 99.90th=[ 8160], 99.95th=[14912],
| 99.99th=[19072]
bw ( KiB/s): min=933312, max=1035776, per=100.00%, avg=1010940.00, stdev=26304.89, samples=16
iops : min=116664, max=129472, avg=126367.50, stdev=3288.11, samples=16
lat (nsec) : 750=67.38%, 1000=22.15%
lat (usec) : 2=10.32%, 4=0.03%, 10=0.02%, 20=0.09%, 50=0.01%
lat (usec) : 100=0.01%, 500=0.01%
cpu : usr=16.21%, sys=82.80%, ctx=169, majf=0, minf=18
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
READ: bw=836MiB/s (876MB/s), 836MiB/s-836MiB/s (876MB/s-876MB/s), io=8192MiB (8590MB), run=9802-9802msec

Disk stats (read/write):
dm-6: ios=160786/4, merge=0/0, ticks=18192/0, in_queue=18192, util=82.98%, aggrios=163842/58, aggrmerge=0/0, aggrticks=17316/0, aggrin_queue=17316, aggrutil=82.76%
dm-4: ios=163842/58, merge=0/0, ticks=17316/0, in_queue=17316, util=82.76%, aggrios=81921/29, aggrmerge=0/0, aggrticks=8616/0, aggrin_queue=8616, aggrutil=82.76%
dm-2: ios=1/0, merge=0/0, ticks=28/0, in_queue=28, util=0.08%, aggrios=33497/70, aggrmerge=130355/10, aggrticks=4324/22, aggrin_queue=152, aggrutil=82.77%
sda: ios=33497/70, merge=130355/10, ticks=4324/22, in_queue=152, util=82.77%
dm-3: ios=163842/58, merge=0/0, ticks=17204/0, in_queue=17204, util=82.76%

And heres a fio sequential write:

root@proxmox01:/mnt/test# fio --name=seqread --rw=write --ioengine=libaio --bs=8k --numjobs=1 --size=8G --runtime=600 --group_reporting
seqread: (g=0): rw=write, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=941MiB/s][w=120k IOPS][eta 00m:00s]
seqread: (groupid=0, jobs=1): err= 0: pid=115508: Thu Jan 30 13:28:18 2020
write: IOPS=91.5k, BW=715MiB/s (750MB/s)(8192MiB/11454msec); 0 zone resets
slat (usec): min=5, max=514, avg= 8.44, stdev= 4.65
clat (nsec): min=724, max=53758, avg=860.79, stdev=432.11
lat (usec): min=6, max=570, avg= 9.44, stdev= 4.83
clat percentiles (nsec):
| 1.00th=[ 748], 5.00th=[ 756], 10.00th=[ 764], 20.00th=[ 772],
| 30.00th=[ 780], 40.00th=[ 788], 50.00th=[ 788], 60.00th=[ 796],
| 70.00th=[ 804], 80.00th=[ 820], 90.00th=[ 844], 95.00th=[ 1080],
| 99.00th=[ 2064], 99.50th=[ 2128], 99.90th=[ 2224], 99.95th=[ 2640],
| 99.99th=[15168]
bw ( KiB/s): min=110992, max=976752, per=100.00%, avg=757565.43, stdev=220784.79, samples=21
iops : min=13874, max=122094, avg=94695.67, stdev=27598.16, samples=21
lat (nsec) : 750=0.96%, 1000=93.96%
lat (usec) : 2=1.96%, 4=3.07%, 10=0.01%, 20=0.03%, 50=0.01%
lat (usec) : 100=0.01%
cpu : usr=11.42%, sys=88.51%, ctx=79, majf=0, minf=56
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1048576,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
WRITE: bw=715MiB/s (750MB/s), 715MiB/s-715MiB/s (750MB/s-750MB/s), io=8192MiB (8590MB), run=11454-11454msec

Disk stats (read/write):
dm-6: ios=0/9681, merge=0/0, ticks=0/6936, in_queue=6936, util=2.34%, aggrios=0/9706, aggrmerge=0/0, aggrticks=0/6696, aggrin_queue=6696, aggrutil=2.73%
dm-4: ios=0/9706, merge=0/0, ticks=0/6696, in_queue=6696, util=2.73%, aggrios=0/4853, aggrmerge=0/0, aggrticks=0/3340, aggrin_queue=3340, aggrutil=2.73%
dm-2: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=9/9658, aggrmerge=0/55, aggrticks=16/6801, aggrin_queue=2924, aggrutil=2.97%
sda: ios=9/9658, merge=0/55, ticks=16/6801, in_queue=2924, util=2.97%
dm-3: ios=0/9706, merge=0/0, ticks=0/6680, in_queue=6680, util=2.73%
 
Last edited:
Random read, looks low to me, but my previous proxmox installs performed just fine and they used a SAN with spinners over a network. No performance issues with those VM's.

root@proxmox01:/mnt/test# fio --name=randread --rw=randread --ioengine=libaio --bs=8k --numjobs=1 --size=1G --runtime=600 --group_reporting
randread: (g=0): rw=randread, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=1401KiB/s][r=175 IOPS][eta 00m:00s]
randread: (groupid=0, jobs=1): err= 0: pid=116855: Thu Jan 30 13:43:13 2020
read: IOPS=152, BW=1220KiB/s (1249kB/s)(715MiB/600002msec)
slat (usec): min=73, max=158310, avg=6540.38, stdev=7871.30
clat (nsec): min=1436, max=321584, avg=6104.69, stdev=2001.03
lat (usec): min=75, max=158318, avg=6548.21, stdev=7871.65
clat percentiles (nsec):
| 1.00th=[ 3792], 5.00th=[ 4080], 10.00th=[ 4320], 20.00th=[ 6112],
| 30.00th=[ 6176], 40.00th=[ 6176], 50.00th=[ 6240], 60.00th=[ 6240],
| 70.00th=[ 6240], 80.00th=[ 6304], 90.00th=[ 6560], 95.00th=[ 7072],
| 99.00th=[10048], 99.50th=[11456], 99.90th=[29824], 99.95th=[38144],
| 99.99th=[54528]
bw ( KiB/s): min= 192, max= 1920, per=100.00%, avg=1220.04, stdev=184.90, samples=1200
iops : min= 24, max= 240, avg=152.48, stdev=23.12, samples=1200
lat (usec) : 2=0.08%, 4=3.53%, 10=95.38%, 20=0.83%, 50=0.16%
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%
cpu : usr=0.32%, sys=1.72%, ctx=91655, majf=0, minf=53
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=91515,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
READ: bw=1220KiB/s (1249kB/s), 1220KiB/s-1220KiB/s (1249kB/s-1249kB/s), io=715MiB (750MB), run=600002-600002msec

Disk stats (read/write):
dm-6: ios=91493/0, merge=0/0, ticks=591084/0, in_queue=591084, util=42.34%, aggrios=91516/1608, aggrmerge=0/0, aggrticks=588704/516, aggrin_queue=589220, aggrutil=42.79%
dm-4: ios=91516/1608, merge=0/0, ticks=588704/516, in_queue=589220, util=42.79%, aggrios=45780/804, aggrmerge=0/0, aggrticks=294552/246, aggrin_queue=294798, aggrutil=42.81%
dm-2: ios=45/0, merge=0/0, ticks=588/0, in_queue=588, util=0.04%, aggrios=92042/1978, aggrmerge=0/491, aggrticks=594022/357, aggrin_queue=458056, aggrutil=42.92%
sda: ios=92042/1978, merge=0/491, ticks=594022/357, in_queue=458056, util=42.92%
dm-3: ios=91516/1608, merge=0/0, ticks=588516/492, in_queue=589008, util=42.81%

Random write:

root@proxmox01:/mnt/test# fio --name=randwrite --rw=randwrite --ioengine=libaio --bs=8k --numjobs=1 --size=4G --runtime=600 --group_reporting
randwrite: (g=0): rw=randwrite, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
randwrite: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [w(1)][87.5%][w=786MiB/s][w=101k IOPS][eta 00m:01s]
randwrite: (groupid=0, jobs=1): err= 0: pid=120022: Thu Jan 30 13:54:01 2020
write: IOPS=75.5k, BW=590MiB/s (618MB/s)(4096MiB/6947msec); 0 zone resets
slat (usec): min=5, max=534, avg=10.98, stdev= 7.75
clat (nsec): min=722, max=59669, avg=1101.73, stdev=740.19
lat (usec): min=6, max=565, avg=12.27, stdev= 8.15
clat percentiles (nsec):
| 1.00th=[ 748], 5.00th=[ 756], 10.00th=[ 764], 20.00th=[ 772],
| 30.00th=[ 780], 40.00th=[ 788], 50.00th=[ 796], 60.00th=[ 812],
| 70.00th=[ 844], 80.00th=[ 1960], 90.00th=[ 2008], 95.00th=[ 2064],
| 99.00th=[ 2160], 99.50th=[ 2192], 99.90th=[ 7456], 99.95th=[14784],
| 99.99th=[19072]
bw ( KiB/s): min=351632, max=825328, per=97.48%, avg=588513.23, stdev=178138.31, samples=13
iops : min=43954, max=103166, avg=73564.31, stdev=22267.27, samples=13
lat (nsec) : 750=2.11%, 1000=73.01%
lat (usec) : 2=12.23%, 4=12.54%, 10=0.01%, 20=0.09%, 50=0.01%
lat (usec) : 100=0.01%
cpu : usr=8.51%, sys=91.43%, ctx=14, majf=0, minf=84
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,524288,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
WRITE: bw=590MiB/s (618MB/s), 590MiB/s-590MiB/s (618MB/s-618MB/s), io=4096MiB (4295MB), run=6947-6947msec

Disk stats (read/write):
dm-6: ios=0/56, merge=0/0, ticks=0/0, in_queue=0, util=0.12%, aggrios=0/69, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.51%
dm-4: ios=0/69, merge=0/0, ticks=0/0, in_queue=0, util=0.51%, aggrios=0/34, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.51%
dm-2: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=8/23, aggrmerge=0/51, aggrticks=4/3, aggrin_queue=0, aggrutil=0.85%
sda: ios=8/23, merge=0/51, ticks=4/3, in_queue=0, util=0.85%
dm-3: ios=0/69, merge=0/0, ticks=0/0, in_queue=0, util=0.51%
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!