Dell R920 VM Performance issues

[tomstephens89]

Hi all,

Just spun up a couple of new Proxmox boxes, these are Dell R920's with 4 x Xeon E7 4880v2's 15 cores each, so 60 cores + hyperthreading. 512GB RAM in the boxes, local storage is a 22 disk RAID 10 of spinners. No SSD's yet, but still not slow.

I have created 2 linux VM's and a windows VM so far and the VM's don't feel very responsive at all. Slow in fact.

I left Hyperthreading ENABLED, I think I might disable it and see what that does in performance terms. I am wondering if such a high core count causes CPU scheduler issues when HT is on.

NUMA is ENABLED. Disabling/Enabling makes no difference.

Does anyone have similar experience when using high core count nodes?

 

[tomstephens89]

Anyone?

 

[czechsys]

Dell Rxxx (4U) with 4x Gold Xeons 12cores = 48 physical, 48 ht cores, ssd array. No performance hit.

 

[sg90]

Hi all,

Just spun up a couple of new Proxmox boxes, these are Dell R920's with 4 x Xeon E7 4880v2's 15 cores each, so 60 cores + hyperthreading. 512GB RAM in the boxes, local storage is a 22 disk RAID 10 of spinners. No SSD's yet, but still not slow.

I have created 2 linux VM's and a windows VM so far and the VM's don't feel very responsive at all. Slow in fact.

I left Hyperthreading ENABLED, I think I might disable it and see what that does in performance terms. I am wondering if such a high core count causes CPU scheduler issues when HT is on.

NUMA is ENABLED. Disabling/Enabling makes no difference.

Does anyone have similar experience when using high core count nodes?

How many cores are you passing through to VM? What CPU type you selected?

 

[tomstephens89]

How many cores are you passing through to VM? What CPU type you selected?

Host type CPU and 4 cores.

 

[tomstephens89]

I have disabled hyperthreading on another identical box thats still in the dev lab and performance on VM's still seems sluggish.

Heres a pveperf output with an lvm disk mounted at /mnt/test

root@proxmox01:~# pveperf /mnt/test
CPU BOGOMIPS: 598857.60
REGEX/SECOND: 1511571
HD SIZE: 1967.61 GB (/dev/mapper/pve-vm--100--disk--0)
BUFFERED READS: 623.98 MB/sec
AVERAGE SEEK TIME: 34.23 ms
FSYNCS/SECOND: 1438.72
DNS EXT: 39.53 ms
DNS INT: 35.05 ms

The local disk set up is 22 1TB 7.2K SAS disks in RAID10 with 2 as hot spares. Stripe size is 128K, controller is a Dell H730P with 2GB Cache.

 

[tomstephens89]

Heres a fio sequential read:

root@proxmox01:/mnt/test# fio --name=seqread --rw=read --ioengine=libaio --bs=8k --numjobs=1 --size=8G --runtime=600 --group_reporting
seqread: (g=0): rw=read, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=981MiB/s][r=126k IOPS][eta 00m:00s]
seqread: (groupid=0, jobs=1): err= 0: pid=115266: Thu Jan 30 13:27:28 2020
read: IOPS=107k, BW=836MiB/s (876MB/s)(8192MiB/9802msec)
slat (nsec): min=1498, max=17983k, avg=6553.16, stdev=40837.90
clat (nsec): min=660, max=258654, avg=805.92, stdev=590.84
lat (usec): min=2, max=17988, avg= 7.44, stdev=40.96
clat percentiles (nsec):
| 1.00th=[ 692], 5.00th=[ 700], 10.00th=[ 708], 20.00th=[ 716],
| 30.00th=[ 716], 40.00th=[ 724], 50.00th=[ 732], 60.00th=[ 740],
| 70.00th=[ 756], 80.00th=[ 772], 90.00th=[ 1012], 95.00th=[ 1176],
| 99.00th=[ 1528], 99.50th=[ 1608], 99.90th=[ 8160], 99.95th=[14912],
| 99.99th=[19072]
bw ( KiB/s): min=933312, max=1035776, per=100.00%, avg=1010940.00, stdev=26304.89, samples=16
iops : min=116664, max=129472, avg=126367.50, stdev=3288.11, samples=16
lat (nsec) : 750=67.38%, 1000=22.15%
lat (usec) : 2=10.32%, 4=0.03%, 10=0.02%, 20=0.09%, 50=0.01%
lat (usec) : 100=0.01%, 500=0.01%
cpu : usr=16.21%, sys=82.80%, ctx=169, majf=0, minf=18
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
READ: bw=836MiB/s (876MB/s), 836MiB/s-836MiB/s (876MB/s-876MB/s), io=8192MiB (8590MB), run=9802-9802msec

Disk stats (read/write):
dm-6: ios=160786/4, merge=0/0, ticks=18192/0, in_queue=18192, util=82.98%, aggrios=163842/58, aggrmerge=0/0, aggrticks=17316/0, aggrin_queue=17316, aggrutil=82.76%
dm-4: ios=163842/58, merge=0/0, ticks=17316/0, in_queue=17316, util=82.76%, aggrios=81921/29, aggrmerge=0/0, aggrticks=8616/0, aggrin_queue=8616, aggrutil=82.76%
dm-2: ios=1/0, merge=0/0, ticks=28/0, in_queue=28, util=0.08%, aggrios=33497/70, aggrmerge=130355/10, aggrticks=4324/22, aggrin_queue=152, aggrutil=82.77%
sda: ios=33497/70, merge=130355/10, ticks=4324/22, in_queue=152, util=82.77%
dm-3: ios=163842/58, merge=0/0, ticks=17204/0, in_queue=17204, util=82.76%

And heres a fio sequential write:

root@proxmox01:/mnt/test# fio --name=seqread --rw=write --ioengine=libaio --bs=8k --numjobs=1 --size=8G --runtime=600 --group_reporting
seqread: (g=0): rw=write, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=941MiB/s][w=120k IOPS][eta 00m:00s]
seqread: (groupid=0, jobs=1): err= 0: pid=115508: Thu Jan 30 13:28:18 2020
write: IOPS=91.5k, BW=715MiB/s (750MB/s)(8192MiB/11454msec); 0 zone resets
slat (usec): min=5, max=514, avg= 8.44, stdev= 4.65
clat (nsec): min=724, max=53758, avg=860.79, stdev=432.11
lat (usec): min=6, max=570, avg= 9.44, stdev= 4.83
clat percentiles (nsec):
| 1.00th=[ 748], 5.00th=[ 756], 10.00th=[ 764], 20.00th=[ 772],
| 30.00th=[ 780], 40.00th=[ 788], 50.00th=[ 788], 60.00th=[ 796],
| 70.00th=[ 804], 80.00th=[ 820], 90.00th=[ 844], 95.00th=[ 1080],
| 99.00th=[ 2064], 99.50th=[ 2128], 99.90th=[ 2224], 99.95th=[ 2640],
| 99.99th=[15168]
bw ( KiB/s): min=110992, max=976752, per=100.00%, avg=757565.43, stdev=220784.79, samples=21
iops : min=13874, max=122094, avg=94695.67, stdev=27598.16, samples=21
lat (nsec) : 750=0.96%, 1000=93.96%
lat (usec) : 2=1.96%, 4=3.07%, 10=0.01%, 20=0.03%, 50=0.01%
lat (usec) : 100=0.01%
cpu : usr=11.42%, sys=88.51%, ctx=79, majf=0, minf=56
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,1048576,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
WRITE: bw=715MiB/s (750MB/s), 715MiB/s-715MiB/s (750MB/s-750MB/s), io=8192MiB (8590MB), run=11454-11454msec

Disk stats (read/write):
dm-6: ios=0/9681, merge=0/0, ticks=0/6936, in_queue=6936, util=2.34%, aggrios=0/9706, aggrmerge=0/0, aggrticks=0/6696, aggrin_queue=6696, aggrutil=2.73%
dm-4: ios=0/9706, merge=0/0, ticks=0/6696, in_queue=6696, util=2.73%, aggrios=0/4853, aggrmerge=0/0, aggrticks=0/3340, aggrin_queue=3340, aggrutil=2.73%
dm-2: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=9/9658, aggrmerge=0/55, aggrticks=16/6801, aggrin_queue=2924, aggrutil=2.97%
sda: ios=9/9658, merge=0/55, ticks=16/6801, in_queue=2924, util=2.97%
dm-3: ios=0/9706, merge=0/0, ticks=0/6680, in_queue=6680, util=2.73%

 

[tomstephens89]

Random read, looks low to me, but my previous proxmox installs performed just fine and they used a SAN with spinners over a network. No performance issues with those VM's.

root@proxmox01:/mnt/test# fio --name=randread --rw=randread --ioengine=libaio --bs=8k --numjobs=1 --size=1G --runtime=600 --group_reporting
randread: (g=0): rw=randread, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=1401KiB/s][r=175 IOPS][eta 00m:00s]
randread: (groupid=0, jobs=1): err= 0: pid=116855: Thu Jan 30 13:43:13 2020
read: IOPS=152, BW=1220KiB/s (1249kB/s)(715MiB/600002msec)
slat (usec): min=73, max=158310, avg=6540.38, stdev=7871.30
clat (nsec): min=1436, max=321584, avg=6104.69, stdev=2001.03
lat (usec): min=75, max=158318, avg=6548.21, stdev=7871.65
clat percentiles (nsec):
| 1.00th=[ 3792], 5.00th=[ 4080], 10.00th=[ 4320], 20.00th=[ 6112],
| 30.00th=[ 6176], 40.00th=[ 6176], 50.00th=[ 6240], 60.00th=[ 6240],
| 70.00th=[ 6240], 80.00th=[ 6304], 90.00th=[ 6560], 95.00th=[ 7072],
| 99.00th=[10048], 99.50th=[11456], 99.90th=[29824], 99.95th=[38144],
| 99.99th=[54528]
bw ( KiB/s): min= 192, max= 1920, per=100.00%, avg=1220.04, stdev=184.90, samples=1200
iops : min= 24, max= 240, avg=152.48, stdev=23.12, samples=1200
lat (usec) : 2=0.08%, 4=3.53%, 10=95.38%, 20=0.83%, 50=0.16%
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%
cpu : usr=0.32%, sys=1.72%, ctx=91655, majf=0, minf=53
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=91515,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
READ: bw=1220KiB/s (1249kB/s), 1220KiB/s-1220KiB/s (1249kB/s-1249kB/s), io=715MiB (750MB), run=600002-600002msec

Disk stats (read/write):
dm-6: ios=91493/0, merge=0/0, ticks=591084/0, in_queue=591084, util=42.34%, aggrios=91516/1608, aggrmerge=0/0, aggrticks=588704/516, aggrin_queue=589220, aggrutil=42.79%
dm-4: ios=91516/1608, merge=0/0, ticks=588704/516, in_queue=589220, util=42.79%, aggrios=45780/804, aggrmerge=0/0, aggrticks=294552/246, aggrin_queue=294798, aggrutil=42.81%
dm-2: ios=45/0, merge=0/0, ticks=588/0, in_queue=588, util=0.04%, aggrios=92042/1978, aggrmerge=0/491, aggrticks=594022/357, aggrin_queue=458056, aggrutil=42.92%
sda: ios=92042/1978, merge=0/491, ticks=594022/357, in_queue=458056, util=42.92%
dm-3: ios=91516/1608, merge=0/0, ticks=588516/492, in_queue=589008, util=42.81%

Random write:

root@proxmox01:/mnt/test# fio --name=randwrite --rw=randwrite --ioengine=libaio --bs=8k --numjobs=1 --size=4G --runtime=600 --group_reporting
randwrite: (g=0): rw=randwrite, bs=(R) 8192B-8192B, (W) 8192B-8192B, (T) 8192B-8192B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
randwrite: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [w(1)][87.5%][w=786MiB/s][w=101k IOPS][eta 00m:01s]
randwrite: (groupid=0, jobs=1): err= 0: pid=120022: Thu Jan 30 13:54:01 2020
write: IOPS=75.5k, BW=590MiB/s (618MB/s)(4096MiB/6947msec); 0 zone resets
slat (usec): min=5, max=534, avg=10.98, stdev= 7.75
clat (nsec): min=722, max=59669, avg=1101.73, stdev=740.19
lat (usec): min=6, max=565, avg=12.27, stdev= 8.15
clat percentiles (nsec):
| 1.00th=[ 748], 5.00th=[ 756], 10.00th=[ 764], 20.00th=[ 772],
| 30.00th=[ 780], 40.00th=[ 788], 50.00th=[ 796], 60.00th=[ 812],
| 70.00th=[ 844], 80.00th=[ 1960], 90.00th=[ 2008], 95.00th=[ 2064],
| 99.00th=[ 2160], 99.50th=[ 2192], 99.90th=[ 7456], 99.95th=[14784],
| 99.99th=[19072]
bw ( KiB/s): min=351632, max=825328, per=97.48%, avg=588513.23, stdev=178138.31, samples=13
iops : min=43954, max=103166, avg=73564.31, stdev=22267.27, samples=13
lat (nsec) : 750=2.11%, 1000=73.01%
lat (usec) : 2=12.23%, 4=12.54%, 10=0.01%, 20=0.09%, 50=0.01%
lat (usec) : 100=0.01%
cpu : usr=8.51%, sys=91.43%, ctx=14, majf=0, minf=84
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,524288,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
WRITE: bw=590MiB/s (618MB/s), 590MiB/s-590MiB/s (618MB/s-618MB/s), io=4096MiB (4295MB), run=6947-6947msec

Disk stats (read/write):
dm-6: ios=0/56, merge=0/0, ticks=0/0, in_queue=0, util=0.12%, aggrios=0/69, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.51%
dm-4: ios=0/69, merge=0/0, ticks=0/0, in_queue=0, util=0.51%, aggrios=0/34, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.51%
dm-2: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=8/23, aggrmerge=0/51, aggrticks=4/3, aggrin_queue=0, aggrutil=0.85%
sda: ios=8/23, merge=0/51, ticks=4/3, in_queue=0, util=0.85%
dm-3: ios=0/69, merge=0/0, ticks=0/0, in_queue=0, util=0.51%