VM I/O Performance with Ceph Storage

shanreich · Jan 17, 2023

Yes, it really seems like those SSDs in particular perform awful, judging from a short googling session [1] - even when compared to other consumer SSDs.

t0mc@ said:
-> Setting the VM Disk Cache to "WriteBack" doesn't really change anything. BUT: Setting this to "WriteBack (unsafe)" massively increases performance of our little "Gimp Install" Test Case.

I would strongly recommend against using 'Writeback (No Cache)' mode for production systems.

t0mc@ said:
BUT: is it really realistic that the VM I/O performance (and notice: just when writing small files) when using NVMEs as backend feels even SLOWER compared when using SAS HDDs with 7.200 rpm? Sure, we understood that consumer NVMEs would not perform as good as enterprise ones... but even SLOWER than pure HDDs? Really?

Seems like on sustained writes those P2's can go as low as 40 Mbps (when the SLC cache gets filled) [1]. Ceph in general already has some issues with many small IO operations, since the overhead is quite large compared to just using the hardware directly. Writeback should alleviate that issue somewhat by queueing write commands. But in your case I'm afraid it would make sense that those P2's are just hitting limits.

Have you tried benchmarking the storage with rados bench, as described in our Ceph benchmarking report [2]? You can also read up a bit about the performance characteristics of different SSDs there.

[1] https://www.tomshardware.com/features/crucial-p2-ssd-qlc-flash-swap-downgrade/2
[2] https://www.proxmox.com/en/downloads/item/proxmox-ve-ceph-benchmark-2020-09

ness1602 · Jan 17, 2023

Crucial P* series is okay for Windows/Linux, but it is really standard consumer ssd, you can maybe use it as a ext4 for VM/CT and nothing more.

stepei · Jan 17, 2023

interesting thread. I have a question - which are the better NVME for a proxmox/ceph environment? We also have HP ProLiant Server G9 and PCIe 3.0

WD Red SN700 NVMe SSD
SEDC1000BM8/960G
Samsung 980 PRO M.2 NVMe

or other Enterprise SSD for more than €1,000

t0mc@ · Jan 17, 2023

@stepei: you anticipated our current questions which are going in exact the same direction

Actually we are going through the Links / PDF from @shanreich's last post and wondering if it is really necessary for ceph to have enterprise SSD/NVMe which costs >1K€ per piece (the PDF is dated 2020).
Or could it be a compromise concerning price / performance to use sth. like Samsung PRO or WD Red series?
Our claim is not to get each and every possible bit of performance out of the hardware, but nevertheless it should be more, than we have at the moment by using the - obviously - slow (for this use case) crucial nvmes.

What we are doing right now, BTW:
-> We moved all VM Disks to the HDD Pool, for freeing the NVMEs
-> We removed the NVMEs from Ceph
-> We are doing some benchmarking as described in the PDF from @shanreich 's Post and from here https://tracker.ceph.com/projects/ceph/wiki/Benchmark_Ceph_Cluster_Performance

t0mc@ · Jan 17, 2023

Here the first benchmarking results:

1.) FIO on NVMEs (all have similar values, no matter if 1TB or 2TB):

Code:

# fio --ioengine=libaio --filename=/dev/nvme2n1 --direct=1 --sync=1 --rw=write --bs=4K --numjobs=1 --iodepth=1 --runtime=60 --time_based --name=fio
fio: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=4520KiB/s][w=1130 IOPS][eta 00m:00s]
fio: (groupid=0, jobs=1): err= 0: pid=1582300: Tue Jan 17 14:14:04 2023
  write: IOPS=1115, BW=4463KiB/s (4570kB/s)(262MiB/60001msec); 0 zone resets
    slat (usec): min=2, max=340, avg=11.75, stdev= 7.20
    clat (usec): min=437, max=16633, avg=881.36, stdev=153.50
     lat (usec): min=451, max=16642, avg=893.44, stdev=153.81
    clat percentiles (usec):
     |  1.00th=[  652],  5.00th=[  816], 10.00th=[  832], 20.00th=[  848],
     | 30.00th=[  857], 40.00th=[  865], 50.00th=[  873], 60.00th=[  873],
     | 70.00th=[  881], 80.00th=[  898], 90.00th=[  922], 95.00th=[ 1123],
     | 99.00th=[ 1156], 99.50th=[ 1172], 99.90th=[ 1221], 99.95th=[ 1385],
     | 99.99th=[ 3785]
   bw (  KiB/s): min= 4320, max= 4576, per=100.00%, avg=4465.95, stdev=45.31, samples=119
   iops        : min= 1080, max= 1144, avg=1116.49, stdev=11.33, samples=119
  lat (usec)   : 500=0.14%, 750=4.54%, 1000=86.90%
  lat (msec)   : 2=8.40%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=0.95%, sys=1.92%, ctx=77710, majf=0, minf=26
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,66946,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=4463KiB/s (4570kB/s), 4463KiB/s-4463KiB/s (4570kB/s-4570kB/s), io=262MiB (274MB), run=60001-60001msec

Disk stats (read/write):
  nvme2n1: ios=6/66803, merge=0/0, ticks=6/58102, in_queue=58108, util=99.91%

--> IOPS=1115, BW=4463KiB/s, lat=0.9ms... ~4x better than Crucial MX100 512GB (IOPS=254, BW=1017KiB/s, lat=3.9ms, see PDF), but WAY slower than for example the next faster in the Table: Intel DC S3500 120GB (IOPS=12099, BW=48398KiB/s, lat=0.08ms)

2.) Rados Bench
We added the 2TB NVMEs at each node to ceph again, made a pool, which is using them

* Write Bench:

Code:

# rados bench -p pool_nvme_test 600 write -b 4M -t 16 --no-cleanup
[...]
Total time run:         600.373
Total writes made:      39608
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     263.889
Stddev Bandwidth:       34.2989
Max bandwidth (MB/sec): 364
Min bandwidth (MB/sec): 160
Average IOPS:           65
Stddev IOPS:            8.57472
Max IOPS:               91
Min IOPS:               40
Average Latency(s):     0.242427
Stddev Latency(s):      0.227764
Max latency(s):         0.683489
Min latency(s):         0.0158156

* Seq Read Bench:

Code:

# rados bench -p pool_nvme_test 600 seq -t 16
Total time run:       586.168
Total reads made:     39608
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   270.284
Average IOPS:         67
Stddev IOPS:          8.61466
Max IOPS:             93
Min IOPS:             42
Average Latency(s):   0.236004
Max latency(s):       0.597122
Min latency(s):       0.00526951

--> Also WAY slower, than the ones from PDF.
But what I notice... The Bandwiths and IOPS are nearly the same for read and write? IMHO Read should be higher than write, shouldn't it?

ITT · Jan 17, 2023

Always the same:
Use better Disk -> Enterprise-Hardware!
If your disk is slow, it is slow.
How are these cheap NVMe´s connected to the DL380? Which PCIe-Slot? How many lanes?
A lot of missing Information here.

ITT · Jan 17, 2023

stepei said:
interesting thread. I have a question - which are the better NVME for a proxmox/ceph environment? We also have HP ProLiant Server G9 and PCIe 3.0

WD Red SN700 NVMe SSD
SEDC1000BM8/960G
Samsung 980 PRO M.2 NVMe

or other Enterprise SSD for more than €1,000

These are all crappy disks for Ceph or ZFS.
We using Kioxia or Microns for Ceph.

t0mc@ · Jan 17, 2023

ITT said:
Always the same:
Use better Disk -> Enterprise-Hardware!
If your disk is slow, it is slow.
How are these cheap NVMe´s connected to the DL380? Which PCIe-Slot? How many lanes?
A lot of missing Information here.

Infos arn't missing, given here (https://forum.proxmox.com/threads/vm-i-o-performance-with-ceph-storage.120929/post-526311)

-> The disks are those:
Crucial P2 CT1000P2SSD8 (1TB)
Crucial P2 CT2000P2SSD8 (2TB)
Connected via PCIe Adapter Card to PCIe 4x Slots

ITT · Jan 17, 2023

t0mc@ said:
Infos arn't missing, given here (https://forum.proxmox.com/threads/vm-i-o-performance-with-ceph-storage.120929/post-526311)

-> The disks are those:
Crucial P2 CT1000P2SSD8 (1TB)
Crucial P2 CT2000P2SSD8 (2TB)
Connected via PCIe Adapter Card to PCIe 4x Slots

Missing Information:
What kind of PCIe to NVMe Adapter?
Which PCIe-Slot connecting the Adapter?

t0mc@ · Jan 17, 2023

In the meantime we benchmarked a VM Disk using the NVMe Ceph Pool inside our Debian Testing VM with fio, and are a little bit surprised:

Code:

#fio --ioengine=psync --filename=/dev/sdb --size=9G --time_based --name=fio --group_reporting --runtime=600 --direct=1 --sync=1 --rw=write --bs=4K --numjobs=1 --iodepth=1
fio: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.25
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=444KiB/s][w=111 IOPS][eta 00m:00s]
fio: (groupid=0, jobs=1): err= 0: pid=564: Tue Jan 17 17:53:47 2023
  write: IOPS=106, BW=427KiB/s (437kB/s)(250MiB/600002msec); 0 zone resets
    clat (usec): min=8535, max=59585, avg=9364.30, stdev=1255.98
     lat (usec): min=8535, max=59585, avg=9364.74, stdev=1255.98
    clat percentiles (usec):
     |  1.00th=[ 8717],  5.00th=[ 8848], 10.00th=[ 8979], 20.00th=[ 8979],
     | 30.00th=[ 9110], 40.00th=[ 9241], 50.00th=[ 9241], 60.00th=[ 9372],
     | 70.00th=[ 9372], 80.00th=[ 9503], 90.00th=[ 9634], 95.00th=[ 9765],
     | 99.00th=[12387], 99.50th=[16057], 99.90th=[33424], 99.95th=[35390],
     | 99.99th=[41157]
   bw (  KiB/s): min=  296, max=  456, per=100.00%, avg=427.10, stdev=13.87, samples=1199
   iops        : min=   74, max=  114, avg=106.77, stdev= 3.47, samples=1199
  lat (msec)   : 10=97.19%, 20=2.55%, 50=0.26%, 100=0.01%
  cpu          : usr=0.09%, sys=0.30%, ctx=192176, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,64051,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=427KiB/s (437kB/s), 427KiB/s-427KiB/s (437kB/s-437kB/s), io=250MiB (262MB), run=600002-600002msec

Disk stats (read/write):
  sdb: ios=55/128078, merge=0/0, ticks=29/598076, in_queue=607596, util=100.00%

--> IOPS=106, BW=427KiB/s, lat=9.3ms

Remember: The plain NVMe fio result was:
--> IOPS=1115, BW=4463KiB/s, lat=0.9ms

So: According to fio the virtual Disk in the Debian VM is ~10 Times(!!) slower, than the NVMe is able to be. Where is the bottleneck? Can this really be solely explained with "If your disk is slow, it is slow."?

t0mc@ · Jan 17, 2023

When setting the VM Disk Cache from "No cache" to "Write Back" things even get worse:

--> IOPS=74, BW=297KiB/s, lat=13.5ms

We thought, "Write Back" would increase write performance... everything very confusing...

t0mc@ · Jan 17, 2023

Last but not least for completing the whole picture:

fio write test of a VM Disk using Ceph Pool with the HDD SAS Disks as Storage:

--> IOPS=7, BW=31.9KiB/s, lat=125.31ms

IMHO bad values even for SAS HDDs, isn't it?
Which again leads us to the assumption, that there is sth. wrong more generally with the Ceph Cluster or its configuration? Proxmox itself? Network? Whatever?
And we can't really remember that the I/O Write performance was as bad as now when the cluster was freshly installed (what must have been when bad consumer hardware shall be responsible for this) but the loss of write performance came over time.

stepei · Jan 17, 2023

ITT said:
These are all crappy disks for Ceph or ZFS.
We using Kioxia or Microns for Ceph.

And which model exactly for 2TB or 1TB do you prefer?

t0mc@ · Jan 17, 2023

ITT said:
Missing Information:
What kind of PCIe to NVMe Adapter?
Which PCIe-Slot connecting the Adapter?

So here the missing Infos:
Adapter Cards of this kind:
https://www.amazon.de/Adapter-Kühlk...er&qid=1673986009&sprefix=pcie,aps,196&sr=8-3

PCIe Slots they are running in:

ITT · Jan 18, 2023

Thanks for provided Information.
Please post the Result of

Code:

lspci -vv | grep -P "[0-9a-f]{2}:[0-9a-f]{2}\.[0-9a-f]|LnkSta:"

ITT · Jan 18, 2023

stepei said:
And which model exactly for 2TB or 1TB do you prefer?

What we use in several Ceph-Clusters with good experience (and Performance):

Ceph:
Kioxia CD6-V Series

PBS:
Micron 7450 MAX

or for cheaper solution Micron 5300/5400 MAX

t0mc@ · Jan 18, 2023

ITT said:
Thanks for provided Information.
Please post the Result of

Code:

lspci -vv | grep -P "[0-9a-f]{2}:[0-9a-f]{2}\.[0-9a-f]|LnkSta:"

More than 16384 characters, have to split the output:

Code:

# lspci -vv | grep -P "[0-9a-f]{2}:[0-9a-f]{2}\.[0-9a-f]|LnkSta:"
00:00.0 Host bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMI2 (rev 02)
        LnkSta:    Speed unknown (downgraded), Width x0 (downgraded)
00:01.0 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 1 (rev 02) (prog-if 00 [Normal decode])
        LnkSta:    Speed 2.5GT/s (downgraded), Width x0 (downgraded)
00:01.1 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 1 (rev 02) (prog-if 00 [Normal decode])
        LnkSta:    Speed 2.5GT/s (downgraded), Width x0 (downgraded)
00:02.0 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 2 (rev 02) (prog-if 00 [Normal decode])
        LnkSta:    Speed 8GT/s (ok), Width x4 (downgraded)
00:02.1 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 2 (rev 02) (prog-if 00 [Normal decode])
        LnkSta:    Speed 8GT/s (ok), Width x0 (downgraded)
00:02.2 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 2 (rev 02) (prog-if 00 [Normal decode])
        LnkSta:    Speed 8GT/s (ok), Width x8 (ok)
00:02.3 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 2 (rev 02) (prog-if 00 [Normal decode])
        LnkSta:    Speed 8GT/s (ok), Width x0 (downgraded)
00:03.0 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02) (prog-if 00 [Normal decode])
        LnkSta:    Speed 2.5GT/s (downgraded), Width x0 (downgraded)
00:03.1 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02) (prog-if 00 [Normal decode])
        LnkSta:    Speed 2.5GT/s (downgraded), Width x0 (downgraded)
00:03.2 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02) (prog-if 00 [Normal decode])
        LnkSta:    Speed 8GT/s (ok), Width x8 (ok)
00:03.3 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02) (prog-if 00 [Normal decode])
        LnkSta:    Speed 8GT/s (ok), Width x0 (downgraded)
00:04.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 0 (rev 02)
00:04.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 1 (rev 02)
00:04.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 2 (rev 02)
00:04.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 3 (rev 02)
00:04.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 4 (rev 02)
00:04.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 5 (rev 02)
00:04.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 6 (rev 02)
00:04.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 7 (rev 02)
00:05.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management (rev 02)
00:05.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Hot Plug (rev 02)
00:05.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 RAS, Control Status and Global Errors (rev 02)
00:05.4 PIC: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 I/O APIC (rev 02) (prog-if 20 [IO(X)-APIC])
00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset SPSR (rev 05)
00:14.0 USB controller: Intel Corporation C610/X99 series chipset USB xHCI Host Controller (rev 05) (prog-if 30 [XHCI])
00:1a.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 (rev 05) (prog-if 20 [EHCI])
00:1c.0 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 (rev d5) (prog-if 01 [Subtractive decode])
        LnkSta:    Speed 2.5GT/s (downgraded), Width x0 (downgraded)
00:1c.2 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #3 (rev d5) (prog-if 00 [Normal decode])
        LnkSta:    Speed 2.5GT/s (downgraded), Width x1 (downgraded)
00:1c.4 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #5 (rev d5) (prog-if 00 [Normal decode])
        LnkSta:    Speed 5GT/s (ok), Width x2 (ok)
00:1c.6 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #7 (rev d5) (prog-if 00 [Normal decode])
        LnkSta:    Speed 2.5GT/s (downgraded), Width x0 (downgraded)
00:1c.7 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #8 (rev d5) (prog-if 00 [Normal decode])
        LnkSta:    Speed 2.5GT/s (downgraded), Width x0 (downgraded)
00:1d.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 (rev 05) (prog-if 20 [EHCI])
00:1f.0 ISA bridge: Intel Corporation C610/X99 series chipset LPC Controller (rev 05)
00:1f.3 SMBus: Intel Corporation C610/X99 series chipset SMBus Controller (rev 05)
01:00.0 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Slave Instrumentation & System Support (rev 06)
        LnkSta:    Speed 2.5GT/s (ok), Width x1 (ok)
01:00.1 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200EH (rev 01) (prog-if 00 [VGA controller])
        LnkSta:    Speed 2.5GT/s (ok), Width x1 (ok)
01:00.2 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Management Processor Support and Messaging (rev 06)
        LnkSta:    Speed 2.5GT/s (ok), Width x1 (ok)
01:00.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (rev 03) (prog-if 00 [UHCI])
        LnkSta:    Speed 2.5GT/s (ok), Width x1 (ok)
02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
        LnkSta:    Speed 5GT/s (ok), Width x2 (downgraded)
02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
        LnkSta:    Speed 5GT/s (ok), Width x2 (downgraded)
02:00.2 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
        LnkSta:    Speed 5GT/s (ok), Width x2 (downgraded)
02:00.3 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
        LnkSta:    Speed 5GT/s (ok), Width x2 (downgraded)
04:00.0 Network controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
        LnkSta:    Speed 8GT/s (ok), Width x8 (ok)
05:00.0 Non-Volatile memory controller: Micron/Crucial Technology Device 540a (rev 01) (prog-if 02 [NVM Express])
        LnkSta:    Speed 8GT/s (ok), Width x4 (ok)
0b:00.0 Serial Attached SCSI controller: Hewlett-Packard Company Smart Array Gen9 Controllers (rev 01)
        LnkSta:    Speed 8GT/s (ok), Width x8 (ok)
7f:08.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0 (rev 02)
7f:08.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0 (rev 02)
7f:09.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1 (rev 02)
7f:09.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1 (rev 02)
7f:0b.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)
7f:0b.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)
7f:0b.2 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)
7f:0c.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
7f:0c.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
7f:0c.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
7f:0c.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
7f:0c.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
7f:0c.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
7f:0c.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
7f:0c.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
7f:0d.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
7f:0d.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
7f:0d.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
7f:0d.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
7f:0f.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)
7f:0f.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)
7f:0f.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)
7f:0f.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)
7f:0f.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)
7f:0f.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)
7f:0f.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)
7f:10.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface (rev 02)
7f:10.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface (rev 02)
7f:10.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)
7f:10.6 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)
7f:10.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)
7f:12.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 (rev 02)
7f:12.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 (rev 02)
7f:12.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 Debug (rev 02)
7f:12.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 1 (rev 02)
7f:12.5 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 1 (rev 02)
7f:12.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 1 Debug (rev 02)
7f:13.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 02)
7f:13.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 02)

t0mc@ · Jan 18, 2023

Code:

7f:13.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02)

7f:13.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02)

7f:13.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 0/1 Broadcast (rev 02)

7f:13.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast (rev 02)

7f:14.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal Control (rev 02)

7f:14.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal Control (rev 02)

7f:14.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR Registers (rev 02)

7f:14.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR Registers (rev 02)

7f:14.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)

7f:14.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)

7f:14.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)

7f:14.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)

7f:16.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers (rev 02)

7f:16.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers (rev 02)

7f:16.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder (rev 02)

7f:16.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder (rev 02)

7f:16.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 2/3 Broadcast (rev 02)

7f:16.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast (rev 02)

7f:17.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 Thermal Control (rev 02)

7f:17.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 1 Thermal Control (rev 02)

7f:17.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 ERROR Registers (rev 02)

7f:17.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 1 ERROR Registers (rev 02)

7f:17.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)

7f:17.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)

7f:17.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)

7f:17.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)

7f:1e.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)

7f:1e.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)

7f:1e.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)

7f:1e.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)

7f:1e.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)

7f:1f.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU (rev 02)

7f:1f.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU (rev 02)

80:00.0 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 0 (rev 02) (prog-if 00 [Normal decode])

        LnkSta:    Speed 2.5GT/s (downgraded), Width x0 (downgraded)

80:01.0 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 1 (rev 02) (prog-if 00 [Normal decode])

        LnkSta:    Speed 2.5GT/s (downgraded), Width x0 (downgraded)

80:01.1 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 1 (rev 02) (prog-if 00 [Normal decode])

        LnkSta:    Speed 2.5GT/s (downgraded), Width x0 (downgraded)

80:02.0 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 2 (rev 02) (prog-if 00 [Normal decode])

        LnkSta:    Speed 8GT/s (ok), Width x4 (downgraded)

80:02.1 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 2 (rev 02) (prog-if 00 [Normal decode])

        LnkSta:    Speed 8GT/s (ok), Width x0 (downgraded)

80:02.2 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 2 (rev 02) (prog-if 00 [Normal decode])

        LnkSta:    Speed 8GT/s (ok), Width x0 (downgraded)

80:02.3 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 2 (rev 02) (prog-if 00 [Normal decode])

        LnkSta:    Speed 8GT/s (ok), Width x0 (downgraded)

80:03.0 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02) (prog-if 00 [Normal decode])

        LnkSta:    Speed 8GT/s (ok), Width x4 (downgraded)

80:03.1 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02) (prog-if 00 [Normal decode])

        LnkSta:    Speed 8GT/s (ok), Width x0 (downgraded)

80:03.2 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02) (prog-if 00 [Normal decode])

        LnkSta:    Speed 8GT/s (ok), Width x0 (downgraded)

80:03.3 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02) (prog-if 00 [Normal decode])

        LnkSta:    Speed 8GT/s (ok), Width x0 (downgraded)

80:04.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 0 (rev 02)

80:04.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 1 (rev 02)

80:04.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 2 (rev 02)

80:04.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 3 (rev 02)

80:04.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 4 (rev 02)

80:04.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 5 (rev 02)

80:04.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 6 (rev 02)

80:04.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 7 (rev 02)

80:05.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management (rev 02)

80:05.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Hot Plug (rev 02)

80:05.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 RAS, Control Status and Global Errors (rev 02)

80:05.4 PIC: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 I/O APIC (rev 02) (prog-if 20 [IO(X)-APIC])

84:00.0 Non-Volatile memory controller: Micron/Crucial Technology Device 540a (rev 01) (prog-if 02 [NVM Express])

        LnkSta:    Speed 8GT/s (ok), Width x4 (ok)

88:00.0 Non-Volatile memory controller: Micron/Crucial Technology Device 540a (rev 01) (prog-if 02 [NVM Express])

        LnkSta:    Speed 8GT/s (ok), Width x4 (ok)

ff:08.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0 (rev 02)

ff:08.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0 (rev 02)

ff:09.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1 (rev 02)

ff:09.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1 (rev 02)

ff:0b.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)

ff:0b.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)

ff:0b.2 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)

ff:0c.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)

ff:0c.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)

ff:0c.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)

ff:0c.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)

ff:0c.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)

ff:0c.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)

ff:0c.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)

ff:0c.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)

ff:0d.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)

ff:0d.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)

ff:0d.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)

ff:0d.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)

ff:0f.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)

ff:0f.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)

ff:0f.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)

ff:0f.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)

ff:0f.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)

ff:0f.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)

ff:0f.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)

ff:10.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface (rev 02)

ff:10.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface (rev 02)

ff:10.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)

ff:10.6 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)

ff:10.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)

ff:12.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 (rev 02)

ff:12.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 (rev 02)

ff:12.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 Debug (rev 02)

ff:12.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 1 (rev 02)

ff:12.5 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 1 (rev 02)

ff:12.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 1 Debug (rev 02)

ff:13.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 02)

ff:13.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 02)

ff:13.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02)

ff:13.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02)

ff:13.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 0/1 Broadcast (rev 02)

ff:13.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast (rev 02)

ff:14.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal Control (rev 02)

ff:14.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal Control (rev 02)

ff:14.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR Registers (rev 02)

ff:14.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR Registers (rev 02)

ff:14.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)

ff:14.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)

ff:14.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)

ff:14.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)

ff:16.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers (rev 02)

ff:16.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers (rev 02)

ff:16.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder (rev 02)

ff:16.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder (rev 02)

ff:16.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 2/3 Broadcast (rev 02)

ff:16.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast (rev 02)

ff:17.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 Thermal Control (rev 02)

ff:17.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 1 Thermal Control (rev 02)

ff:17.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 ERROR Registers (rev 02)

ff:17.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 1 ERROR Registers (rev 02)

ff:17.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)

ff:17.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)

ff:17.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)

ff:17.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)

ff:1e.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)

ff:1e.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)

ff:1e.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)

ff:1e.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)

ff:1e.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)

ff:1f.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU (rev 02)

ff:1f.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU (rev 02)

ITT · Jan 18, 2023

Code:

04:00.0 Network controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
        LnkSta:    Speed 8GT/s (ok), Width x8 (ok)
05:00.0 Non-Volatile memory controller: Micron/Crucial Technology Device 540a (rev 01) (prog-if 02 [NVM Express])
        LnkSta:    Speed 8GT/s (ok), Width x4 (ok)
0b:00.0 Serial Attached SCSI controller: Hewlett-Packard Company Smart Array Gen9 Controllers (rev 01)
        LnkSta:    Speed 8GT/s (ok), Width x8 (ok)

Looks good.
Sadly to say, your Bottleneck are the Consumer-Grade-NVMe Disks.
Hopefully you have set the MTU to 9000 on your Mellanoxes (Ceph-Network)

One hint: Use more smaller Disks for Ceph instead few big Disks.
Minimum 3 per Node (and Pool).

t0mc@ · Jan 18, 2023

@ITT
We have ordered 3 Enterprise NVMes each 1TB in size and now hoping the best....
And yes, we set the MTU to 9000

But - sorry for coming back to this thoughts again - I'm really not sure if something else is going completely wrong in the cluster and this "consumer nvmes are bad" - thing of course comes on top but completely distracts from the main problem.

To summarize what we found out so far with the fio tests:

FIO Test	IOPS	Bandwith	Lat
NVMe hardware directly at the Proxmox Node	1115	4463KiB/s	0.9ms
VM Disk using NVMe based storage Pool - no Cache	106	427KiB/s	9.3ms
VM Disk using NVMe based storage Pool - WriteBack Cache	74	297KiB/s	13.5ms
VM Disk using SAS HDD based storage Pool - no Cache	7	32KiB/s	125.3ms

What I really want to get answered are these questions, which no one really seems to notice till now:

Is it common, that the VM Disk performs 10(!) times slower, than the underlying storage hardware could be (compare rows 1 - 2)?
Shouldn't "WriteBack Cache" Setting give higher write performance, not lower (compare rows 2 - 3)?
Are the values in the last row really normal, even for SAS HDDs? This can of course nothing have to do with consumer NVMes....

And just noticed another fact, that - for me - will not really fit to this "consumer nvme are solely responsible" thing: When moving a VM Disk from the SAS HDD based storage Pool to the NVMe based one, write performance seems not so bad in my eyes:

--> 32GB in 6:42....

VM I/O Performance with Ceph Storage

Proxmox Staff Member

Renowned Member

New Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

New Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member