show stoping nvme performance in guest. 15% of native/pass through.

[lens06]

i have a proxmox 6.4-15 box. I want to order a shiny new server for work. trying to evaluate if i can get enough io performance to justify the cost. im not sure if im poisoning my own results because of the ancient server and software version im trying this on

where things suffer are the random 4k high qd and thread tests. Both tests bellow the values given are for this test.

PCIE Pass through:
native (pcie pass through to server 2022 guest): 1900MB/s read and 1200 write.
same results (plus or minus 100mb/s) were obtained sticking this in a gaming computers second nvme slot

LVM Pool virtual drive:
300MB/s read and 300MB/s write. (4k rnd qd:32 t16)

i've got the controller set to virt io scsi single. I've got IO thread box checked...

where it gets interesting is if i create a second VD on the same NVME/LVM group... and stripe both in the guest I get around 600MB/s for that test. I would never trust this in production.

to me this indicates it has to do with not enough threading even with io thread enabled? or maybe my e7v3 is just too slow?

either way i would love input on this before i purchase a new and expensive server (with proper enterprise nvme)

 

[Dunuin]

As far as I understand, with "virtio SCSI single" each vDisk gets its own single thread. With "virtio SCSI" 8 vDisks will share a thread.

 

[zzz09700]

What we do in our place is, if someone need serious disk performance, buy a disk and RDM or passthrough that disk to his VM.
Virtual disks always suck, in one way or the other.

 

[lens06]

i might go back to kvm. there is some cool work being done there https://vmsplice.net/~stefan/stefanha-kvm-forum-2020.pdf= but then i break the "can the next guy pick this up if i leave" question lol.

im setting this up for a factory. it will bearly need to grow or change but i need certain performance levels in our key app. maybe i just pass two drives through and raid 1 them in there.

 

[lens06]

As far as I understand, with "virtio SCSI single" each vDisk gets its own single thread. With "virtio SCSI" 8 vDisks will share a thread.

that appears to be what im seeing too. (and how i understood it reading about it).

it also makes sense with why i can get better QD and Thread performance if i raid two virtual drives in the guest.

hopefully that also means that when i step from a 10 year old xeon with pcie3 to a modern epic and enterprise nvme gen 4... i see a few extra 100s of mb/s.

i guess worst case i can do a pass through. its just dirty and i need to do two because i want fail safe redundancy

 

[gfngfn256]

i have a proxmox 6.4-15 box

If its not to much to ask, (maybe other difficulties in upgrading), could you try your testing on a fully updated / new Proxmox install. I really would be interested in seeing the differing results.

 

[lens06]

If its not to much to ask, (maybe other difficulties in upgrading), could you try your testing on a fully updated / new Proxmox install. I really would be interested in seeing the differing results.

i think i will be able to. worse comes to worse and its not good enough i will stick two more drives in and pass them through so this app can run with the bandwidth it needs. in between i should be able to benchmark proxmox 8.what ever on the new hardware.

then possibly on the old hardware.

The idea is leave everything as is untill i get the new box. Set up the critical ap on top of proxmox on the new box then slowly load it till its running everything.

when that happens its time to upgrade the old box so it can be stand by. give me a few weeks

 

[lens06]

If its not to much to ask, (maybe other difficulties in upgrading), could you try your testing on a fully updated / new Proxmox install. I really would be interested in seeing the differing results.

well, ive got a poweredge r7615 on order as of today.

should have it on the 3rd of may

storage Hardware wise/Rough performance breakdown (incase this helps anyone):

• i ended up ditching direct attach for the perc12 (965i)
  • perc12 to host is pci-e 4.0 and x16 = theoretical 32GB/s
    • perc 12 cache is ddr4 3200 which iirc is around 27GB/s
  • per drive you get x2 PCIe 4.0 lanes
    • each PCIe 4.0 drive ends up capped at 4GB/s
    • when i was going to do direct attach i was going to do 2x 6.4tb drives in raid 1. Due to the x4 being halved i opted for two 3.2TB drives in raid 10 to hopefully recoop the speed with a stripe. we shall see.

I would expect the storage subsystem manages 6-8GB/s sequential and at least 20GB/s on a controller cache hit I will report back actual numbers when i have them and then report what fraction LVM pool vd gives me.

in current system testing:
I notice if i do the write back page cache on my existing ancient box i do 12GB/s for a page cache hit on the VD. Thats on DDR4 2133 on a Xeon E7v3. IIRC like 17GB/s is theoretical max for that memory so 12GB/s virtual drive cache hit is not too shabby. I cant wait to see how it scales with the Epyc 9xxx line.

I will report back what it looks like