How to REALLY Improve Disk Performance?

[Fatz]

I've been benchmarking some server configurations, and i find that a baremetal Debian installation with mdadm tested with fio, tends to be 2-2.5x faster than setting up an array in the same manner under Proxmox. This is even with things like 'Write Back' enabled.

Are there some ways to really improve the speed of disks under Proxmox beyond the usual suggestions such as that above? I don't understand how a RAID configured in the exact same manner on baremetal can be so much quicker than under Proxmox on the same host.

I understand ZFS is recommended, and i use that on some hosts. But i'm trying to build the fastest host possible (as far as storage goes) and from what i've seen, mdadm would be a better fit. But as i say, Proxmox seems to be hampering this too, and i'd rather not just run it as baremetal outside of my cluster.

Note: I've tested this outside of fio benchmarks also, and found the real life performance of my databases to be noticeably lacking too, which is why i began benchmarking so much.

Many thanks for reading.

 

[_gabriel]

true numbers and specs ?

 

[zzz09700]

Hm, I know I'm not in the party but here's what I got from a test VM.

root@u2004-test1:~# dd if=/dev/urandom of=/root/test1.img bs=1G count=4 oflag=dsync
4+0 records in
4+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 23.8894 s, 180 MB/s

qcow2 on one single Sandisk SSD Plus and the numbers are pretty much what you get by squeezing the disk dry.
Here's a screenshot of the VM's disk configuration, if anyone is interested.


So what kind of "slow" are you guys talking about? Personally I don't see much performance difference between PVE and ESXi (i.e. 4K random generally sucks, especially with low QD, sequential r/w is pretty much okay and, of course, there is always resource contention).
If you do need crazy disk performance then try passthrough the disk or the disk controller to VM.

 

[Fatz]

true numbers and specs ?

I was more asking in a general sense in case there's features/options that are often overlooked that i could try further tests with. I would need to dig up my notes, but here's an example of 12x 4TB Inland nvmes in Debian 12 tested with yabs.sh (so, fio) on an mdadm (default config) array:

Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ----
Read       | 641.66 MB/s (160.4k) | 8.36 GB/s   (130.7k)
Write      | 643.35 MB/s (160.8k) | 8.41 GB/s   (131.4k)
Total      | 1.28 GB/s   (321.2k) | 16.77 GB/s  (262.1k)
           |                      |
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ----
Read       | 31.92 GB/s   (62.3k) | 35.60 GB/s   (34.7k)
Write      | 33.61 GB/s   (65.6k) | 37.97 GB/s   (37.0k)
Total      | 65.53 GB/s  (127.9k) | 73.58 GB/s   (71.8k)

... i don't have the results to hand from the runs i did on Proxmox, but i remember specifically that the 1m total was ~32GB/s, so a bit less than half vs running on straight Debian.

 

[Fatz]

Hm, I know I'm not in the party but here's what I got from a test VM.

root@u2004-test1:~# dd if=/dev/urandom of=/root/test1.img bs=1G count=4 oflag=dsync
4+0 records in
4+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 23.8894 s, 180 MB/s

qcow2 on one single Sandisk SSD Plus and the numbers are pretty much what you get by squeezing the disk dry.
Here's a screenshot of the VM's disk configuration, if anyone is interested.
View attachment 68614

So what kind of "slow" are you guys talking about? Personally I don't see much performance difference between PVE and ESXi (i.e. 4K random generally sucks, especially with low QD, sequential r/w is pretty much okay and, of course, there is always resource contention).
If you do need crazy disk performance then try passthrough the disk or the disk controller to VM.

Yeah, i didn't try passthrough or look in to SR-IOV etc yet. But the thing is that since Proxmox is based on Debian, i was wondering if there might be something that is hampering it from the offset. Typically hypervisors shouldn't be causing so much overhead, but maybe at these kind of speeds some cracks are beginning to show?

I've generally been happy with Proxmox on lesser systems i've run it on, but obviously i'd like to avoid leaving 50%+ of my storage throughput on the table with this host.

 

[zzz09700]

Typically hypervisors shouldn't be causing so much overhead

Where do you get this idea? Of courses there are hypervisors overheads in io operations. How much overhead is there depends on actual setup, with some crazy disk setup such as 12x 4TB Inland NVMEs the CPU could very much be a problem.

 

[Fatz]

Where do you get this idea? Of courses there are hypervisors overheads in io operations. How much overhead is there depends on actual setup, with some crazy disk setup such as 12x 4TB Inland NVMEs the CPU could very much be a problem.

The CPU typically seems untaxed whenever i monitor it while testing. It's also a high clock in this specific case (Threadripper PRO 5975WX, 512GB RAM) so i think whether multi or single threaded is king shouldn't matter too much.

And it still doesn't explain away the fact that Debian outperforms Proxmox by more than double. I never said that there isn't overhead, but really, 50%+? Should i typically just forget about using Proxmox with more bleeding edge setups then?

 

[alexskysilk]

i find that a baremetal Debian installation with mdadm tested with fio, tends to be 2-2.5x faster than setting up an array in the same manner under Proxmox

that is silly. Proxmox IS "bare metal" Debian. perhaps you have different existing loads on the system?

Are there some ways to really improve the speed of disks under Proxmox beyond the usual suggestions such as that above?

You need to consider what you are actually after. "speed" is not relevant in and of itself; just as alluded above, the disk subsystem needs to be able to satisfy its load with adequate tolerances, so "performance" is only of use as a metric when taken as a function of what you are asking of it.
example: you have a single, high speed stream required with a minimum required bandwidth of 200MB/S to satisfy the client. best solution here is a single wide stripe- the benchmarks can show over 1GB/S on sequential read tests with relatively modern hardware- even hard drives. This will provide great performance for that use case- . If you then try to use it for virtualization it will perform quite poorly.

But why is it slower then [insert subjective number]: who cares. design the storage to fulfill your NEEDS, not an arbitrary benchmark.

 

[Fatz]

that is silly. Proxmox IS "bare metal" Debian. perhaps you have different existing loads on the system?


You need to consider what you are actually after. "speed" is not relevant in and of itself; just as alluded above, the disk subsystem needs to be able to satisfy its load with adequate tolerances, so "performance" is only of use as a metric when taken as a function of what you are asking of it.
example: you have a single, high speed stream required with a minimum required bandwidth of 200MB/S to satisfy the client. best solution here is a single wide stripe- the benchmarks can show over 1GB/S on sequential read tests with relatively modern hardware- even hard drives. This will provide great performance for that use case- . If you then try to use it for virtualization it will perform quite poorly.

But why is it slower then [insert subjective number]: who cares. design the storage to fulfill your NEEDS, not an arbitrary benchmark.

Well actually as far as i understand it, Promox uses a modified kernel, so i was wondering if therein laid the issue or if there was some other element of its hypervisor features that could be getting in the way. I've tested all i can think of including checking/changing things such as the scheduler, cache method, etc. Hell, even my 'testing' being done directly on the host (not inside a VM) with zero workloads or VMs running is already unrealistic, because of course the IO cycles are going to be split between multiple VMs eventually.

That said - you're right about needing more specific testing. But as i mentioned originally, i have tested this on the real-life (databasing) usecases i have in mind for this system, and exactly as with the fio tests, it's performed worse under Proxmox. To your first point - when running the other synthetic tests, i had no other workloads running.

 

[zzz09700]

50% overhead

The CPU typically seems untaxed whenever i monitor it while testing. It's also a high clock in this specific case (Threadripper PRO 5975WX, 512GB RAM) so i think whether multi or single threaded is king shouldn't matter too much.

And it still doesn't explain away the fact that Debian outperforms Proxmox by more than double. I never said that there isn't overhead, but really, 50%+? Should i typically just forget about using Proxmox with more bleeding edge setups then?

50% overhead is, contrast to many people would believe, pretty common in system-call heavy applications running inside a VM.
That's one of reasons why Docker is so popular.

And the overall CPU be untaxed while one or two cores, or sometimes two threads from same core, can be fully occupied and causing bottleneck, we see this many, many time over already.

 

[Fatz]

50% overhead

50% overhead is, contrast to many people would believe, pretty common in system-call heavy applications running inside a VM.
And the overall CPU be untaxed while one or two cores, or sometimes two threads from same core, can be fully occupied and causing bottleneck, we see this many, many time over already.

OK then that answers my question i guess. I never ran in to such a noticeable different on any of my PVE hosts prior to this. So I suppose Proxmox won't fit for this particular usecase. That's actually what i was expecting before i made this thread, even if i hoped it wasn't the case. Can't win them all!

 

[zzz09700]

OK then that answers my question i guess. I never ran in to such a noticeable different on any of my PVE hosts prior to this. So I suppose Proxmox won't fit for this particular usecase. That's actually what i was expecting before i made this thread, even if i hoped it wasn't the case. Can't win them all!

You can consider other, environment isolation solutions including Docker. Those usually impose lower io overheads.
In my place we have VM acting as a router with Intel I350 passed through, 6 Coffee Lake Cores and 12G RAM. Some 4 core 4690S(Haswell, low power variant, if anyone can't recall) + dual Intel I210 baremetal setup beats the shit of it.

 

[Fatz]

Yeah, i actually do use some baremetal setups with services running instead under systemd/docker for other usecases. I just enjoy the flexibility of Proxmox and its features, so i prefer to fit everything i can under my clustered environment. But alas.

 

[alexskysilk]

Well actually as far as i understand it, Promox uses a modified kernel

"modified" is a strong word. its just the Ubuntu kernel since it is better maintained/updated. If you suspect your performance is a result of kernel differences your problem is with your hardware.

Of courses there are hypervisors overheads in io operations.

uhh... do you have any data to support? its not that this isnt true, but I think you are massively overstating the impact.

How much overhead is there depends on actual setup, with some crazy disk setup such as 12x 4TB Inland NVMEs the CPU could very much be a problem.

This isnt a "hypervisor" problem. its an OS problem. the impact would be the same on bare metal. also, I dont know why you would consider 12 NVMEs crazy- JBOFs are a real thing and they work just fine, without it being "very much a problem."

 

[zzz09700]

This is getting old.
Why would anyone believe system call has the same price tag with/without a hypervisor layer?
So the hypervisor layer is practically free then?

 

[alexskysilk]

Why would anyone believe system call has the same price tag with/without a hypervisor layer?

Nothing to do with belief. if you *think* something is, say so. if you're stating it as fact, have some numbers.

So the hypervisor layer is practically free then?

nothing is free- but virtio is very efficient.