Ram Benchmarking Speed (with processing) declined on moving to Proxmox from ESXI on same hardware

[nimishbansal]

On ESXI, I was benchmarking my RAM sequential read speed(with some processing) using following commands

mkdir /benchmark
esxcli system visorfs ramdisk add -M 4096 -m 1024 -n benchmark -t /benchmark -p 0755
head -4100000 /dev/random > /benchmark/random.txt
date;wc /benchmark/random.txt;date;

This got completed in 11 speed.

But when I write similar benchmarking script for proxmox, it is taking about 42 seconds.

mkdir /benchmark
mount -t tmpfs -o size=4G tmpfs /benchmark
head -4100000 /dev/random > /benchmark/random.txt
date;wc /benchmark/random.txt;date;

What could cause such a big difference?

 

[zzz09700]

You do understand reading from /dev/random can be blocking, right?
The chances of blocking are actually much higher for VMs.

There are reasons why you got /dev/urandom and /dev/zero out there.

 

[zzz09700]

Also I don't think measuring memory speed inside a VM that's backed by VT-x/AMD-V has any practical value.

You are just measuring if VT-x in the CPU is capable of saturating memory bandwidth, and in most cases the answer is a simple yes so VMs have no performance penalty in large memory read/write operations.

There could be performance problems when multiple VMs compete for cache lines and hardware interrupt (IO operations), but that obviously not what OP is after, yes?

 

[Ramalama]

The faster one is directly on the Host, tested simply with Hirens Boot CD, so no drivers, probably not max speed, dunno.
The Second (Slower one) is inside a WS2019 VM, with all drivers etc...

There is definitively a big difference, but in my case, thats anyway all that fast that it simply doesn't matters.
Cheers

 

[nimishbansal]

@zzz09700 I am not measuring the speed on the VMs, I am measuring on the ESXI host and Proxmox host directly.

You do understand reading from /dev/random can be blocking, right?

But I am doing "wc" on already saved text at "/benchmark/random.txt"

 

[Ramalama]

@zzz09700 I am not measuring the speed on the VMs, I am measuring on the ESXI host and Proxmox host directly.


But I am doing "wc" on already saved text at "/benchmark/random.txt"

Ah thats a different story. But then i believe that the benchmark itself acts differently, it could be that random on vmware is actually urandom.
can you retest with /dev/urandom on both?

/dev/random is known to be slow, and as far as i know, its not even anywhere used on proxmox.

 

[nimishbansal]

When I completely eliminated /dev/random or /dev/urandom from the picture and used the same file containing only the English words.

wc on same file(approx 780 MB) took 3 seconds in ESXI but around 5 seconds on proxmox

 

[justinclift]

Maybe a CPU affinity problem?

 

[Ramalama]

When I completely eliminated /dev/random or /dev/urandom from the picture and used the same file containing only the English words.

wc on same file(approx 780 MB) took 3 seconds in ESXI but around 5 seconds on proxmox

didnt seen that, youre right.
What platform is that? i have strange issues with Genoa and hyperthreading here, mentioned in another thread, but i still had no chance to debug further, its simply in production (makes hard to experiment with)
The hyperthreading issue i have, doesn't exist on any Intel Hardware (any xeon) or Ryzen 5800x (the only other amd pseudo server i have)
I have the issue on both Genoa Servers, so its either Genoa related or Epyc (Milan/Rome) either.

I have like 12 Servers, but only one of them is AMD (actually 2x Genoa) lol, all others are Xeons, so i could only test on Consumer Zen3 as the only other AMD System.

You have some Epyc?