Proxmox VE 7.4 extremely poor performance

Can 7.3 participate in a cluster with 7.4 does anyone know? I have another HP server that I could bring in for testing and set it up with 7.3.
 
maybe we should go back to basics.

Please post your vmid.conf. Also useful would be a description of your hardware.


Yes. but that would only impact migrations, not performance.
Hardware description is in the original post unless you need more info than that.

Here is A vmid.conf file for one machine.

agent: 1
boot: order=virtio0;ide2
cores: 4
cpu: host
ide2: none,media=cdrom
machine: pc-i440fx-7.2
memory: 16384
meta: creation-qemu=7.2.0,ctime=1680298352
name: w10Dave
net0: virtio=C2:F8:7F:70:42:0E,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsihw: virtio-scsi-single
smbios1: uuid=102b3a93-0191-4d34-aa22-3152db14aee4
sockets: 2
vga: qxl,memory=32
virtio0: store:101/vm-101-disk-0.raw,discard=on,iothread=1,size=953872M
vmgenid: 236fdd91-be43-4ac1-b610-b2b00dc6f268
 
Last edited:
rdware are two HP DL380 Gen 9 servers with 128GB of ram with a couple of mirrored 1TB hard disks for booting the OS (the servers came with the drives so we just used what we had - overkill but whatever). The storage they are connected to is a TrueNAS machine running on one of the Dell R510s with a Perc H200 in IT mode with a ZFS Raid Z1 with 10x 3TB drives and a Cache VDEV on a 120GB SSD.

UPDATE: I moved one of the VMs to the local storage eliminating the shared storage and it is still very slow so now I'm at a loss - Windows 10 VM running on the HP hardware and local RAID 10 and very slow.

Apologies, didnt see that. What kind of disks are in the local raid10?
 
Would it slow to that bad though? I am talking almost instant response on the working machine (with 10k rpm drives) to over a minute just to sign in on the slower machine. And I believe the one that is working is using RAID 5 on an HP embedded raid controller.
 
the only thing I see that gives me pause is the 2 sockets in your config. Set it to 1, and enable NUMA. It doesnt look like ram balooning is enabled, but double check that it isnt.

beyond that, the slowness you describe is probably in the guest itself. to verify, install a fresh windows vm and compare the behavior.
 
  • Like
Reactions: leesteken
How many similar VMs with 2x4 cores and 16GB of memory do you run on 128GB and how many physical cores? Why the two virtual sockets without enabling NUMA? Does your system have a NUMA and/or multiple sockets? Note that ZFS takes half of your memory until you limit and Proxmox works best when the VM allocations are less than 80% of memory.
 
the only thing I see that gives me pause is the 2 sockets in your config. Set it to 1, and enable NUMA. It doesnt look like ram balooning is enabled, but double check that it isnt.

beyond that, the slowness you describe is probably in the guest itself. to verify, install a fresh windows vm and compare the behavior.
Yes, I believe so - I'm not running Windows 10 on the one that I'm not having problems on either. It is Windows 7 and Windows XP guests - and I just installed a Windows 7 guest on this one and it's lightning fast as well - so it's something with the Windows 10.

The multiple sockets was another person trying to throw virtual hardware at the problem to see if that was part of it. I've heard it's better to do a single socket with multiple cores.
 
How many similar VMs with 2x4 cores and 16GB of memory do you run on 128GB and how many physical cores? Why the two virtual sockets without enabling NUMA? Does your system have a NUMA and/or multiple sockets? Note that ZFS takes half of your memory until you limit and Proxmox works best when the VM allocations are less than 80% of memory.
The majority are 1x4 cores and most are between 4 and 8gb of memory - only the servers are allocated up to 32gb. We have 64gb allocated to the SQL server we are running.
 
The majority are 1x4 cores and most are between 4 and 8gb of memory - only the servers are allocated up to 32gb. We have 64gb allocated to the SQL server we are running.
I read your reply as a few Windows Servers with 32GB plus one with 64GB plus several desktop VMs with on average 6GB. That way more than the 62GB you have available per host (after 64GB for ZFS and 2GB for Proxmox).
If you don't want to share the exact numbers, maybe you can add the allocated memory yourself and check the zfs_arc_max? Proxmox does not work well when over-committing on memory.
If you have already tried running just a single (small) VM on a host and that was still very slow, then it's probably not the memory (yet). Apologies if you already ruled this out and I missed it in this thread. I'm just a random stranger on the internet and guessing here.
 
I read your reply as a few Windows Servers with 32GB plus one with 64GB plus several desktop VMs with on average 6GB. That way more than the 62GB you have available per host (after 64GB for ZFS and 2GB for Proxmox).
If you don't want to share the exact numbers, maybe you can add the allocated memory yourself and check the zfs_arc_max? Proxmox does not work well when over-committing on memory.
If you have already tried running just a single (small) VM on a host and that was still very slow, then it's probably not the memory (yet). Apologies if you already ruled this out and I missed it in this thread. I'm just a random stranger on the internet and guessing here.
The proxmox host itself is not running ZFS. It's running on the shared storage - different server.

I have a bit more than 128 gb apparently. This is according to proc/meminfo: MemTotal: 198012156 kB
 
  • Like
Reactions: leesteken
I have followed the best practices document for windows 10 guests. It still is orders of magnitude slower than the test windows 7 vm I set up. Right now we're running a Windows 10 debloatware app on it to see if that does anything.
 
What local storage controller on your HP DL380 Gen 9 ?
is host use swap host ?
 
pretty unlikely. the forums are full of user reports of windows 10 guests without issue; I myself dont virtualize windows workstation guests so I cant speak directly.
I can't speak to Proxmox and Windows 10 but I can say for certainty that it is an issue on VMWare ESXi and I can say that in this install the finger is starting to point strongly in Windows 10's direction. I am about to do a fresh install of Windows 10 on my 7.3 install and see if I have similar issues.
 
What local storage controller on your HP DL380 Gen 9 ?
is host use swap host ?
Just the integrated RAID controller, but it is not in the design for the VMs to run off the internal storage. There is a 10GE nic on a separated 10GE network going to a TrueNAS NFS share for shared storage. We moved it to local storage to attempt to eliminate the shared storage as a possibility.

Not sure what you're asking about swap host. I believe the host has swap space as it shows on the summary screen that swap is being used. My other install that is not having issues is not using any swap.
 
you need to double check if system is faster with local storage or not.
Do you use embedded SATA or hp Smart Array ?
what hp Smart Array model ?
is FWCB is present ?
is your couple of mirrored 1TB hard disks for booting the OS setup with zfs or hw controller ?
HP disable disk writecache on their ProLiant (embedded SATA and HW controller)
so without FWCB local storage is slow but secure by default.
if host use many swap on an unproperly setup of local storage, it will be slow too.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!