Proxmox 5.3.8 - IO delay issues explained ?

[Mecanik]

Hello,

I am trying to understand what is going on, but unfortunately I cannot find the cause. To keep this issue description short, I made a new installation of Proxmox 5.3.8 using the following hardware:

Specs:

• CPU: Intel Xeon E5-1650v3 - 6c/12t - 3.5GHz
• RAM: 128GB DDR4 ECC 2133 MHz
• Disks: SoftRAID 2x2TB SATA (HGST_HUS724020ALA640)

Proxmox:

• Kernel Version: Linux 4.15.18-10-pve #1 SMP PVE 4.15.18-31 (Mon, 14 Jan 2019 10:59:31 +0100)
• PVE Manager Version: pve-manager/5.3-8/2929af8e

Created Windows VM to test:

• CPU: 1 core
• RAM: 2 GB
• HDD: 32 GB
• Windows Server 2016 x64 with latest updates

VM details:

• agent: 1
• bootdisk: ide0
• cores: 1
• cpu: host,flags=+pcid;+spec-ctrl
• cpulimit: 0.5
• ide0: local:103/vm-103-disk-0.qcow2,cache=writeback,size=32G
• ide2: cdrom,media=cdrom
• memory: 2048
• name: supertest
• net0: virtio=02:00:00:5e:4a:86,bridge=vmbr0,rate=12
• numa: 0
• ostype: win10
• scsihw: virtio-scsi-single
• smbios1: uuid=c7042f18-9fbb-4c00-ab26-6eca2969b982
• sockets: 1
• vcpus: 1
• vmgenid: 5d26efe7-cb77-43b1-97e4-76f68bf7a79a

Now the question is... why am I having IO delay ? And I`m talking about... 3/5/15/25% randomly with 1 VM running ( and doing some work like donwloading something, or installing updates ).

I know for a fact that if I had SSD drives I would not have this IO delay, but I would like to understand why is this happening in the first place.

• Is it the CPU ?
• Is it the config ?
• Is it the drives ?

Please advise.

 

[FastLaneJB]

You've a software RAID with just 2 spinning disks. The IO delay is because of the disks.

 

[Mecanik]

You've a software RAID with just 2 spinning disks. The IO delay is because of the disks.

That was my suspicion, but I needed to be absolute sure. So there is no "heal" for this at all ? I`m trying not to replace with SSD... how about 3 x disks ?

Or ZFS would help ?

Appreciate it.

 

[tom]

You can think of using ZFS mirror and add a really fast and reliable SSD as ZIL.

Better: use SSDs only.

 

[Mecanik]

You can think of using ZFS mirror and add a really fast and reliable SSD as ZIL.

Better: use SSDs only.

Thank you tom, however SSD's are much more expensive. And the company where I buy these from usually sell 2x480 gb in softraid automatically configured.

I am trying to find a solution to use these cheaper disks and not have IO delay.

 

[tom]

Thank you tom, however SSD's are much more expensive.

The initial purchase prices, yes. If you compare price vs performance, no.

So in almost all cases, SSDs are the way to go for new projects for hosting VM/CTs.

 

[FastLaneJB]

I don't think you can fix this. Older versions of Proxmox were better but that's only because you could easier configure them to lie about commits to disk. You could likely achieve the same on Proxmox 5 by just setting your KVM VM caching to Unsafe. Just be aware if you have a power cut or crash you'd lose some writes in those instances.

Performance isn't worse really here in Proxmox than if you installed Windows 10 directly on the native hardware. Just Windows isn't making it obviously about your IO Wait is all. The CPU is still going to be sat there wasting cycles waiting for your slower spinning disks to supply it with the data it needs.

If performance is still acceptable you could ignore the IO Wait. However I expect you can only run a few light IO VM's before you'll hit major performance issues. As they say above, you really do want some SSD's these days.

 

[Mecanik]

I don't think you can fix this. Older versions of Proxmox were better but that's only because you could easier configure them to lie about commits to disk. You could likely achieve the same on Proxmox 5 by just setting your KVM VM caching to Unsafe. Just be aware if you have a power cut or crash you'd lose some writes in those instances.

Performance isn't worse really here in Proxmox than if you installed Windows 10 directly on the native hardware. Just Windows isn't making it obviously about your IO Wait is all. The CPU is still going to be sat there wasting cycles waiting for your slower spinning disks to supply it with the data it needs.

If performance is still acceptable you could ignore the IO Wait. However I expect you can only run a few light IO VM's before you'll hit major performance issues. As they say above, you really do want some SSD's these days.

Thank you, I appreciate it. I'm trying to see what drives I can find affordable...

 

[Mecanik]

Just one last question, if I was to still use normal HDD instead of SSD but I would have like 4 x HDD in softraid, the IO delay issue would still be there ?

Thanks

 

[RedneckBob]

Thank you, I appreciate it. I'm trying to see what drives I can find affordable...

I know zero about your environment and depending on how "affordable" you want, look on ebay for Intel DC SSD drives. These are datacenter rated drives and unless they were seriously abused, likely you'll never wear them out. Ask the sellers to send you the SMART data and confirm the drives are still under Intel warranty. Be wary of anyone who won't send your the SMART data or let you check the warranty at Intel.

I've stuffed all my homelab machines full of these things as I've found fantastic deals. Most all are S3500s but I have ZFS array with a bunch of Samsung datacenter SSDs.

For instance, a 1.6T MLC Intel S3510 (DC) SSD for $270 (https://www.ebay.com/itm/New-Intel-...A-III-6-0Gb-s-SSD-SSDSC2BB016T6P/163312085202).

Only had one vendor ship me a drive that had some serious hours and read/writes, but it still had 98% of it's health and has run for last year with no issues.