IO Delay help/advice

[Cecil]

Hey Everyone

So I have a server with a raid controller with 8Gb cache onboard and some nearline SAS 7200 (yes i know they are not the best drives) in raid 5.
Server is an ibm SR550
The max disk performance sustained is about 150mb/s read and write.

My problem is that when I restore a 300Gb database to my sql VM the IO delay shoots through the roof and all the other VM's start to become non responsive.

This machine comes with a M.2 PCI-E interface slot and I'm thinking of installing a 250 or 512gb SSD there and setting it up as the LVM cache drive but I'm worried that once the cache is full on that the io delay will just appear again.

Is there anything I can do to make sure a single vm doesn't kill all the others?

 

[fireon]

What do you use? LVM, or LVM-Thin for your VM's. What says:

pvesm status
pveversion

 

[Cecil]

What do you use? LVM, or LVM-Thin for your VM's. What says:

pvesm status
pveversion

I think its lvm-thin, I'm not 100% sure how I even set it up, I think I just followed the defaults in the guide.
My storage config looks like this.
dir: local
path /var/lib/vz
content iso,backup,vztmpl

lvmthin: local-lvm
thinpool data
vgname pve
content images,rootdir

dir: backup
path /media/usb
content backup
maxfiles 2
shared 0


pvesm status:
Name Type Status Total Used Available %
backup dir active 98559220 34049940 59459732 34.55%
local dir active 98559220 34049940 59459732 34.55%
local-lvm lvmthin active 19405156352 4373922241 15031234110 22.54%


pve-manager/5.1-41/0b958203 (running kernel: 4.13.13-4-pve)

 

[fireon]

Ok, thank. So think yes it is lvm-thin. Lvm-Thin can be very slow on faulty Raidcachemodul, or faulty cachratio. Also it is not the best solution for older servers. Like 5 years and older. Can you also give me the output of "pveperf" please?

 

[Cecil]

This server is brand new, It just came to market in September 2017. In fact it took a while because the Rain controller was even supported by the proxmox kernel.
I've also done one round of updates on the controller firmware. It's kinda of a pain since IBM only supports windows and red hat for easy updates, my only way of updating it now is making a red hat boot disk with the update packs and rebooting the entire server to do that so I can only really do it on weekends.

pvperf output:
root@pve:~# pveperf
CPU BOGOMIPS: 134400.00
REGEX/SECOND: 2363028
HD SIZE: 93.99 GB (/dev/mapper/pve-root)
BUFFERED READS: 568.34 MB/sec
AVERAGE SEEK TIME: 6.71 ms
FSYNCS/SECOND: 72.96
DNS EXT: 106.16 ms
DNS INT: 0.88 ms (raeeng.local)

My concern is that I can add a large amount of cache but will that just delay the IO problems by a few minutes until the cache fills up?

 

[fireon]

This server is brand new, It just came to market in September 2017.

Sorry about haven't searched for in the internet. I know this issue only on older server and damaged hardware.

Your Fsyncs/Second value is really bad. You should have 3000 and more for an normal working system. It's really hard to say what's going wrong, but yes, do an firmwareupdate and check the hardware. Maybe after update it is working fine. I'm pretty sure that LVM-Thin with Redhat does also not working fine. For me it look like an hardware or hardware/firmware configuration issue.

It's kinda of a pain since IBM only supports windows and red hat for easy updates

Really? No onlineupdate with some hardwaremanaged interface? On DELL this works online from IDRAC, on HP there are also rpm-packages for redhat, but they can also be used in pve to update raidcontroller and other things. Strange IBM. Supermicro much more better then they all.

 

[Cecil]

Ah well that gives me somewhere to start looking, thanks!
*edit*
So I checked my Raid controller and it was set for write-through instead of write-back..
With the write-back setting im getting:
FSYNCS/SECOND: 2886.22

Yeah some updates can be done through web gui but not for that raid controller and a few other ones

 

[fireon]

FSYNCS/SECOND: 2886.22

YES

So test it again. It should work now fine with lvm-thin too. I'am already curious!

 

[Cecil]

So this sort of helped but when dealing with really large amounts of data IO delay still goes to 25% and everything gets crazy slow.
The only thing that really causes it for now is when I restore a database that's 100+ Gb big into SQL Server because it causes both read and write IO.

So back to my first question, lets say I add a proper enterprise 512GB SSD on the m.2 slot will it absorb the spike in read/writes and then slowly trickle the data back without IO delay or will it still try and overwhelm the hdd's with the data and cause the same IO delay?

When I say add as cache I mean to use:
lvconvert --type cache --cachepool
kinda like this example ( https://rwmj.wordpress.com/2014/05/22/using-lvms-new-cache-feature/ )

 

[fireon]

I've never worked with that. Maybe some other

 

[Andrew Kobelkov]

Restoring a VM to the lvm-thin storage is a well-known headache, even on SSD.
https://forum.proxmox.com/threads/huge-io-delay-when-recovering-vm.38703/
https://forum.proxmox.com/threads/restore-vm-slowdown-host.38326/
Try to use cstream:

cstream -t 30000000 -i /mnt/pve/nfsvzjhb4/dump/vzdump-qemu-102-2017_12_06-19_54_59.vma.lzo | lzop -cd | qmrestore - 102 --force --storage storage_name

30000000 means 30 MB/s

 

[Cecil]

Restoring a VM to the lvm-thin storage is a well-known headache, even on SSD.
https://forum.proxmox.com/threads/huge-io-delay-when-recovering-vm.38703/
https://forum.proxmox.com/threads/restore-vm-slowdown-host.38326/
Try to use cstream:

cstream -t 30000000 -i /mnt/pve/nfsvzjhb4/dump/vzdump-qemu-102-2017_12_06-19_54_59.vma.lzo | lzop -cd | qmrestore - 102 --force --storage storage_name

30000000 means 30 MB/s

Thanks for the advice, my situation is slightly different though, I'm not recovering the whole VM but recovering a database from inside the VM.
So using the info you provided would it help then to set disk throttle limits on the VM like this: