Hi. I am trying to fix some IO Delay issues I seem to be having when running massive mongoDB updates on one of my VM's. It throws the entire system out of whack, and I'm trying to determine if it is proxmox or Openfiler (My SAN software) causing these issues.
My IT helped me run some tests, and we found out that the RAID card in my SAN wasn't the best and didn't allow for write-back (or write-thru, can't remember which). Anyways we replaced the RAID card and then did a few optimizations with the OS itself as far as deadline and such. We then ended up with this report:
http://serverbear.com/benchmark/2013/07/13/oQzdyEqjNWSU0gUL
You can see the I/O tab has a dd and FIO test. The dd test is good in speed performance (but still could be a bit faster). The FIO is garbage though. And this is the actual speed that my VM's seem to be getting when it comes to writing files to the SAN. When I am trying to update mongoDB with gigabytes of information this pratically brings the entire SAN down with all the IO delay and lags all the VM's (even the ones on other hypervisors). The IO Delay spikes up past the CPU % and the I/O on the VM itself shoots up to about 30MB but it doesn't actually seem to be getting those speeds.
I'm not really sure where to look at this point, so I am asking Proxmox and Openfiler people alike for their opinions and suggestions.
Thanks for any help.
EDIT: Adding a bit more of useful information:
SAN HW configuration:
1230v2 HT Processor (8 cores w/ HT)
8 gb of Ram
60 gb SSD Local OS drive
2 x 300 15k SAS HDD's in RAID 1 (iSCI targets)
2 x 1TB in RAID 1 (nfs, used for file storage only)
Proxmox Nodes:
1230v2 HT Processor (8 cores w/ HT)
8gb of RAM on one, and 16gb on another node
60 gb SSD local OS drives
1 Gigabit connection private network between proxmox nodes and SAN
Output of pveversion -v on proxmox1:
Output of pveversion -v on proxmox2:
Output of pveperf on proxmox1 (VM's still running):
My IT helped me run some tests, and we found out that the RAID card in my SAN wasn't the best and didn't allow for write-back (or write-thru, can't remember which). Anyways we replaced the RAID card and then did a few optimizations with the OS itself as far as deadline and such. We then ended up with this report:
http://serverbear.com/benchmark/2013/07/13/oQzdyEqjNWSU0gUL
You can see the I/O tab has a dd and FIO test. The dd test is good in speed performance (but still could be a bit faster). The FIO is garbage though. And this is the actual speed that my VM's seem to be getting when it comes to writing files to the SAN. When I am trying to update mongoDB with gigabytes of information this pratically brings the entire SAN down with all the IO delay and lags all the VM's (even the ones on other hypervisors). The IO Delay spikes up past the CPU % and the I/O on the VM itself shoots up to about 30MB but it doesn't actually seem to be getting those speeds.
I'm not really sure where to look at this point, so I am asking Proxmox and Openfiler people alike for their opinions and suggestions.
Thanks for any help.
EDIT: Adding a bit more of useful information:
SAN HW configuration:
1230v2 HT Processor (8 cores w/ HT)
8 gb of Ram
60 gb SSD Local OS drive
2 x 300 15k SAS HDD's in RAID 1 (iSCI targets)
2 x 1TB in RAID 1 (nfs, used for file storage only)
Proxmox Nodes:
1230v2 HT Processor (8 cores w/ HT)
8gb of RAM on one, and 16gb on another node
60 gb SSD local OS drives
1 Gigabit connection private network between proxmox nodes and SAN
Output of pveversion -v on proxmox1:
Output of pveversion -v on proxmox2:
Output of pveperf on proxmox1 (VM's still running):
Output of pveperf on proxmox2 (VM's still running):
Last edited: