HD Passthrough or more than 2 hostPCI

[seer_tenedos]

I am running a NAS (freenas) inside proxmox. I have done this for over a year in ESXI but i have descided to make the switch to Proxmox because it seems to give much better performance and i love the web interface. The issue i am having is that i have 2 pcie sata controllers for my storeage drives and these passthough correctly via hostpci config in my VM and all the 10 disks are available as expected. The issue comes with the 2 SSD cache drives. If i connect them directly to the sata controllers i passed though they don't show up and since i did not have them in this config in ESXI then this could be an issue with the cards and SSD support. This means that i either need to pass though the HD's directly or pass though one of my 3 onboard sata controllers but that would be the 3rd hostpci device and only 2 are supported.

I saw an article around somewhere talking about disk passthrough as
ide1: sda
in my case i tried
sata0: sdl
sata1: sdn
and
sata0: /dev/sdl
sata1: /dev/sdn
but both gave me the error of device or addess not found. I confirmed they are in my /dev directory and i am 95% sure they are my SSD's

Can anyone point me to the best way to pass though these 2 SSD drives as i want to get max performance i can fromt hem in a virtual environment. I used HD passthrough on ESXI and it actually worked really well for this use case but i have not been able to get anything to work in proxmox.

I am using KVM VM and 3.0 RC2 of proxmox.

 

[seer_tenedos]

solved it. turned out to be an issue with the connection on the raid card. Now i can get the SSD's working on the 2 cards i am already passing through

 

[seer_tenedos]

solved it. turned out to be an issue with the connection on the raid card. Now i can get the SSD's working on the 2 cards i am already passing through

seems i may have spoken too soon. Now it is getting strange. Exactly 10 HDD will pass though and i can't work out any reason for all 12 not to pass through. 1 SSD and 1 normal HDD is missing. Tried switching cables etc in case that was the problem but the 2 same disks stay missing. Rebooting host does not help and checked power cords and they seem fine.

Anyone have nay ideas on what i can try next? i am thinking to try removing another HD and see if one of the missing ones shows up to confirm if it is a HD issue or a limit of 10 disks for some reason. The fact that it is a newish SSD that is one of the disks and that the SSD has been fine under ESXI makes me think it is not a disk issue but i have not idea of where to go from there.

 

[seer_tenedos]

this one just gets stranger and stranger. It seems i lose between 2-3 Hdd and that if i check the dmsg on debian after proxmox boots and can see that proxmox did not detect the drives itself before it passes them through. One of my controllers missed 1 disk and the other controller misses 1-2 disks. I am using 2 IBM ServeRAID M1015 with an updated firmware that was recommend to remove the raid and get max performance if you are using software raids like ZFS. This has been working for over a year in ESXI but i did not have the 2 SSD's connected to it as well.

Each card has 2 ports for 4 port min sas sable and on each controller there is at least 1 HD working off each of the cables to suggest they are not at fault. Just in case i change the cables on some of the working drives to the ones i think are failing and get the same result. The same HD fails even on the cable that was working before. This would normally make me think it was the HD but to lose 2-3 HD's of different brands, mix of SSD and HDD on the same night right when i change to proxmox is very unlikely.

Hoping someone can point me in the right direction to try to resolve or isolate the issue. The other issue i have is that i can't reboot very often as i resell my internet and my firewall is a VM. My users are already complaining about the outages. Trying to limit my tests to 2-3 full reboots a day at the min.

 

[seer_tenedos]

[Solved]Re: HD Passthrough or more than 2 hostPCI

Finally worked it all out. Seems the last port of the first card does not work for some reason. The other issue is that one of my mini sas to sata cables is just plan dodgy and all the drives on that cable drop in and out. suspect crappy broken wires or something. Changed it over and all is good again.

Makes me want to get a case with a proper multiline hotswap and sas expanders to stop this sort of issue. Noticed SAS expanders are finally out and are now only 200-300. I remember looking before and they were 3k and almost impossible to find.

 

[kobuki]

Re: [Solved]Re: HD Passthrough or more than 2 hostPCI

Sorry to barge in with a kinda offtopic problem, but since I'm doing the same with the same card (single one), I thought I'd risk asking. Do you experience something similar to what I posted earlier? -> http://forum.proxmox.com/threads/13868-Intel-IOMMU-problems-in-3-0RC2