migrate vm with local devices ?

[whitewater]

Hello everybody,

I have a VM with a local device. it's a pci passtrough with hostpci argument.
I would to apply HA on it.

If a migrate it, i have this : "can't migrate VM which uses local devices".

the qm manual show a -force option to do migrate of vm with local device :
http://pve.proxmox.com/wiki/Manual:_qm

qm migrate <vmid> <target> [OPTIONS]
Migrate virtual machine. Creates a new migration task.

<vmid> integer (1 - N)

The (unique) ID of the VM.

<target> string

Target node.

-force boolean

Allow to migrate VMs which use local devices. Only root may
use this option.

-online boolean
Use online/live migration.

it'is possible to apply the -force option to a VM with HA ?

thank you for your answer.
Eric.

 

[dietmar]

I have a VM with a local device. it's a pci passtrough with hostpci argument.
I would to apply HA on it.

If a migrate it, i have this : "can't migrate VM which uses local devices".

This VM uses a local PCI device, so you cannot migrate it and HA will not work
because the local PCI device is not available on any other node.

 

[whitewater]

Hello dietmar,
thank you for your answer.


I have forgotton to mention that the two proxmox server are identical. It's two supermicro whith the same hardware. same Ethernet cards added too.


Here my vm config :

boot: cdn
bootdisk: virtio0
cores: 1
hostpci0: 02:00.1
hostpci1: 04:00.0
ide2: none,media=cdrom
memory: 1024
name: pfsense-test
net0: virtio=8A:62:7C:08:AB:4E,bridge=vmbr0
ostype: other
scsihw: virtio-scsi-pci
sockets: 1
tablet: 0
virtio0: drbdr0:vm-10899-disk-1,size=40G

Here lscpi command from the two servers :

root@proxmox01:~# lspci | grep Eth
02:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
02:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
05:00.0 Ethernet controller: Intel Corporation 82598EB 10-Gigabit AT2 Server Adapter (rev 01)
root@proxmox01:~#

root@proxmox02:~# lspci | grep Eth
02:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
02:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
05:00.0 Ethernet controller: Intel Corporation 82598EB 10-Gigabit AT2 Server Adapter (rev 01)
root@proxmox02:~#

such as network cards are the same, I think the migration with local devices is possible.

 

[udo]

such as network cards are the same, I think the migration with local devices is possible.

Hi,
but there are different MAC-Adresses on the ethernet-card! Ok, you can load your own mac-address to the card, but during live-migration the OS don't know that this must done another time...

I guess, this don't work. But what speech again simply to use bridges? Hmm - i see pfsense... (don't have experience with performance nics on pfsense).

Udo

 

[whitewater]

Hi Udo,
MAC address doesn't important for this. For exemple, if i run the VM on the other node, the VM and the internet connection on the two lan will work.

 

[udo]

Hi Udo,
MAC address doesn't important for this. For exemple, if i run the VM on the other node, the VM and the internet connection on the two lan will work.

With live-migration? Or do you mean if you start the VM on the sond node? (Yes - in this case the OS know the "new" mac-address).
And if you don't flush the arp-cache it's can't work - all traffic inside this network was send to the old ethernet-card's until they (the sender) updates the arp-cache.

Udo

 

[whitewater]

I t start the VM on the second node for test :
- i remove "hostpci" line in vm configuration.
- do a migration to the other node.
- add the "hostpci" line and start the vm.

i did not think about the arp cache.
for testing, it isn't possible to add the "-force" option ?

what it the best way to have pfsense HA on proxmox ?
Maybie by using CARP ? it'is possible to use pfsense CARP with PPPoE adsl link (for ipsec vpn, with public ip on pfsense) ?

 

[dietmar]

I have forgotton to mention that the two proxmox server are identical.

That does not matter at all (you simply can't move the physical state of the card to another node).

 

[whitewater]

thank you Dietmar.
I understand it's impossible if the vm is running, but what does if is stoped ?


I would like to use migration of local device in HA cluster, if it is possible like this :


- 3 nodes. node A, B and C. A & B running VM on drbd shared storage. C only arbitrary host for fencing.
- pfsense VM run on node A with 2 local devices, hostpci0 and hostpci1.
- pfsense VM run with HA.
- if node A have a hardware failure, pfsense VM on node A is stopped,
and HA will try to start pfsense VM on node B.
- node B have same hardware. So, VM Should start with same hostpci0 and hostpci1 arguments.


if maybie it's possible...

 

[athompso]

Do you need to use PCI passthrough for performance?
If not, stop using it and create a new bridge that connects only that card to only that VM, and migration will then be allowed.
Generally speaking, I have never thought it made any sense to virtualize a system, and then tie it to a specific piece of hardware...
-Adam

 

[whitewater]

Hello, Athompso,
First, Happy New Year 2014

I need it because i haven't found a solution to use pfsense with pppoe and ipsec connection with bridge network
http://forum.proxmox.com/threads/17340-How-to-well-done-a-pfsense-migration-to-proxmox

thank you.

 

[whitewater]

Happy New Year 2014 for everybody too

I had done this test.
pfsense vm is 108. on proxmox01, with local device in the config file :

hostpci0: 02:00.1
hostpci1: 04:00.0

then, done this command file for testing migration :
vm running :

qm migrate 108 proxmox01 -force -online 1

--> error.

vm stopped :

qm migrate 108 proxmox01 -force

--> successfull migration.

So, maybie it should work for HA migration, because the state of vm is stopped ?
my goal is only HA migration.

 

[athompso]

Sorry, but what you're asking for is so far outside the scope of any virtualization hypervisor, that I don't think you'll ever get a solution, at least not for this exact configuration.
Dietmar already explained the fundamental problem: there's no way to mirror the state of non-virtual hardware, and the whole point of HA is that the entire execution environment remains in exactly the same state after the migration.

You should not need PCI passthrough to run PPPoE through a bridge - that should "just work". IPsec is not relevant until you have PPPoE working.

Per my suggestions previously, try creating a new bridge, and assign only that NIC and that VM to the new bridge, and see if that works for you.

If it still fails on a brand-new bridge (vmbrX), please perform a packet capture on the tapX.Y device, the bridge device, and the real physical ethernet device (can be all at the same time, or on separate tests) and send the packet captures directly to someone who can look at them for anomalies. Don't post it here, as it might contain your password!

Obviously something is wrong here, but you're trying to fix the wrong problem IMHO. Fix the correct problem (PPPoE over bridging) and then HA will work for you.
-Adam

 

[whitewater]

Thank you Athompso for taking time to explain this in detail for me.
i will try all that this week and will give feedback.

 

[whitewater]

after somes tests, i have found the problem.

information about hosts network :
name, active state, autostart, ports/slaves, ip adress
eth0, yes, no, ip adress empty
eth1, no, no, ip adress empty
eth2, no, no, ip adress empty
eth3, yes, yes, 192.168.192.1
vmbr0, yes, yes, eth0, 192.168.1.2
vmbr1, yes, yes, eth1, ip adress empty
vmbr2, yes, yes, eth2, ip adress empty

eth3 is reserved for drbd sync.
i access to proxmox with 192.168.1.2 adress.

eth1 isn't being actived, i have put ip adress in vmbr1 and vmbr2, like 1.1.1.1 and 2.2.2.2 then reboot.
state of eth1 and eth2 change to yes and it's ok :
pppoe work.

so, my problem was very stupid. sorry for this big noise.
i thought vmbr1 actived was sufficient.

i would like to understand how best configure ip address for vmbr cards i don't need in local.

if i put address like :
vmbr1 -> 192.168.1.3
i think there will be a route problem.
so, maybie use address like 1.1.1.1 ?

 

[athompso]

You don't have to set IP addresses at all on ethX or vmbrX interfaces that you aren't using for IP - just leave the address field blank and ensure the auto start check box is turned on.

 

[whitewater]

ok, thank you again Athompso.
i have done this and it's ok.

I have seen many people say that they used it in production. My pfsense running in VM now. I'll see over time. thank you again.