Keep in mind that the Raspi will need a ARM compiled version of all software. There is a unofficial PVE release for the Raspi (google for pimox) but then you still got the problem with the different uncompatible architectures. Atleast if you are now running those VMs/LXCs on a x86-64 machine.