VirtualBMC possible?

[agnon]

Hello!

Is it possible to integrate something like this:

https://github.com/openstack/virtualbmc

into Proxmox? It would be really nice to have.

I'm fiddling with some test installations of RedHat products for instance OpenShift. And if you're going down the installer provisioned infrastructure the installer need to talk to the VMs over IPMI. Of course, one could provision everything yourself but it would really help to have this feature

I did try this but with no luck as there's no libvirt socket to connect to. At least that i'm aware of. And I read some other topic about installing libvirt/libvirt-clients where it pretty much broke proxmox? I didn't dare to try anything after reading that.

Anyone with more experience running proxmox that have managed to pull this off?

I'm running Proxmox 7.1-11.

Best regards
Marcus

 

[LnxBil]

Anyone with more experience running proxmox that have managed to pull this off?

You can do that through the Proxmox VE API with installed QEMU guest agent. I use similar techniques with the docker-machine driver. Yes, it's not as nice as having IPMI, but it works.

 

[agnon]

You can do that through the Proxmox VE API with installed QEMU guest agent. I use similar techniques with the docker-machine driver. Yes, it's not as nice as having IPMI, but it works.

Interesting!

Thing is, i need standards like IPMI, this is for an unassisted installer (ocp4) that rely on communicating with the BMC (which i want to emulate on my local computer for all hosts during the install procedure (for now)) of each cluster node.

But thanks for pointing out the Proxmox VE API. Guess one could fork the virtualbmc project and "just change" how it communicates with the host system for powering on/off setting boot device and so on and so forth when it receives IPMI requests. If i ever get the time

cheers
Marcus

 

[LnxBil]

If i ever get the time

I heard that one before Still looking forward to it.

However, it looks like there is an BMC emulation already present in QEMU, pointed out in this documentation. Just add this to your VM conf:

args: -device ipmi-bmc-sim,id=bmc0 -device isa-ipmi-kcs,bmc=bmc0,irq=5

After that I have an internal BMC which is able to e.g. do power stuff with the VM.

 

[agnon]

I heard that one before Still looking forward to it.

However, it looks like there is an BMC emulation already present in QEMU, pointed out in this documentation. Just add this to your VM conf:

args: -device ipmi-bmc-sim,id=bmc0 -device isa-ipmi-kcs,bmc=bmc0,irq=5

After that I have an internal BMC which is able to e.g. do power stuff with the VM.

Were you able to control what port you wanted it to run on?

Anyways, I took the time and adapted the VirtualBMC to work against the proxmox VE API.

For those interested it's here: https://github.com/agnon/proxmoxbmc

Hopefully it's all in alignment with the license. It uses Proxmoxer to communicate with the PVE API using an API token.

I used it to run a "bmc server" in proxmox while installing a 6 node (Proxmox VMs) OpenShift (4.10) cluster with the "Installer Provisioned Infrastructure". Worked very well.

Cheers
Marcus

 

[LnxBil]

Were you able to control what port you wanted it to run on?

No, it was only local.

Anyways, I took the time and adapted the VirtualBMC to work against the proxmox VE API.

Very cool.

How did you set that up "in the real world"? One IP (maybe the one from your PVE host) that has IPMI for various machines running on a given port per VM?

 

[agnon]

No, it was only local.


Very cool.

How did you set that up "in the real world"? One IP (maybe the one from your PVE host) that has IPMI for various machines running on a given port per VM?

Yes one IP. Added all the nodes that I needed BMC for with different ports.

pbmc add ......... -p 6623 VMID_1
pbmc add ......... -p 6624 VMID_2
....
pbmc start id1 id2 .....

I used a virtual machine for this since openshift use this for baremetal installs even after the installation is finished. So I wanted some sort of "persistent" server.

//MarcuS

 

[LnxBil]

I used a virtual machine for this since openshift use this for baremetal installs even after the installation is finished. So I wanted some sort of "persistent" server.

good idea, would have done the same.

Can you describe what is done behind the scenes with the infrastructure provisioning as you describe in your first post?

 

[agnon]

What the installer basically does is setting up a DHCP + ability to pxe boot the machines on a separate provisioning network.

It then sets the boot device to network using ipmi, powers them on in sequence booting coreos with some ignition files which provision the node, it then set the boot device to disk and reboots. That's pretty much it it constantly checks to see if the nodes are powered on aswell.

Simple but needed, if you want to use Installer provisioned. I didn't find anything saying otherwise in their documentation.

//Marcus

 

[LnxBil]

Thank you.

 

[spacimek]

What the installer basically does is setting up a DHCP + ability to pxe boot the machines on a separate provisioning network.

It then sets the boot device to network using ipmi, powers them on in sequence booting coreos with some ignition files which provision the node, it then set the boot device to disk and reboots. That's pretty much it it constantly checks to see if the nodes are powered on aswell.

Simple but needed, if you want to use Installer provisioned. I didn't find anything saying otherwise in their documentation.

//Marcus

Hi Marcus,
I really like this elegant solution! I gave you a small PR in your repo to fix the version, otherwise it works nicely. I have stumbled a few times, so if I can give a few more intermediate steps to newbies like me, I am happy to - can make PR on your readme, if you're OK with that.

Unfortunately in my case I gave up, OCP 4.10 install failed. It might be that I ran it from Proxmox server directly, not from a separate VM, because I might need qemu-kvm package to be installed. But I am using OVH, there networking is tricky, so I decided to go for UPI after all.

This is the error I've got at the end:

...
DEBUG libvirt_ignition.bootstrap: Creation complete after 9m3s [id=/var/lib/libvirt/openshift-images/c1-sjszt-bootstrap/c1-sjszt-bootstrap.ign;8303f7fc-7876-4b24-9227-6d7a01a9d6f9]
DEBUG libvirt_volume.bootstrap: Creation complete after 0s [id=/var/lib/libvirt/openshift-images/c1-sjszt-bootstrap/c1-sjszt-bootstrap]
DEBUG libvirt_domain.bootstrap: Creating...       
ERROR                                             
ERROR Error: Error defining libvirt domain: virError(Code=8, Domain=10, Message='invalid argument: could not get preferred machine for /usr/bin/qemu-system-x86_64 type=kvm')
ERROR                                             
ERROR   on ../../tmp/openshift-install-bootstrap-3245837519/main.tf line 44, in resource "libvirt_domain" "bootstrap":
ERROR   44: resource "libvirt_domain" "bootstrap" {
ERROR                                             
ERROR                                             
FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change

 

[fahadysf]

Hi Marcus,
I really like this elegant solution! I gave you a small PR in your repo to fix the version, otherwise it works nicely. I have stumbled a few times, so if I can give a few more intermediate steps to newbies like me, I am happy to - can make PR on your readme, if you're OK with that.

Unfortunately in my case I gave up, OCP 4.10 install failed. It might be that I ran it from Proxmox server directly, not from a separate VM, because I might need qemu-kvm package to be installed. But I am using OVH, there networking is tricky, so I decided to go for UPI after all.

This is the error I've got at the end:

...
DEBUG libvirt_ignition.bootstrap: Creation complete after 9m3s [id=/var/lib/libvirt/openshift-images/c1-sjszt-bootstrap/c1-sjszt-bootstrap.ign;8303f7fc-7876-4b24-9227-6d7a01a9d6f9]
DEBUG libvirt_volume.bootstrap: Creation complete after 0s [id=/var/lib/libvirt/openshift-images/c1-sjszt-bootstrap/c1-sjszt-bootstrap]
DEBUG libvirt_domain.bootstrap: Creating...     
ERROR                                           
ERROR Error: Error defining libvirt domain: virError(Code=8, Domain=10, Message='invalid argument: could not get preferred machine for /usr/bin/qemu-system-x86_64 type=kvm')
ERROR                                           
ERROR   on ../../tmp/openshift-install-bootstrap-3245837519/main.tf line 44, in resource "libvirt_domain" "bootstrap":
ERROR   44: resource "libvirt_domain" "bootstrap" {
ERROR                                           
ERROR                                           
FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change

I have a "feeling", nested virtualization is what's being missed there? Can you check whether it's enabled in options?
I'm embarking on a similar journey (OKD4.10 using IPI with the above virtual BMC solution). Might reach out to you if I get stuck before getting to the point you got that error

 

[flybyray]

I used it to run a "bmc server" in proxmox while installing a 6 node (Proxmox VMs) OpenShift (4.10) cluster with the "Installer Provisioned Infrastructure". Worked very well.

did you tried connect with a tool against the bmc instances? what ipmi version are they using?

python pbmc.py list

+------+---------+---------+------+
| VMID | Status  | Address | Port |
+------+---------+---------+------+
| 105  | running | ::      | 6625 |
| 107  | running | ::      | 6626 |
+------+---------+---------+------+

$ IPMIPASSWORD=passw0rd bin/pyghmiutil [BMC IP address] username my_new_feature_command

ref: https://opendev.org/x/pyghmi
I need to bug this tool, to change the udp port in the debugger. It seems to try a ipmi v2 connection. But crashes with some hmac stuff.

Or with ipmitool ipmitool -a -H 127.0.0.1 -p 6625 -U admin chassis power status ?

Password:
IANA PEN registry open failed: No such file or directory
Error: Unable to establish LAN session
Error: Unable to establish IPMI v1.5 / RMCP session

Watching for connections netstat -a -u -p -c 1

udp        0      0 localhost.localdo:45300 localhost.localdom:6625 ESTABLISHED 189087/ipmitool   
udp6       0      0 [::]:6625               [::]:*                              182501/python     
udp6       0      0 [::]:6626               [::]:*                              182500/python

PR to allow proxmox cluters with offline nodes. https://github.com/agnon/proxmoxbmc/pull/4

 

[flybyray]

Or with ipmitool ipmitool -a -H 127.0.0.1 -p 6625 -U admin chassis power status ?

Was my mistake. Without -I option it uses old 1.5 ipmi.

ipmitool -a -I lanplus -H 127.0.0.1 -p 6625 -U admin power off

Password: 
IANA PEN registry open failed: No such file or directory
Chassis Power Control: Down/Off

Thanks @agnon

 

[sabari]

This is awesome. Need to do a POC of Openshift IPI and stumbled on this. Was successful in testing in vsphere. Will give it a try on Proxmox next week and update..