Installing proxmox on Ubuntu using virt-manager

[reshminilus]

I have two laptops running on Ubuntu. I have installed proxmox on a VM on virt-manager on both laptops. Both the laptops can ping each other, but I cannot get the two proxmox web consoles to load. I have opened all the necessary ports. Here is some info:

I cannot open each web console on its own laptop. I understand it is not customary to install proxmox on wireless, and will appreciate any help you can give me

 

[BobhWasatch]

I believe that virt-manager by default sets up a NAT network. So you'd need to not just "open" ports but set up port forwarding. Otherwise each PVE is basically behind a NAT firewall and can't receive connections from outside.

 

[reshminilus]

I believe that virt-manager by default sets up a NAT network. So you'd need to not just "open" ports but set up port forwarding. Otherwise each PVE is basically behind a NAT firewall and can't receive connections from outside.

In virt manager, I have made the Network source into a Bridge Device and set the Device name to virbr0 and it is not on the NAT network.

 

[BobhWasatch]

I don't know enough about virt-manager to offer any suggestions. Sorry. Maybe somebody else?

 

[esi_y]

I have two laptops running on Ubuntu. I have installed proxmox on a VM on virt-manager on both laptops.

back to the NAT network and the web consoles work on each individual laptop. Both laptops can ping each other and each laptop can also ping its own proxmox node.

Your problem really nails down to setting up a bridge in Ubuntu, since you are on desktop the netplan refers to NetworkManager, have a look at e.g. here:

https://www.tecmint.com/create-network-bridge-in-ubuntu/

Once you have that bridge you make use of it as NIC for that PVE host (which is a VM like any other for you on that virt* setup).

 

[BobhWasatch]

Oh, wait a minute. I must have misunderstood your initial post. There are two laptops involved. I was thinking it was just one with two instances of PVE. Yeah, that won't work. Wifi does not work with bridging. Access points normally reject frames that don't come from the MAC that initially associated with the AP. That's why the bridge didn't work.

It appears that virt-manager doesn't make it as easy to set up forwarding as other desktop virtualization products like VirtualBox do. You'll have to use iptables directly. Here's an example where they use a hook script to set up forwarding when the VM starts:

https://wiki.libvirt.org/Networking.html#Forwarding_Incoming_Connections

 

[BobhWasatch]

To be honest if this is just a learning exercise things would be a lot simpler if you put two instances of PVE on the same laptop. Then they can talk to each other and to the host without doing anything special.

 

[esi_y]

Wifi does not work with bridging. Access points normally reject frames that don't come from the MAC that initially associated with the AP. That's why the bridge didn't work.

My bad. I somehow ignored that part when reading it focusing on the fact the OP was attaching virbr0 which is used for the NAT'ed network only.

two laptops to use as servers, have proxmox on each, and create a cluster with both of the nodes

This is just an exercise, obviously, correct?

Do you recommend that I use VirtualBox instead of virtmanager?

Your issue is not the virtualisation solution. It's the networking over WiFi. Use regular ethernet, USB adapters if necessary. Or...

would be a lot simpler if you put two instances of PVE on the same laptop.

But nothing wrong with bridges and virt-manager, it's just about adding a bridge on the host OS. Not for WiFi NIC, though. Would be anyhow crazy considering how the cluster communication is low latency dependent with PVE.

 

[esi_y]

Its not an exercise, he expects his this storage solution to be used by the company. Got it, so I have to add a bridge on the laptop, but both laptops already have virbr0 and br0. What would my next step be? Sorry, I am a beginner.

Alright, I was just about to say that it would theoretically work with the port forwarding too, but it's really pushing it (even for an exercise) when it comes to corosync (intra-cluster) communication.

I have to admit I am flabbergasted by this is meant to be actually used for anything "production". Putting aside all the other obvious (nothing in that setup would be "supported"), two questions come to mind:

1) Why haven't you installed PVE directly onto the laptops? Can you?
2) Do these laptops have ethernet NICs?

 

[esi_y]

1) i previously did this, but I was not able to recieve a basic ping from one node to the other. I also was not able to access the webconsole at all on any devices.
2) how do I check if they have ethernetNICs? They are currently connected to wifi.

Well, do you have anywhere to plug a regular ethernet (RJ-45 terminated) cable into the laptops? Alternatively, can you get ethernet adapters?

I am sure someone would chip in and help how to get your particular wireless cards work on the Debian that PVE is, but it's useless in my view to even try to set up any cluster over WiFi. The other thing is, there's no real benefit in having 2 nodes only (you would at least need a Q device to make it anything sensible in terms of "stability" - but using the word in this thread is almost sarcastic without intending to be).

 

[esi_y]

Unfortunately, I can only plug one laptop into the ethernet, I just edited my previous message on what I had tried for port-forwarding, any thoughts on this? Also, he has given me three laptops but I am trying right now to just make a cluster of 2 and then add in the third one after the cluster creation and joining is successful.

I just glanced up, is that chatgpt sort of thing? Do you understand the implications of you are doing here? You will have a laptop on WiFi (on a congested network dropping frames whenever someone turns on e.g. a microwave), then you will have KVM (does not matter if virt-manager, the desktop OS is just killing it) running a synthetic NAT'ed network into which you will be port forwarding to access 8006 port (you would then have to port forward also ports for corosync, etc). You would then have to access the PVE GUI on the laptop's IP (and I am not sure if desktop Ubuntu has some firewall already going on when freshly installed, I guess ufw with all the clutter tables would be already there).

If you were to set this successfully up - I am imagining some sort of student dorm project as a proof-of-concept - it would be unusable for anything real, especially if you start adding more nodes.

I understand you want to make the best out of it, but two major points to start with:

1) Do not run PVE inside another KVM.
2) Get - if you absolutely have to - USB dongles, but get it connected over ethernet.

... and if you completely ignore me on these consider that even if you were to make use of those wireless cards, you would be always better off doing it on that plain Debian (i.e. getting it work with the WiFi card) and not testing KVM (virt-* on Ubuntu) on KVM (PVE / its guests).

 

[esi_y]

I just want to add why the whole thing (WiFi) is a problem and why going for a "cluster" makes zero sense for the setup. I do not know the motivation behind having it as a cluster (it will kill the hardware, especially drives over time, but that aside), but one of the typical reasons would be to use "high availablity" ... however that will get your nodes rebooting anytime there's a problem with quorum (i.e. intra-cluster comm problems).

When you look at official docs [1] - the key part is:

Network Requirements​

The Proxmox VE cluster stack requires a reliable network with latencies under 5 milliseconds (LAN performance) between all nodes to operate stably. While on setups with a small node count a network with higher latencies may work, this is not guaranteed and gets rather unlikely with more than three nodes and latencies above around 10 ms.
The network should not be used heavily by other members, as while corosync does not uses much bandwidth it is sensitive to latency jitters; ideally corosync runs on its own physically separated network. Especially do not use a shared network for corosync and storage [...]

So for that reason ... I glanced up again at your last answer ... I never suggested virt-manager just to be clear. I wanted to help you make it work properly assuming this is some sort of school project. If you are able to run Ubuntu on those laptops, they should be able to run Debian (i.e. PVE), I believe the only issue you had was you were not connected on the wireless network, that's all. Once you get wired connectivity, directly installed PVE on each laptop should work for you.

Also to be clear - nothing in this setup (laptops) is production, but WiFi (for the reasons above) is something I would not even troubleshoot. I can imagine someone at home using 3 old laptops (ideally with built-in ethernet cards) for this as a learning project.

The second issue - you mentioned it's meant for S3-like storage ... so again problem with the network is ... your S3 (or migration/replication even within PVE) can saturated that one network. Which is why a production setup would use separate NICs ... if you want to press on, you may want to have 2 dongles for each laptop. Keep in mind your S3 performance will be at 1GBps max anyhow, but I think you know that.

Don't get me wrong, all this is great learning experience, even bridge, virt-*, iptables, etc. but if you will be the one then called when things break down your internship will be miserable going forward... boss is an IT person? Maybe the best strategy is to impress him with your due dilligence and revert back with an (in)feasibility study...

[1] https://pve.proxmox.com/wiki/Cluster_Manager

 

[esi_y]

I will order more ethernet adapters to attach all laptops.

Oh and one more thing ... be sure to get something well tested with Linux, e.g. ASIX [1] based ones work well (again, I would not call it production though.

[1] https://www.asix.com.tw/en/product/USBEthernet