Network setup for 3-node Proxmox/Ceph cluster

[victorhooi]

Hi

We're setting up a small Proxmox HA cluster, to run two Windows VMs. This will be used to serve a handful of users.

We have three physical servers.

We have a single 10GbE switch (Unifi US-16-XG)

Each server has a Intel X520-DA2 card in it, with two DAC cables per server going to the switch.

Our goal is to set this up with a three-node Proxmox cluster, as well as three-node Ceph cluster, and have HA for the two VMs.

I've installed Proxmox 5.2 on all three nodes.

I can setup a cluster, as per the Cluster documentation.

My idea was two use one 10GbE link for the Ceph storage sync, and another 10GbE link for everything else. (I believe the general practice is to use another network again for corosync communication - but for this simple setup with 2 VMs, I'd like to start with just one link for storage, and one link for everything else, and confirm that works first.)

Out of the box, it seems like the Proxmox installer has setup a bridge interface (vmbr0) against the first 10GbE link (enp23s0f0).

My question is - how should I set it up so that Proxmox and everything else go over this link, and the second link is used for Ceph?

Should I create a second bridge interface (e.g. vmbr1), and copy the same configuration against enp23s0f1 - and then just assign a second IP address to that?

Or should I be using VLAN tagging, and setting up a completely separate subnet/range?

If there's a good guide for a simple 3-node setup that you know of, please let me know as well.

Cheers,
Victor

 

[victorhooi]

Hi,

I tried to configure the second NIC port with a different IP address, but the same subnet mask and gateway - however, it gives me an error:

gateway: Default gateway already exists on interface 'vmbr0'.
OK

Is there some trick to adding a second NIC?

I read through https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_network_configuration, and couldn't seem to spot anything obvious I missed - but I assume there is.

Cheers,
Victor

 

[bobmc]

So if I read your description correctly, you are using proxmox to create a 3 node ceph cluster to provide ceph storage to the rest of the network (physical) clients. Additionally, you want to run a couple of Windows VM's

The proxmox has already setup vmbr0 with your default network gateway, so you can reach the proxmox hosts from your network. All three hosts are plugged into the same Unifi switch.

The next step would be to setup vmbr1 on the second port with a new network range.

So, if vmbr0 is on 192.168.50.x with subnet mask of 255.255.255.0 and gateway of 192.168.50.254 - for example, then use something like 192.168.200.x for vmbr1 with a subnet mask of 255.255.255.0 and no gateway. This would become your ceph storage sync network.

This is discussed here:- https://pve.proxmox.com/wiki/Ceph_Server#Network_for_Ceph where they use a 10..10.10.x network instead.