Proxmox/Ceph HA - How to configure switch? (VLANs?)

[victorhooi]

I'm trying to setup a 3-node HA cluster for Proxmox, with Ceph as shared storage.

We have a 10G Ethernet switch.

Each node has 2 SFP+ ports (10G), as well as 4 x 1GBase-T ports.

I believe I can use one SFP+ port for VM traffic, and the second for Ceph traffic, correct?

And then I could use one of the 1GBase-T ports for corsync?

My big question is - how should I configure the switch for this?

I assume I will be using VLANs to segment out all the traffic - or is there a better way?

Is the main VM traffic port meant to be in access mode? Or trunk mode?

If anybody has a sample network configuration file, that would be awesome!

 

[Richard]

I'm trying to setup a 3-node HA cluster for Proxmox, with Ceph as shared storage.

We have a 10G Ethernet switch.

Each node has 2 SFP+ ports (10G), as well as 4 x 1GBase-T ports.

I believe I can use one SFP+ port for VM traffic, and the second for Ceph traffic, correct?

Yes

And then I could use one of the 1GBase-T ports for corsync?

Yes

My big question is - how should I configure the switch for this?

I assume I will be using VLANs to segment out all the traffic - or is there a better way?

Is the main VM traffic port meant to be in access mode? Or trunk mode?

If anybody has a sample network configuration file, that would be awesome!

Finally you will have 3 networks:

* Ceph (10GB/s)
* corosync (1GB/s, possibly 2 networks for security)
* VM Traffic (10GB/s)

These networks will (and should) be physically independent from each other. Ideally you have also 3 (or 4, if you have 2 corosync LANs) different switches; if not you have to define of course them as VLANs, but they will not be tagged, i.e. not visible as VLAN at the server's NICs, respectively access mode.


Use Proxmox WEB GUI for configuration. Ceph and corosync will remain as single NIC, for VM traffic it depends:

- if you VMs will have direct access to provider's network bridge the respective NIC to the bridge you use for VM network (probably vmbr0)

- if not (i.e. routed via NAT) don't assign it to a bridge.

 

[victorhooi]

HI Richard,

These networks will (and should) be physically independent from each other. Ideally you have also 3 (or 4, if you have 2 corosync LANs) different switches; if not you have to define of course them as VLANs, but they will not be tagged, i.e. not visible as VLAN at the server's NICs, respectively access mode.

Ok, I only have a single switch for Proxmox's purposes, but it does support VLAN tagging.

Do you mean I should use VLAN access mode (not trunk) for each of these three ports?

What about the VM traffic - if I wanted different VM's on different VLAN's? Or is that not possible with Proxmox/Ceph HA? (My assumption was I could put VM port in trunk mode - the VM management traffic would go on one VLAN ID, and I could tag each VM for a different VLAN ID).

Use Proxmox WEB GUI for configuration. Ceph and corosync will remain as single NIC, for VM traffic it depends:

- if you VMs will have direct access to provider's network bridge the respective NIC to the bridge you use for VM network (probably vmbr0)

- if not (i.e. routed via NAT) don't assign it to a bridge.

This part I don't quite understand - what do you mean by "Ceph and corosync will remain as single NIC".

Do you mean they each get direct access to the NIC?

And that I can use vmbr0 for the VM traffic?

Finally - in ceph.conf - you don't specify the actual network interface, just the IP address, right? And then the OS figures out which interface to send it over?

 

[Richard]

What about the VM traffic - if I wanted different VM's on different VLAN's? Or is that not possible with Proxmox/Ceph HA? (My assumption was I could put VM port in trunk mode - the VM management traffic would go on one VLAN ID, and I could tag each VM for a different VLAN ID).

Yes, possible.

This part I don't quite understand - what do you mean by "Ceph and corosync will remain as single NIC".

Do you mean they each get direct access to the NIC?

More precise: will not be part of a bridge, i.e. not used for any traffic from/to VMs

And that I can use vmbr0 for the VM traffic?

Yes

Finally - in ceph.conf - you don't specify the actual network interface, just the IP address, right? And then the OS figures out which interface to send it over?

Use "pveceph" respectively our WEB GUI. See https://pve.proxmox.com/pve-docs/chapter-pveceph.html