Does Ceph Public need to be on ProxMox Cluster network?

[techi]

When I put Ceph Public network on dedicated VLAN that is NOT routed, ceph fails with "Got timeout(500)".

What needs to talk to Ceph?

Frustrating as the all the documents and guides say to separate Ceph Pubic and Cluster(private) on their own networks.

Planning for production with those as air-gaped switches.
However, when I put Ceph Public on a vlans which is not routed, Ceph fails with communication errors.

EDIT: updated to reflect that it's only air-gaped Pubic network that causes Ceph to fail. The Cluster (or private) network is fine being isolated.

EDIT2: Part of the issue was my fault. Laugh at me. I did not have jumbo frames on the Ceph Public switch.
Now as long as I put Ceph Public on the ProxMox Cluster network, it does work.

Is this supposed to be a requirement?

 

[techi]

I thought this video would be the solution.
https://www.youtube.com/watch?v=PZ0tPGGaca0

He put the Ceph Public network on the ProxMox Cluster Linux VLAN.

I tried doing that, but cannot re-produce success.

Once I install Ceph on the 2nd node, i get "Got timeout(500)"

EDIT: This DOES work...I made the mistake of not enabling Jumbo Frames on that switch.
However, if I put them on DIFFERENT networks it does NOT work.

Is this a requirement?

 

[techi]

Update: found a post which says they should NOT be the same network...yet when I put them apart Ceph fails.

Note: the forum is highlighting the wrong section of this post...see my quote below, or click the link and read that full comment from

Christian St.​

C

Post in thread 'Ceph Network public and cluster some questions'

Mar 14, 2021

This is, what the Wiki says (https://pve.proxmox.com/wiki/Deploy_Hyper-Converged_Ceph_Cluster):

• Public Network: You should setup a dedicated network for Ceph, this setting is required. Separating your Ceph traffic is highly recommended, because it could lead to troubles with other latency dependent services, e.g., cluster communication may decrease Ceph’s performance, if not done.
• Cluster Network: As an optional step you can go even further and separate the OSD replication & heartbeat traffic as well. This will relieve the public network and could lead to...

• Christian St.

• 

"It could be sometimes a little bit confusing, because Proxmox has also a Cluster Network. The "Proxmox Cluster Network" is Corosync which is independent from Ceph. Ceph traffic and the Corosync traffic should always be seperated from each other, because latency problems in the corosync connection can lead to troubles with your whole cluster."