Proxmox Ceph without PVE Cluster
- Thread starter brosky
- Start date
Only the other way around - PVE as node in cluster without ceph to only run VMs and nothing more or without VMs as quorum-only node.
But I don't understand the problem, having a node in a cluster alone isn't generating any load (besides the corosync traffic and some pings) .
Last edited:
From where is this info? Yes, problems arise when corosync ping times go up or time out over 60 seconds between nodes.
To prevent this, a dedicated (and redundant) network with enough bandwidth (1GBit is enough) for corosync alone is recommended.
If you run this in a vlan, other traffic on the same cable will steal the bandwidth and drive pings for your corosync-vlan to the sky.
You maybe can work around with traffic limits, but an extra net with extra bandwidth is in every way the better approach.
To prevent this, a dedicated (and redundant) network with enough bandwidth (1GBit is enough) for corosync alone is recommended.
If you run this in a vlan, other traffic on the same cable will steal the bandwidth and drive pings for your corosync-vlan to the sky.
You maybe can work around with traffic limits, but an extra net with extra bandwidth is in every way the better approach.
There is no hard limit how many Proxmox VE nodes can form a cluster. Up to 25 Nodes is what we hear works well. Above, you might run into some issues that need some Corosync tuning. But again, no hard numbers and with more powerful hardware, some of the issues that show up in larger clusters also disappear. So please don't take this number as a hard fact when future you reads this in 5 or 10 years. 
To the original question. If we are talking about a Proxmox VE + Ceph (HCI) cluster, then all nodes that are part of the Ceph cluster need to be part of the Proxmox VE cluster as well. The
To the original question. If we are talking about a Proxmox VE + Ceph (HCI) cluster, then all nodes that are part of the Ceph cluster need to be part of the Proxmox VE cluster as well. The
ceph.conf file is shared via the pmxcfs (/etc/pve) with all the other nodes, and certain Ceph specific API calls on the Proxmox VE layer need to be relayed to the actual node.https://pve.proxmox.com/wiki/Cluster_Manager#pvecm_cluster_network_requirements
5ms, it can be tuned, but as said dedicated network only for this traffic is the best solution. A node not reached for 60s (seconds here!) will be kicked to reboot.
5ms, it can be tuned, but as said dedicated network only for this traffic is the best solution. A node not reached for 60s (seconds here!) will be kicked to reboot.
Last edited:
Again, it is hard to say a hard limit. With a larger cluster, the timeout increases, for example. The recommendations you see are good rule of thumbs.
The man page for corosync.conf explains the config options quite well. You want to look at the
Another thing to keep in mind, in very large clusters, is CPU speed. If the CPU is too slow to keep up, you will also see issues.
The man page for corosync.conf explains the config options quite well. You want to look at the
knet_ping_* and token options to get a better idea how timeouts are calculated. The rule of thumbs are useful to keep it low enough as they add up with each trip needed.Another thing to keep in mind, in very large clusters, is CPU speed. If the CPU is too slow to keep up, you will also see issues.