I was looking to see if anyone was aware with any of the modifications to the Proxmox Ceph release when compared to the upstream repository that can affect how you would get an external client to rbd volumes and cephfs pools. I recently built up a test cluster of all pve 8.1.3 (3 machines, 1 SSD OSD each) with the intention to use it for a dedicated kubernetes cluster using ceph as persistent storage. For some reason, while all of the features within proxmox and it's GUI work fine, any attempts to connect other clients from outside the cluster seem to not work. I even tried enabling the dashboard module which from the terminal looked just fine but even with the firewall disabled, leads to connection refused. I saw another thread with someone posting similar issues interacting with the cluster, but pveproxy was fine, maybe this is the key to it?
I've tried multiple methods to get a working interaction between prox hosting a ceph cluster and anything else, but the farthest I have been able to get is only cephfs as a volume mount, which is not enough to fully drive a kubernetes cluster default storage or CSI. Is it even possible to work in the opposite direction, and let proxmox manage a ceph cluster that exists outside its cluster by installing all the right packages and modifying the configurations to the external nodes?
TL;DR: Does the proxmox ceph package change the location of any configs other than the /etc/pve/priv stuff, and should I be building configurations differently to set up persistent volume claims for my k8s cluster when as far as I can tell the port numbers appear correct and ceph commands on nodes in the cluster appear to be reporting no abnormalities?
I've tried multiple methods to get a working interaction between prox hosting a ceph cluster and anything else, but the farthest I have been able to get is only cephfs as a volume mount, which is not enough to fully drive a kubernetes cluster default storage or CSI. Is it even possible to work in the opposite direction, and let proxmox manage a ceph cluster that exists outside its cluster by installing all the right packages and modifying the configurations to the external nodes?
TL;DR: Does the proxmox ceph package change the location of any configs other than the /etc/pve/priv stuff, and should I be building configurations differently to set up persistent volume claims for my k8s cluster when as far as I can tell the port numbers appear correct and ceph commands on nodes in the cluster appear to be reporting no abnormalities?