Hi there.
Has anyone used proxmox with directly connected 10G network cards, or with multiple subnets / networks for the ceph network?
We have a working ceph cluster, 5 nodes and had the dual 10G NICS all connected to a 1G switch, with them setup as slaves to a bond interface, which had the IP assigned (10.0.0.1/24 etc).
We had 3 monitors configured on 10.0.0.1, 10.0.0.2 and 10.0.0.3, or the IP of the bond0 on each of the first 3 nodes
We want to directly connect the dual 10G interfaces, and are using quagga and ospf to advertsie routes etc, and we are able to ping to each IP address on any of the hosts, with routes showing up in each nodes routing table for all of the 5x /30 networks.
We've removed monitor 2 and 3 and have left the 1 existing monitor, the monitor which was created when node 1 was using bond0 on 10.0.0.1/24.
I have edited ceph.conf directly and added all 5x networks, for public and cluster networks
[global]
auth client required = cephx
auth cluster required = cephx
auth service required = cephx
cluster network = 10.0.0.0/30,10.0.0.4/30,10.0.0.8/30,10.0.0.12/30,10.0.0.16/30
fsid = 953d5cc2-aa41-4c28-8382-2f2ba943
keyring = /etc/pve/priv/$cluster.$name.keyring
mon allow pool delete = true
osd journal size = 5120
osd pool default min size = 2
osd pool default size = 3
public network = 10.0.0.0/30,10.0.0.4/30,10.0.0.8/30,10.0.0.12/30,10.0.0.16/30
debug mon = 10
[mds]
keyring = /var/lib/ceph/mds/ceph-$id/keyring
[osd]
keyring = /var/lib/ceph/osd/ceph-$id/keyring
[mds.wms-oc-01]
host = wms-oc-01
mds standby for name = pve
[mon.wms-oc-01]
host = wms-oc-01
mon addr = 10.0.0.1:6789
The firewall has been turned off.
IP forward is enabled on the hosts and on the quagga router.
![ceph network diagrams.JPG ceph network diagrams.JPG](https://forum.proxmox.com/data/attachments/9/9767-d786652dd4c22b65e5e3d85615608e6e.jpg)
We are unable to add new monitors as we have no quorum, even after specifying the mon-address on one of the other nodes, it seems like the existing monitor is now un-reachable from the other nodes. Haven't tried removing the existing monitor yet, but I suspect proxmox won't let us.
Snippet of netstat;
tcp 0 0 10.0.0.1:6789 10.0.0.6:45796 ESTABLISHED
tcp 0 0 10.0.0.1:6789 10.0.0.2:59510 TIME_WAIT
tcp 0 0 10.0.0.1:6789 10.0.0.17:55994 TIME_WAIT
tcp 0 0 10.0.0.1:6789 10.0.0.10:46776 TIME_WAIT
tcp 0 0 10.0.0.1:6789 10.0.0.2:59432 TIME_WAIT
/var/log/ceph/ceph...log
2019-04-17 09:39:18.122357 7f7294af2100 4 rocksdb: [/home/builder/source/ceph-12.2.11/src/rocksdb/db/db_impl.cc:343] Shutdown complete
2019-04-17 09:39:18.122374 7f7294af2100 -1 rocksdb: IO error: lock /var/lib/ceph/mon/ceph-wms-oc-01/store.db/LOCK: Resource temporarily unavailable
2019-04-17 09:39:18.122377 7f7294af2100 -1 error opening mon data directory at '/var/lib/ceph/mon/ceph-hostname': (22) Invalid argument
We have a lot of these now in the messages
Apr 17 13:33:27 hostname kernel: [ 5334.428085] libceph: mon0 10.0.0.1:6789 socket closed (con state OPEN)
Apr 17 13:33:32 hostname kernel: [ 5339.428808] libceph: mon0 10.0.0.1:6789 socket closed (con state OPEN)
Apr 17 13:33:37 hostname kernel: [ 5344.429202] libceph: mon0 10.0.0.1:6789 socket closed (con state OPEN)
Has anyone had any experience with this sort of setup?
Many Thanks
Andy
Has anyone used proxmox with directly connected 10G network cards, or with multiple subnets / networks for the ceph network?
We have a working ceph cluster, 5 nodes and had the dual 10G NICS all connected to a 1G switch, with them setup as slaves to a bond interface, which had the IP assigned (10.0.0.1/24 etc).
We had 3 monitors configured on 10.0.0.1, 10.0.0.2 and 10.0.0.3, or the IP of the bond0 on each of the first 3 nodes
We want to directly connect the dual 10G interfaces, and are using quagga and ospf to advertsie routes etc, and we are able to ping to each IP address on any of the hosts, with routes showing up in each nodes routing table for all of the 5x /30 networks.
We've removed monitor 2 and 3 and have left the 1 existing monitor, the monitor which was created when node 1 was using bond0 on 10.0.0.1/24.
I have edited ceph.conf directly and added all 5x networks, for public and cluster networks
[global]
auth client required = cephx
auth cluster required = cephx
auth service required = cephx
cluster network = 10.0.0.0/30,10.0.0.4/30,10.0.0.8/30,10.0.0.12/30,10.0.0.16/30
fsid = 953d5cc2-aa41-4c28-8382-2f2ba943
keyring = /etc/pve/priv/$cluster.$name.keyring
mon allow pool delete = true
osd journal size = 5120
osd pool default min size = 2
osd pool default size = 3
public network = 10.0.0.0/30,10.0.0.4/30,10.0.0.8/30,10.0.0.12/30,10.0.0.16/30
debug mon = 10
[mds]
keyring = /var/lib/ceph/mds/ceph-$id/keyring
[osd]
keyring = /var/lib/ceph/osd/ceph-$id/keyring
[mds.wms-oc-01]
host = wms-oc-01
mds standby for name = pve
[mon.wms-oc-01]
host = wms-oc-01
mon addr = 10.0.0.1:6789
The firewall has been turned off.
IP forward is enabled on the hosts and on the quagga router.
![ceph network diagrams.JPG ceph network diagrams.JPG](https://forum.proxmox.com/data/attachments/9/9767-d786652dd4c22b65e5e3d85615608e6e.jpg)
We are unable to add new monitors as we have no quorum, even after specifying the mon-address on one of the other nodes, it seems like the existing monitor is now un-reachable from the other nodes. Haven't tried removing the existing monitor yet, but I suspect proxmox won't let us.
Snippet of netstat;
tcp 0 0 10.0.0.1:6789 10.0.0.6:45796 ESTABLISHED
tcp 0 0 10.0.0.1:6789 10.0.0.2:59510 TIME_WAIT
tcp 0 0 10.0.0.1:6789 10.0.0.17:55994 TIME_WAIT
tcp 0 0 10.0.0.1:6789 10.0.0.10:46776 TIME_WAIT
tcp 0 0 10.0.0.1:6789 10.0.0.2:59432 TIME_WAIT
/var/log/ceph/ceph...log
2019-04-17 09:39:18.122357 7f7294af2100 4 rocksdb: [/home/builder/source/ceph-12.2.11/src/rocksdb/db/db_impl.cc:343] Shutdown complete
2019-04-17 09:39:18.122374 7f7294af2100 -1 rocksdb: IO error: lock /var/lib/ceph/mon/ceph-wms-oc-01/store.db/LOCK: Resource temporarily unavailable
2019-04-17 09:39:18.122377 7f7294af2100 -1 error opening mon data directory at '/var/lib/ceph/mon/ceph-hostname': (22) Invalid argument
We have a lot of these now in the messages
Apr 17 13:33:27 hostname kernel: [ 5334.428085] libceph: mon0 10.0.0.1:6789 socket closed (con state OPEN)
Apr 17 13:33:32 hostname kernel: [ 5339.428808] libceph: mon0 10.0.0.1:6789 socket closed (con state OPEN)
Apr 17 13:33:37 hostname kernel: [ 5344.429202] libceph: mon0 10.0.0.1:6789 socket closed (con state OPEN)
Has anyone had any experience with this sort of setup?
Many Thanks
Andy