Hi,
Another question. Three node proxmox, with a meshed ceph-network (10.10.89.1/2/3) on eth2 and eth3, on 10Gbit intel nics. eth0 is vmbr0 with the external ip / dns.
ifconfig looks like this, nota bene the TX and TX values:
Looking at this, it seems that only 50 - 75G of traffic has gone through the 10.10.89.0 (10G ceph) network, and the rest (3.4- 4 TiB!) has gone though vmbr0, my 1G external ip nic.
Here is my ceph.conf:
As far as I know, the above tells ceph to use 10.10.89.0 for ceph traffic, and not the public a.b.c.0/24 nic.
Why so much traffic on my vmbr0, and so little on the eth2/eth3?
Setup of the meshed ceph network is taken from the wiki, and iperf tells me that my ceph network is running at 10G
Can someone explain my observations? Do I need to post other config files?
Another question. Three node proxmox, with a meshed ceph-network (10.10.89.1/2/3) on eth2 and eth3, on 10Gbit intel nics. eth0 is vmbr0 with the external ip / dns.
ifconfig looks like this, nota bene the TX and TX values:
eth0 Link encap:Ethernet HWaddr 68:05:ca:43:3a:8c
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4942557097 errors:0 dropped:32942 overruns:0 frame:0
TX packets:3722437289 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3953319763652 (3.5 TiB) TX bytes:4635593470910 (4.2 TiB)
Interrupt:29 Memory:c73a0000-c73c0000
eth2 Link encap:Ethernet HWaddr 0c:c4:7a:6e:81:e2
inet addr:10.10.89.2 Bcast:10.10.89.255 Mask:255.255.255.0
inet6 addr: fe80::ec4:7aff:fe6e:81e2/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:74782497 errors:0 dropped:0 overruns:0 frame:0
TX packets:59444042 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:81229954895 (75.6 GiB) TX bytes:56338716325 (52.4 GiB)
eth3 Link encap:Ethernet HWaddr 0c:c4:7a:6e:81:e3
inet addr:10.10.89.2 Bcast:10.10.89.255 Mask:255.255.255.0
inet6 addr: fe80::ec4:7aff:fe6e:81e3/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:54370350 errors:0 dropped:0 overruns:0 frame:0
TX packets:57351493 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:53756934479 (50.0 GiB) TX bytes:58702860158 (54.6 GiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:141397335 errors:0 dropped:0 overruns:0 frame:0
TX packets:141397335 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:1820422977978 (1.6 TiB) TX bytes:1820422977978 (1.6 TiB)
tap103i0 Link encap:Ethernet HWaddr 02:18:0a:79:7c:91
inet6 addr: fe80::18:aff:fe79:7c91/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:1750478 errors:0 dropped:0 overruns:0 frame:0
TX packets:5408562 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:11046378551 (10.2 GiB) TX bytes:1426699038 (1.3 GiB)
tap109i0 Link encap:Ethernet HWaddr 0e:21:e8:7b:ba:25
inet6 addr: fe80::c21:e8ff:fe7b:ba25/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:648 errors:0 dropped:270043 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:55445 (54.1 KiB)
vmbr0 Link encap:Ethernet HWaddr 68:05:ca:43:3a:8c
inet addr:a.b.c.d Bcast:a.b.c.255 Mask:255.255.255.0
inet6 addr: fe80::6a05:caff:fe43:3a8c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2990815528 errors:0 dropped:0 overruns:0 frame:0
TX packets:817429329 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3761733493779 (3.4 TiB) TX bytes:4418077825360 (4.0 TiB)
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4942557097 errors:0 dropped:32942 overruns:0 frame:0
TX packets:3722437289 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3953319763652 (3.5 TiB) TX bytes:4635593470910 (4.2 TiB)
Interrupt:29 Memory:c73a0000-c73c0000
eth2 Link encap:Ethernet HWaddr 0c:c4:7a:6e:81:e2
inet addr:10.10.89.2 Bcast:10.10.89.255 Mask:255.255.255.0
inet6 addr: fe80::ec4:7aff:fe6e:81e2/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:74782497 errors:0 dropped:0 overruns:0 frame:0
TX packets:59444042 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:81229954895 (75.6 GiB) TX bytes:56338716325 (52.4 GiB)
eth3 Link encap:Ethernet HWaddr 0c:c4:7a:6e:81:e3
inet addr:10.10.89.2 Bcast:10.10.89.255 Mask:255.255.255.0
inet6 addr: fe80::ec4:7aff:fe6e:81e3/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:54370350 errors:0 dropped:0 overruns:0 frame:0
TX packets:57351493 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:53756934479 (50.0 GiB) TX bytes:58702860158 (54.6 GiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:141397335 errors:0 dropped:0 overruns:0 frame:0
TX packets:141397335 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:1820422977978 (1.6 TiB) TX bytes:1820422977978 (1.6 TiB)
tap103i0 Link encap:Ethernet HWaddr 02:18:0a:79:7c:91
inet6 addr: fe80::18:aff:fe79:7c91/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:1750478 errors:0 dropped:0 overruns:0 frame:0
TX packets:5408562 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:11046378551 (10.2 GiB) TX bytes:1426699038 (1.3 GiB)
tap109i0 Link encap:Ethernet HWaddr 0e:21:e8:7b:ba:25
inet6 addr: fe80::c21:e8ff:fe7b:ba25/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:648 errors:0 dropped:270043 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:55445 (54.1 KiB)
vmbr0 Link encap:Ethernet HWaddr 68:05:ca:43:3a:8c
inet addr:a.b.c.d Bcast:a.b.c.255 Mask:255.255.255.0
inet6 addr: fe80::6a05:caff:fe43:3a8c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2990815528 errors:0 dropped:0 overruns:0 frame:0
TX packets:817429329 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3761733493779 (3.4 TiB) TX bytes:4418077825360 (4.0 TiB)
Looking at this, it seems that only 50 - 75G of traffic has gone through the 10.10.89.0 (10G ceph) network, and the rest (3.4- 4 TiB!) has gone though vmbr0, my 1G external ip nic.
Here is my ceph.conf:
[global]
auth client required = cephx
auth cluster required = cephx
auth service required = cephx
cluster network = 10.10.89.0/24
filestore xattr use omap = true
fsid = 1397f1dc-7d94-43ea-ab12-8f8792eee9c1
keyring = /etc/pve/priv/$cluster.$name.keyring
osd journal size = 5120
osd pool default min size = 1
public network = a.b.c.0/24
[osd]
keyring = /var/lib/ceph/osd/ceph-$id/keyring
[mon.2]
host = pm3
mon addr = 10.10.89.3:6789
[mon.0]
host = pm1
mon addr = 10.10.89.1:6789
[mon.1]
host = pm2
mon addr = 10.10.89.2:6789
auth client required = cephx
auth cluster required = cephx
auth service required = cephx
cluster network = 10.10.89.0/24
filestore xattr use omap = true
fsid = 1397f1dc-7d94-43ea-ab12-8f8792eee9c1
keyring = /etc/pve/priv/$cluster.$name.keyring
osd journal size = 5120
osd pool default min size = 1
public network = a.b.c.0/24
[osd]
keyring = /var/lib/ceph/osd/ceph-$id/keyring
[mon.2]
host = pm3
mon addr = 10.10.89.3:6789
[mon.0]
host = pm1
mon addr = 10.10.89.1:6789
[mon.1]
host = pm2
mon addr = 10.10.89.2:6789
As far as I know, the above tells ceph to use 10.10.89.0 for ceph traffic, and not the public a.b.c.0/24 nic.
Why so much traffic on my vmbr0, and so little on the eth2/eth3?
Setup of the meshed ceph network is taken from the wiki, and iperf tells me that my ceph network is running at 10G
Can someone explain my observations? Do I need to post other config files?