I don't understand if you were able to make your omping test work (you have to run it at the same time in all nodes), from your pasted output seems it just failed (or you did not it right).
TOTEM problem (corosync)
- Thread starter Bart Brouwer
- Start date
Allright! I did and found a backup from my nas is starting at 6 oclock. But the strange thing is, the router / switch is very fast (mikrotik CCR1009). So i dont understand why this is a problem 
but oke! you have an idea?
I have this configuration at this moment:
-2 PROXMOX SERVERS (2 eth; 1 internet, 1 switch.
-1 SYNOLOGY NAS (with VM disks) with SSD (4 eth; 1 internet, 3 switch bonding)
-1 MIKROTIK SWITCH
all hardware is connected with one switch.
When i recieve my new servers (G9 HP) with 4 eth ports, what do you recommend?
My idea:
another switch (2 total)
-switch for data
-switch for cluster information (simple cheap switch?)
ServerA ethernet: 1 internet, 1 cluster switch and 2 to the data switch
ServerB ethernet: 1 internet, 1 cluster switch and 2 to the data switch
is this a logical setup?
Greetz and tnx in advance
Bart
but oke! you have an idea? I have this configuration at this moment:
-2 PROXMOX SERVERS (2 eth; 1 internet, 1 switch.
-1 SYNOLOGY NAS (with VM disks) with SSD (4 eth; 1 internet, 3 switch bonding)
-1 MIKROTIK SWITCH
all hardware is connected with one switch.
When i recieve my new servers (G9 HP) with 4 eth ports, what do you recommend?
My idea:
another switch (2 total)
-switch for data
-switch for cluster information (simple cheap switch?)
ServerA ethernet: 1 internet, 1 cluster switch and 2 to the data switch
ServerB ethernet: 1 internet, 1 cluster switch and 2 to the data switch
is this a logical setup?
Greetz and tnx in advance
Bart
Last question, is it possible to install proxmox on synology docker? for HA (2 server and 1 docker) oppertunity?
I am seeing similar TOTEM logs on my cluster. I have Proxmox 5 newly installed on 3 new servers. I have no control over the network switches.
I have not seen any issues or errors with the status of the cluster, although it is not in production yet, as I want to make sure it's stable.
The cluster is operating on a 192.168.0.1/24 address range on vmbr1 and this is bridged to a tagged VLAN. I went through all the steps listed in this thread and changed window_size to 150 then 300, but with no effect.
Any ideas?
I have not seen any issues or errors with the status of the cluster, although it is not in production yet, as I want to make sure it's stable.
The cluster is operating on a 192.168.0.1/24 address range on vmbr1 and this is bridged to a tagged VLAN. I went through all the steps listed in this thread and changed window_size to 150 then 300, but with no effect.
Any ideas?
Here is the test from one server, with the same command running on both others, and with similar reports. So this indicates 100% multicast loss? I am going to ask my provider if they can enable multicast/igmp snooping.
root@bvmhost0:/var/log# omping -c 10000 -i 0.001 -F -q bvmhost0 bvmhost1 bvmhost2
bvmhost1 : waiting for response msg
bvmhost2 : waiting for response msg
bvmhost2 : joined (S,G) = (*, 232.43.211.234), pinging
bvmhost1 : waiting for response msg
bvmhost1 : joined (S,G) = (*, 232.43.211.234), pinging
bvmhost2 : given amount of query messages was sent
bvmhost1 : waiting for response msg
bvmhost1 : server told us to stop
bvmhost1 : unicast, xmt/rcv/%loss = 9334/9334/0%, min/avg/max/std-dev = 0.052/0.107/2.524/0.049
bvmhost1 : multicast, xmt/rcv/%loss = 9334/0/100%, min/avg/max/std-dev = 0.000/0.000/0.000/0.000
bvmhost2 : unicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.050/0.108/3.537/0.063
bvmhost2 : multicast, xmt/rcv/%loss = 10000/0/100%, min/avg/max/std-dev = 0.000/0.000/0.000/0.000
root@bvmhost0:/var/log# omping -c 10000 -i 0.001 -F -q bvmhost0 bvmhost1 bvmhost2
bvmhost1 : waiting for response msg
bvmhost2 : waiting for response msg
bvmhost2 : joined (S,G) = (*, 232.43.211.234), pinging
bvmhost1 : waiting for response msg
bvmhost1 : joined (S,G) = (*, 232.43.211.234), pinging
bvmhost2 : given amount of query messages was sent
bvmhost1 : waiting for response msg
bvmhost1 : server told us to stop
bvmhost1 : unicast, xmt/rcv/%loss = 9334/9334/0%, min/avg/max/std-dev = 0.052/0.107/2.524/0.049
bvmhost1 : multicast, xmt/rcv/%loss = 9334/0/100%, min/avg/max/std-dev = 0.000/0.000/0.000/0.000
bvmhost2 : unicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.050/0.108/3.537/0.063
bvmhost2 : multicast, xmt/rcv/%loss = 10000/0/100%, min/avg/max/std-dev = 0.000/0.000/0.000/0.000
Ok the answer to this is firewalling. If I disable the global firewall temporarily, I can ping on the multicast address.
I have enabled some global firewalling rules at the datacenter level, but this is to allow access from external hosts to SSH and port 8006. There are no specific rules for the internal network (192.168.0/0/24) and nothing that would block ICMP or UDP. Is there a default firewall rule which is blocking something between servers?
I have enabled some global firewalling rules at the datacenter level, but this is to allow access from external hosts to SSH and port 8006. There are no specific rules for the internal network (192.168.0/0/24) and nothing that would block ICMP or UDP. Is there a default firewall rule which is blocking something between servers?
We have a default anti block police for localnet.
You can check you localnet with this command
see for port information
https://pve.proxmox.com/wiki/Ports
You can check you localnet with this command
Code:
pve-firewall localnetsee for port information
https://pve.proxmox.com/wiki/Ports
It picks up the local network fine
And the firewall rules look like the default
So I don't understand why multicast is being blocked. Resorting to unicast is a possible solution since this cluster will never be more than 4 nodes.
local hostname: bvmhost1
local IP address: 192.168.213.5
network auto detect: 192.168.213.0/24
using detected local_network: 192.168.213.0/24
local IP address: 192.168.213.5
network auto detect: 192.168.213.0/24
using detected local_network: 192.168.213.0/24
And the firewall rules look like the default
-A PVEFW-DropBroadcast -m addrtype --dst-type MULTICAST -j DROP
-A PVEFW-HOST-IN -s 192.168.213.0/24 -p udp -m addrtype --dst-type MULTICAST -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-OUT -p udp -m addrtype --dst-type MULTICAST -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-IN -s 192.168.213.0/24 -p udp -m addrtype --dst-type MULTICAST -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-OUT -p udp -m addrtype --dst-type MULTICAST -m udp --dport 5404:5405 -j RETURN
So I don't understand why multicast is being blocked. Resorting to unicast is a possible solution since this cluster will never be more than 4 nodes.