TOTEM problem (corosync)

I don't understand if you were able to make your omping test work (you have to run it at the same time in all nodes), from your pasted output seems it just failed (or you did not it right).
 
Allright! I did and found a backup from my nas is starting at 6 oclock. But the strange thing is, the router / switch is very fast (mikrotik CCR1009). So i dont understand why this is a problem :) but oke! you have an idea?

I have this configuration at this moment:
-2 PROXMOX SERVERS (2 eth; 1 internet, 1 switch.
-1 SYNOLOGY NAS (with VM disks) with SSD (4 eth; 1 internet, 3 switch bonding)
-1 MIKROTIK SWITCH
all hardware is connected with one switch.


When i recieve my new servers (G9 HP) with 4 eth ports, what do you recommend?
My idea:
another switch (2 total)

-switch for data
-switch for cluster information (simple cheap switch?)

ServerA ethernet: 1 internet, 1 cluster switch and 2 to the data switch
ServerB ethernet: 1 internet, 1 cluster switch and 2 to the data switch

is this a logical setup?
Greetz and tnx in advance
Bart
 
Separate your networks.
It is never good to have a storage and cluster communication on one network.
Storage networks should always be separate.
 
I am seeing similar TOTEM logs on my cluster. I have Proxmox 5 newly installed on 3 new servers. I have no control over the network switches.

I have not seen any issues or errors with the status of the cluster, although it is not in production yet, as I want to make sure it's stable.

The cluster is operating on a 192.168.0.1/24 address range on vmbr1 and this is bridged to a tagged VLAN. I went through all the steps listed in this thread and changed window_size to 150 then 300, but with no effect.

Any ideas?
 
Do you test with omping and multicast is working?
 
Here is the test from one server, with the same command running on both others, and with similar reports. So this indicates 100% multicast loss? I am going to ask my provider if they can enable multicast/igmp snooping.

root@bvmhost0:/var/log# omping -c 10000 -i 0.001 -F -q bvmhost0 bvmhost1 bvmhost2
bvmhost1 : waiting for response msg
bvmhost2 : waiting for response msg
bvmhost2 : joined (S,G) = (*, 232.43.211.234), pinging
bvmhost1 : waiting for response msg
bvmhost1 : joined (S,G) = (*, 232.43.211.234), pinging
bvmhost2 : given amount of query messages was sent
bvmhost1 : waiting for response msg
bvmhost1 : server told us to stop

bvmhost1 : unicast, xmt/rcv/%loss = 9334/9334/0%, min/avg/max/std-dev = 0.052/0.107/2.524/0.049
bvmhost1 : multicast, xmt/rcv/%loss = 9334/0/100%, min/avg/max/std-dev = 0.000/0.000/0.000/0.000
bvmhost2 : unicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.050/0.108/3.537/0.063
bvmhost2 : multicast, xmt/rcv/%loss = 10000/0/100%, min/avg/max/std-dev = 0.000/0.000/0.000/0.000
 
Ok the answer to this is firewalling. If I disable the global firewall temporarily, I can ping on the multicast address.

I have enabled some global firewalling rules at the datacenter level, but this is to allow access from external hosts to SSH and port 8006. There are no specific rules for the internal network (192.168.0/0/24) and nothing that would block ICMP or UDP. Is there a default firewall rule which is blocking something between servers?
 
We have a default anti block police for localnet.

You can check you localnet with this command
Code:
pve-firewall localnet

see for port information
https://pve.proxmox.com/wiki/Ports
 
It picks up the local network fine
local hostname: bvmhost1
local IP address: 192.168.213.5
network auto detect: 192.168.213.0/24
using detected local_network: 192.168.213.0/24​

And the firewall rules look like the default
-A PVEFW-DropBroadcast -m addrtype --dst-type MULTICAST -j DROP
-A PVEFW-HOST-IN -s 192.168.213.0/24 -p udp -m addrtype --dst-type MULTICAST -m udp --dport 5404:5405 -j RETURN
-A PVEFW-HOST-OUT -p udp -m addrtype --dst-type MULTICAST -m udp --dport 5404:5405 -j RETURN​

So I don't understand why multicast is being blocked. Resorting to unicast is a possible solution since this cluster will never be more than 4 nodes.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!