70-96% Loss on all cluster servers only

[Bidi]

Hello guys,

We have a cluster with a couple of servers on it, all servers had Proxmox 6.2 from what i remember and we update all servers to Proxmox 6.3.3, after a couple of weeks just by accident i was mtr to see some routes and i show there is a big loss on all nodes 70%-96% :| , i remember we didnt had this issue on 6.2.

We have another server but is not on cluster, is with proxmox 6.3 with 2 personal VMs and another VM with TrueNAS and the NAS we use it on the Cluster mounted as NFS to create backups on some customers VM only who opt it for this. On this server there is no loss at all 0%

So i decide to Create a new dedicated with Proxmox 6.3.., create the server, installed, configured and i checked the LOSS was 0% so i was like hm..., i updated Proxmox and all the thinks i forget to check loss, others servers has some problems, the new dedicated server complet empty 2 Cpus, 64Gb Ram, SSD ..etc so i added to the Cluster with other servers, after i added it i check the loss on it

Host Loss% Snt Last Avg Best Wrst StDev
1. s0-176.XXXXX 87.8% 10 0.2 0.2 0.2 0.2 0.0
2. 10.1XXXXX 0.0% 10 0.9 7.5 0.6 62.9 19.5
3. 10.2XXXX 0.0% 10 13.7 12.9 12.3 13.7 0.4
4. www.hoXXXXX 0.0% 10 11.3 12.0 11.2 14.9 1.3

:| how is this posible ?

On the single server we use for backups

Host Loss% Snt Last Avg Best Wrst StDev
1. s0-176.XXXXX 0.0% 3 0.1 0.1 0.1 0.1 0.0
2. 10.1XXXX 0.0% 3 0.7 0.7 0.6 0.8 0.1
3. 10.2XXX 0.0% 2 11.4 11.5 11.4 11.6 0.2
4. www.hXXXX 0.0% 2 11.4 11.4 11.3 11.4 0.1


Complet empty dedicated server new, alone 0% Loss, after join cluster 87.8-96% loss
All the servers ar on the same network.

We have 3 dedicated servers for webhosting, same network 0% loss

 

[Bidi]

Anyone ?

 

[UdoB]

No definitive answer from my side.

But: the first line is the very first hop an IP-package is transmitted. It does this by leaving the physical host using the physical connection - usually Ethernet. Check your cabling, replace the specific cable by another one (for testing) even if you can not proof that it is bad. Sometimes just pull/plug in a connector changes critical electrical connection quality...

Just guessing...

 

[Bidi]

No definitive answer from my side.

But: the first line is the very first hop an IP-package is transmitted. It does this by leaving the physical host using the physical connection - usually Ethernet. Check your cabling, replace the specific cable by another one (for testing) even if you can not proof that it is bad. Sometimes just pull/plug in a connector changes critical electrical connection quality...

Just guessing...

We have 10GB network on 4 nodes and 1GB on others the problem is even betwin 10gb and we even changed the cables,

And this problem shows only after a node join the cluster if is standalone like we have some other servers its fine no loss at all.

 

[H4R0]

Since the destination has 0% loss there is no problem.

What happens if you directly ping the pve node, 0% loss I assume.

Dropping low ttl packets isn't a bad thing.

 

[Bidi]

Since the destination has 0% loss there is no problem.

What happens if you directly ping the pve node, 0% loss I assume.

Dropping low ttl packets isn't a bad thing.

If i ping outside from another location on any node there is no loss.
There is loss only when i mtr from any servers in cluster outside of any internet destination.

When i was using 3.4 or first v of 6 there was such problems, i update the nodes and now there is loss, i dont understand why.

I just made 2 new servers to create a cluster for an customer with 6.2-4 stand alone servers mtr 0 loss, made upgrade to 6.3-3 0 loss, i created the cluster joined the server to cluster 0 loss, i dont get it why mine ones has loss and only when they join the cluster

M i the only one who have this issues ?
Dose anyone has any cluster with v 6.3-3 to report me if they have the same issue or not ?

 

[Bidi]

Anyone ? :|

 

[Bidi]

I founded the issue, witch for me i do not understand at all and i think this is not normal.

Some of ower nodes ar on diffrent ip range, after i changed ip of the nodes to be on the same range, the loss is gone, how is this posible or normal ? :| the ip ranges ar on the same network, port ..etc