I lose the network connection randomly

[pga57]

Hello,
I'm moving my conversation to this thread hoping to find a solution.
To resume, I lose the network connection randomly (after 1 hour to 4 days of operation).
I don't know what the cause is.
I had this problem with 7.4 and unfortunately also with 8.0.3 (fresh install with all updates).
How do I find what is wrong and fix it?
The syslog is not helping me at the moment.

At the request of Spirit, I am posting the various results he requested.

 

[mjaxon]

I had a similar issue that related specifically to the r8169 realtek driver being used. This post has the steps I used to resolve this however you should make sure you have a realtek card and the driver is correct.

https://forum.proxmox.com/threads/proxmox-ve-8-0-released.129320/post-568121

 

[pga57]

I've not a realtek but an Intel X540 10Go. I had the same problem with the integrated network card Intel I211-AT.
So I think there is another reason... which I can't find :-(

Thank's for your suggestion.

 

[fiona]

Hi,
there's nothing special in the logs (assuming it's a standalone node ) except for

Jun 29 11:28:31 pve pvestatd[1201]: storage 'Diskstation' is not online

Was the connection lost at that point?

I've not a realtek but an Intel X540 10Go. I had the same problem with the integrated network card Intel I211-AT.
So I think there is another reason... which I can't find :-(

So the same issue occurred with two completely different network cards?

Please share the output of pveversion -v and after the issue occurre, the file created by journalctl -b0 > /tmp/journal.log. Is the status of the interface DOWN in the output of ip a after you lose connection?

Pinging @spirit because he had already answered you in the other thread.

 

[pga57]

there's nothing special in the logs (assuming it's a standalone node ) except for

Jun 29 11:28:31 pve pvestatd[1201]: storage 'Diskstation' is not online

Was the connection lost at that point?

Diskstation (my Synology Nas) was online but pve network down.

So the same issue occurred with two completely different network cards?

Yes, exactly.

Please share the output of pveversion -v and after the issue occurre, the file created by journalctl -b0 > /tmp/journal.log.

Pve network ok since 1 day. I'm waiting and will post the results when it happens.

Is the status of the interface DOWN in the output of ip a after you lose connection?

I can not access pve when it is append, and have no monitor but i will see to it next time.

Pinging @spirit because he had already answered you in the other thread.

Ok.
Thank's
Philippe

 

[fiona]

Pve network ok since 1 day. I'm waiting and will post the results when it happens.
I can not access pve when it is append, and have no monitor but i will see to it next time.

If you need to reboot to access the node again, you need to use the command journalctl -b-1 > /tmp/journal.log instead to get the log from the previous boot (where the issue occurred).

 

[pga57]

Hi,
Network connection lost this night ;-(
Here the journal.log from the previous boot.

 

[pga57]

Lost connection again this morning.
Could this incident be the result of PVE not being able to connect via NFS to my NAS within a given time? (the connection can sometimes take several tens of seconds)

 

[spirit]

Do you also have network lost with a simple ping ?

or it is only on some specific protocol ? (nfs (tcp?udp ?) , corosync ,... ) ?

I ask that, because I had similar problem with kernel 6.2 with corosync in udp mode. (Never find the problem, but too much fragmented udp packet was breaking my kernel stack, only for udp, and other protocol (icmp, tcp) were working fine).
This was occuring between 1-4 days, but I was able to reproduce it with flooding bad udp packets in some minutes.
This was with mellanox nic, but I don't think that it's nic model related.

 

[pga57]

pve is unreachable when it's happen. No respond with ping.

 

[spirit]

pve is unreachable when it's happen. No respond with ping.

ok seem different problem.
do you have any kernel logs in #dmesg ?

 

[pga57]

ok seem different problem.
do you have any kernel logs in #dmesg ?

Apparently not.

 

[fiona]

Lost connection again this morning.
Could this incident be the result of PVE not being able to connect via NFS to my NAS within a given time? (the connection can sometimes take several tens of seconds)

Does the issue happen around the time you take backups? You could try setting a bandwidth limit there and reduce the amount of workers: https://forum.proxmox.com/threads/t...ad-behavior-during-backup.118430/#post-513106

 

[pga57]

Does the issue happen around the time you take backups?

it seems so.

You could try setting a bandwidth limit there and reduce the amount of workers: https://forum.proxmox.com/threads/t...ad-behavior-during-backup.118430/#post-513106

I have no idea how much limit the bandwidth.
I didn't find where configure the amount of workers because I don't have a cluster directory on my pve as mentioned in the thread.

Yesterday, however, I created a new nfs share folder for the backups and no loss of network... for the moment ;-)