Ok after spending a day, I found a stable network setting. So it might help some one else:
On a fresh install of Pve 6, /etc/network/interface :
auto lo
iface lo inet loopback
iface enp35s0 inet manual
auto vmbr0
iface vmbr0 inet static
address 135.181.117.227 # Main Ip
netmask...