UPDATE: If I use DHCP instead of Static IP for LXC, everything is working fine! So as a workaround I for now, I'm using DHCP with static IP assignment to that LXC from my router.
Another thing I have noticed is if I ping the LXC which is configured with DHCP, the ping time/latency is much lower...