[SOLVED] LXC can't access internet and I can only SSH to it one time

C

crawforc3

New Member
Apr 6, 2023
12
1
3
I have a small home lab with three nodes. The hosts are pve1, pve2, pve3.

The problem is that I have an ubuntu container on pve3 that is having network(?) issues. This just started happening recently, I'm not sure why.


What I know:

1. I can reliably access the container shell from the pve3 web GUI


2. I can only SSH into the container right after it boots, and only once. After that I get "No route to host" errors.

Code: 
ssh: connect to host 192.168.1.57 port 22: No route to host


3. I cannot ping 1.1.1.1

Code: 
# ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
From 191.168.1.57 icmp_seq=1 Destination Host Unreachable


4. I cannot ping the gateway

Code: 
# ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
From 191.168.1.57 icmp_seq=1 Destination Host Unreachable




ip route show
Code: 
default via 192.168.1.0 dev eth0 proto static 
191.168.1.0/24 dev eth0 proto kernel scope link src 191.168.1.57 
192.168.1.0 dev eth0 proto static scope link



systemctl status networkd-dispatcher.service networking.service network-online.target network-pre.target network.target | cat
Code: 
Unit networking.service could not be found.
* networkd-dispatcher.service - Dispatcher daemon for systemd-networkd
     Loaded: loaded (/lib/systemd/system/networkd-dispatcher.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2023-06-12 15:55:44 UTC; 2min 42s ago
   Main PID: 105 (networkd-dispat)
      Tasks: 1 (limit: 77000)
     Memory: 10.1M
        CPU: 79ms
     CGroup: /system.slice/networkd-dispatcher.service
             `-105 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers

Jun 12 15:55:44 ubuntucont systemd[1]: Starting Dispatcher daemon for systemd-networkd...
Jun 12 15:55:44 ubuntucont networkd-dispatcher[105]: No valid path found for iwconfig
Jun 12 15:55:44 ubuntucont networkd-dispatcher[105]: No valid path found for iw
Jun 12 15:55:44 ubuntucont systemd[1]: Started Dispatcher daemon for systemd-networkd.

* network-online.target - Network is Online
     Loaded: loaded (/lib/systemd/system/network-online.target; static)
     Active: active since Mon 2023-06-12 15:57:44 UTC; 42s ago
       Docs: man:systemd.special(7)
             https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget

Jun 12 15:57:44 ubuntucont systemd[1]: Reached target Network is Online.

* network-pre.target - Preparation for Network
     Loaded: loaded (/lib/systemd/system/network-pre.target; static)
     Active: active since Mon 2023-06-12 15:55:44 UTC; 2min 43s ago
       Docs: man:systemd.special(7)
             https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget

Jun 12 15:55:44 ubuntucont systemd[1]: Reached target Preparation for Network.

* network.target - Network
     Loaded: loaded (/lib/systemd/system/network.target; static)
     Active: active since Mon 2023-06-12 15:55:44 UTC; 2min 42s ago
       Docs: man:systemd.special(7)
             https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget

Jun 12 15:55:44 ubuntucont systemd[1]: Reached target Network.


These are the warnings and errors I found since the last boot (journalctl --boot=0)
Code: 
Jun 12 16:49:19 ubntu systemd-networkd[91]: Failed to increase receive buffer size for general netlink socket, ignoring: Operation not permitted
Jun 12 16:49:19 ubntu systemd-networkd[91]: /etc/systemd/network/eth0.network:8: DHCP=v6 is deprecated, please use DHCP=ipv6 instead.


Jun 12 16:49:19 ubntu rsyslogd[106]: imklog: cannot open kernel log (/proc/kmsg): Permission denied.
Jun 12 16:49:19 ubntu rsyslogd[106]: activation of module imklog failed [v8.2112.0 try https://www.rsyslog.com/e/2145 ]

Jun 12 16:49:31 ubntu login[115]: pam_systemd(login:session): Failed to create session: Seat has no VTs but VT number not 0

Jun 12 16:51:19 ubntu systemd-networkd-wait-online[99]: Timeout occurred while waiting for network connectivity.
Jun 12 16:51:19 ubntu systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Jun 12 16:51:19 ubntu systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Jun 12 16:51:19 ubntu systemd[1]: Failed to start Wait for Network to be Configured.

Jun 12 16:51:33 ubntu [407]: Critical: libusb_init failed



I'm not sure what to look at next?
 
Last edited:
Just now, I was able to ping 1.1.1.1.

So I quickly looked at ip a and ip route show and then immediately tried to ping 1.1.1.1 again and it no longer worked.

As I was typing this response, it suddenly started pinging again. in fact, if I just leave ping running, it starts and stops intermittently.





Code: 
$# ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=3.97 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=57 time=3.87 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=57 time=2.78 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=57 time=3.53 ms
64 bytes from 1.1.1.1: icmp_seq=5 ttl=57 time=3.58 ms
^C
--- 1.1.1.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4007ms
rtt min/avg/max/mdev = 2.784/3.547/3.970/0.416 ms

$# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0@if37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 1e:8e:6a:ff:d3:56 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.1.57/24 brd 192.168.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::1c8e:6aff:feff:d356/64 scope link
       valid_lft forever preferred_lft forever

$# ip route show
default via 192.168.1.1 dev eth0 proto static
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.57

$# ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.

... 2 minutes later ...

64 bytes from 1.1.1.1: icmp_seq=154 ttl=57 time=3.30 ms
64 bytes from 1.1.1.1: icmp_seq=155 ttl=57 time=3.55 ms
64 bytes from 1.1.1.1: icmp_seq=156 ttl=57 time=3.98 ms
 
Last edited:
Looking at timestamps for pings over the last hour is super weird...

The network seems to be go up for about 2 minutes, down for 4 minutes, then repeats.

up 2 -- down 4 -- up 2 -- down 4
 
Try to find out if you have another device claiming the same IP address. "up 2 -- down 4 -- up 2 -- down 4" sounds exactly like this.

How is .57 assigned? By DHCP? Statically? Try "the other" method...
 
UdoB said:
Try to find out if you have another device claiming the same IP address. "up 2 -- down 4 -- up 2 -- down 4" sounds exactly like this.

How is .57 assigned? By DHCP? Statically? Try "the other" method...
Click to expand...
I was wondering it was an IP collision too. I set it statically but I switched to DHCP and was still getting weird behavior.

I ended up powering everything down, unplugged/re-plugged all the ethernet cables (I have young kids, so who knows if they were touching things). Anyway, things work now. I kind of think the router was the problem. I rebooted it a few times but hadn't removed power.
 
Recent ping timestamps from the LXC show it had a break for exactly 4 minutes.

Code: 
Jul 08 16:08:39 64 bytes from 1.1.1.1: icmp_seq=23 ttl=57 time=3.45 ms
Jul 08 16:12:39 64 bytes from 1.1.1.1: icmp_seq=258 ttl=57 time=3.49 ms


Have you created a new Ubuntu container (on pve3, same Ubuntu version) to see if this has the same problem(s) ?
Click to expand...
I have not done this, I will try and report back.


Can pve3 host itself ping 1.1.1.1 without issues?
Click to expand...
Yes, the host can ping 1.1.1.1 without issues
 
crawforc3 said:
Recent ping timestamps from the LXC show it had a break for exactly 4 minutes.

Code: 
Jul 08 16:08:39 64 bytes from 1.1.1.1: icmp_seq=23 ttl=57 time=3.45 ms
Jul 08 16:12:39 64 bytes from 1.1.1.1: icmp_seq=258 ttl=57 time=3.49 ms



I have not done this, I will try and report back.



Yes, the host can ping 1.1.1.1 without issues
Click to expand...

I was wondering why you did not create a fresh new Ubuntu container earlier.
Is this "faulty" container so important for you? Does it run special software?

- Perhaps your existing Ubuntu lxc template has an error in it. Delete this template and download a new fresh copy and re-deploy it.
 
No, it's not that important. I did not create a new LXC because I'm a networking noob and thought I might learn something about it first. It is just so weird that the outage is like clockwork.
 
@Spoonman2002 I created a Debian 12 LXC. The static IP is the same as the old LXC. After letting ping run for a while, I came back and the issue still persists. Meanwhile, the host never loses connection.
 
Found this in the syslogs, but I'm not sure it's relevent:
Code: 
Jul 09 21:10:19 lxc systemd-networkd-wait-online[84]: Timeout occurred while waiting for network connectivity.
Jul 09 21:10:19 lxc systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Jul 09 21:10:19 lxc systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Jul 09 21:10:19 lxc systemd[1]: Failed to start systemd-networkd-wait-online.service - Wait for Network to be Configured.
Jul 09 21:10:19 lxc systemd[1]: Reached target network-online.target - Network is Online.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back