[SOLVED] LXC can't access internet and I can only SSH to it one time

C

crawforc3

New Member
Apr 6, 2023
12
1
3
I have a small home lab with three nodes. The hosts are pve1, pve2, pve3.

The problem is that I have an ubuntu container on pve3 that is having network(?) issues. This just started happening recently, I'm not sure why.


What I know:

1. I can reliably access the container shell from the pve3 web GUI


2. I can only SSH into the container right after it boots, and only once. After that I get "No route to host" errors.

Code: 
ssh: connect to host 192.168.1.57 port 22: No route to host


3. I cannot ping 1.1.1.1

Code: 
# ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
From 191.168.1.57 icmp_seq=1 Destination Host Unreachable


4. I cannot ping the gateway

Code: 
# ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
From 191.168.1.57 icmp_seq=1 Destination Host Unreachable




ip route show
Code: 
default via 192.168.1.0 dev eth0 proto static 
191.168.1.0/24 dev eth0 proto kernel scope link src 191.168.1.57 
192.168.1.0 dev eth0 proto static scope link



systemctl status networkd-dispatcher.service networking.service network-online.target network-pre.target network.target | cat
Code: 
Unit networking.service could not be found.
* networkd-dispatcher.service - Dispatcher daemon for systemd-networkd
     Loaded: loaded (/lib/systemd/system/networkd-dispatcher.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2023-06-12 15:55:44 UTC; 2min 42s ago
   Main PID: 105 (networkd-dispat)
      Tasks: 1 (limit: 77000)
     Memory: 10.1M
        CPU: 79ms
     CGroup: /system.slice/networkd-dispatcher.service
             `-105 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers

Jun 12 15:55:44 ubuntucont systemd[1]: Starting Dispatcher daemon for systemd-networkd...
Jun 12 15:55:44 ubuntucont networkd-dispatcher[105]: No valid path found for iwconfig
Jun 12 15:55:44 ubuntucont networkd-dispatcher[105]: No valid path found for iw
Jun 12 15:55:44 ubuntucont systemd[1]: Started Dispatcher daemon for systemd-networkd.

* network-online.target - Network is Online
     Loaded: loaded (/lib/systemd/system/network-online.target; static)
     Active: active since Mon 2023-06-12 15:57:44 UTC; 42s ago
       Docs: man:systemd.special(7)
             https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget

Jun 12 15:57:44 ubuntucont systemd[1]: Reached target Network is Online.

* network-pre.target - Preparation for Network
     Loaded: loaded (/lib/systemd/system/network-pre.target; static)
     Active: active since Mon 2023-06-12 15:55:44 UTC; 2min 43s ago
       Docs: man:systemd.special(7)
             https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget

Jun 12 15:55:44 ubuntucont systemd[1]: Reached target Preparation for Network.

* network.target - Network
     Loaded: loaded (/lib/systemd/system/network.target; static)
     Active: active since Mon 2023-06-12 15:55:44 UTC; 2min 42s ago
       Docs: man:systemd.special(7)
             https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget

Jun 12 15:55:44 ubuntucont systemd[1]: Reached target Network.


These are the warnings and errors I found since the last boot (journalctl --boot=0)
Code: 
Jun 12 16:49:19 ubntu systemd-networkd[91]: Failed to increase receive buffer size for general netlink socket, ignoring: Operation not permitted
Jun 12 16:49:19 ubntu systemd-networkd[91]: /etc/systemd/network/eth0.network:8: DHCP=v6 is deprecated, please use DHCP=ipv6 instead.


Jun 12 16:49:19 ubntu rsyslogd[106]: imklog: cannot open kernel log (/proc/kmsg): Permission denied.
Jun 12 16:49:19 ubntu rsyslogd[106]: activation of module imklog failed [v8.2112.0 try https://www.rsyslog.com/e/2145 ]

Jun 12 16:49:31 ubntu login[115]: pam_systemd(login:session): Failed to create session: Seat has no VTs but VT number not 0

Jun 12 16:51:19 ubntu systemd-networkd-wait-online[99]: Timeout occurred while waiting for network connectivity.
Jun 12 16:51:19 ubntu systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Jun 12 16:51:19 ubntu systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Jun 12 16:51:19 ubntu systemd[1]: Failed to start Wait for Network to be Configured.

Jun 12 16:51:33 ubntu [407]: Critical: libusb_init failed



I'm not sure what to look at next?
 
Last edited:
Just now, I was able to ping 1.1.1.1.

So I quickly looked at ip a and ip route show and then immediately tried to ping 1.1.1.1 again and it no longer worked.

As I was typing this response, it suddenly started pinging again. in fact, if I just leave ping running, it starts and stops intermittently.





Code: 
$# ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=3.97 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=57 time=3.87 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=57 time=2.78 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=57 time=3.53 ms
64 bytes from 1.1.1.1: icmp_seq=5 ttl=57 time=3.58 ms
^C
--- 1.1.1.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4007ms
rtt min/avg/max/mdev = 2.784/3.547/3.970/0.416 ms

$# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0@if37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 1e:8e:6a:ff:d3:56 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.1.57/24 brd 192.168.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::1c8e:6aff:feff:d356/64 scope link
       valid_lft forever preferred_lft forever

$# ip route show
default via 192.168.1.1 dev eth0 proto static
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.57

$# ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.

... 2 minutes later ...

64 bytes from 1.1.1.1: icmp_seq=154 ttl=57 time=3.30 ms
64 bytes from 1.1.1.1: icmp_seq=155 ttl=57 time=3.55 ms
64 bytes from 1.1.1.1: icmp_seq=156 ttl=57 time=3.98 ms
 
Last edited:
Looking at timestamps for pings over the last hour is super weird...

The network seems to be go up for about 2 minutes, down for 4 minutes, then repeats.

up 2 -- down 4 -- up 2 -- down 4
 
Try to find out if you have another device claiming the same IP address. "up 2 -- down 4 -- up 2 -- down 4" sounds exactly like this.

How is .57 assigned? By DHCP? Statically? Try "the other" method...
 
UdoB said:
Try to find out if you have another device claiming the same IP address. "up 2 -- down 4 -- up 2 -- down 4" sounds exactly like this.

How is .57 assigned? By DHCP? Statically? Try "the other" method...
Click to expand...
I was wondering it was an IP collision too. I set it statically but I switched to DHCP and was still getting weird behavior.

I ended up powering everything down, unplugged/re-plugged all the ethernet cables (I have young kids, so who knows if they were touching things). Anyway, things work now. I kind of think the router was the problem. I rebooted it a few times but hadn't removed power.
 
Recent ping timestamps from the LXC show it had a break for exactly 4 minutes.

Code: 
Jul 08 16:08:39 64 bytes from 1.1.1.1: icmp_seq=23 ttl=57 time=3.45 ms
Jul 08 16:12:39 64 bytes from 1.1.1.1: icmp_seq=258 ttl=57 time=3.49 ms


Have you created a new Ubuntu container (on pve3, same Ubuntu version) to see if this has the same problem(s) ?
Click to expand...
I have not done this, I will try and report back.


Can pve3 host itself ping 1.1.1.1 without issues?
Click to expand...
Yes, the host can ping 1.1.1.1 without issues
 
crawforc3 said:
Recent ping timestamps from the LXC show it had a break for exactly 4 minutes.

Code: 
Jul 08 16:08:39 64 bytes from 1.1.1.1: icmp_seq=23 ttl=57 time=3.45 ms
Jul 08 16:12:39 64 bytes from 1.1.1.1: icmp_seq=258 ttl=57 time=3.49 ms



I have not done this, I will try and report back.



Yes, the host can ping 1.1.1.1 without issues
Click to expand...

I was wondering why you did not create a fresh new Ubuntu container earlier.
Is this "faulty" container so important for you? Does it run special software?

- Perhaps your existing Ubuntu lxc template has an error in it. Delete this template and download a new fresh copy and re-deploy it.
 
No, it's not that important. I did not create a new LXC because I'm a networking noob and thought I might learn something about it first. It is just so weird that the outage is like clockwork.
 
@Spoonman2002 I created a Debian 12 LXC. The static IP is the same as the old LXC. After letting ping run for a while, I came back and the issue still persists. Meanwhile, the host never loses connection.
 
Found this in the syslogs, but I'm not sure it's relevent:
Code: 
Jul 09 21:10:19 lxc systemd-networkd-wait-online[84]: Timeout occurred while waiting for network connectivity.
Jul 09 21:10:19 lxc systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Jul 09 21:10:19 lxc systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Jul 09 21:10:19 lxc systemd[1]: Failed to start systemd-networkd-wait-online.service - Wait for Network to be Configured.
Jul 09 21:10:19 lxc systemd[1]: Reached target network-online.target - Network is Online.
 
You must log in or register to reply here.
Top Bottom