networking has stopped for guests

I

Ismael

New Member
Jun 28, 2018
12
2
3
78
The server here is in a remote data center. I can ssh in, and from there networking seems fine. But none of the guest machines on the server can reach anything when they all worked at last test, a couple of weeks ago. I had nothing to do with configuring this machine to start with, and I'm not sure where I should be looking for problems.

/etc/network/interfaces
Code: 
auto lo
iface lo inet loopback

auto eth1
iface eth1 inet manual

auto eth1.71
iface eth1.71 inet manual

auto vmbr0
iface vmbr0 inet static
        address 10.20.100.150
        netmask 255.255.252.0
        gateway 10.20.100.1
        bridge_ports eth1.71
        bridge_stp off
        bridge_fd 0

auto vmbr1
iface vmbr1 inet manual
        bridge_ports eth1
        bridge_stp off
        bridge_fd 0

ifconfig
Code: 
eth1      Link encap:Ethernet  HWaddr 24:6e:96:44:ac:d0  
          inet6 addr: fe80::266e:96ff:fe44:acd0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:77939 errors:0 dropped:0 overruns:0 frame:0
          TX packets:42854 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:17139780 (16.3 MiB)  TX bytes:28252527 (26.9 MiB)

eth1.1    Link encap:Ethernet  HWaddr 24:6e:96:44:ac:d0  
          inet6 addr: fe80::266e:96ff:fe44:acd0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:388 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:61966 (60.5 KiB)

eth1.71   Link encap:Ethernet  HWaddr 24:6e:96:44:ac:d0  
          inet6 addr: fe80::266e:96ff:fe44:acd0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:55520 errors:0 dropped:0 overruns:0 frame:0
          TX packets:35560 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:14835899 (14.1 MiB)  TX bytes:27672242 (26.3 MiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:38001 errors:0 dropped:0 overruns:0 frame:0
          TX packets:38001 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:24270216 (23.1 MiB)  TX bytes:24270216 (23.1 MiB)

tap101i0  Link encap:Ethernet  HWaddr ca:d7:c9:06:3c:fc  
          inet6 addr: fe80::c8d7:c9ff:fe06:3cfc/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:1182 errors:0 dropped:0 overruns:0 frame:0
          TX packets:21918 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:174515 (170.4 KiB)  TX bytes:1467561 (1.3 MiB)

tap152i0  Link encap:Ethernet  HWaddr 7e:73:3f:ce:c9:fc  
          inet6 addr: fe80::7c73:3fff:fece:c9fc/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:596 errors:0 dropped:0 overruns:0 frame:0
          TX packets:82 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:96258 (94.0 KiB)  TX bytes:5520 (5.3 KiB)

venet0    Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
          inet6 addr: fe80::1/128 Scope:Link
          UP BROADCAST POINTOPOINT RUNNING NOARP  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:3 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

vmbr0     Link encap:Ethernet  HWaddr 24:6e:96:44:ac:d0  
          inet addr:10.20.100.150  Bcast:10.20.103.255  Mask:255.255.252.0
          inet6 addr: fe80::266e:96ff:fe44:acd0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:55228 errors:0 dropped:0 overruns:0 frame:0
          TX packets:35462 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:14821407 (14.1 MiB)  TX bytes:27665670 (26.3 MiB)

vmbr1     Link encap:Ethernet  HWaddr 24:6e:96:44:ac:d0  
          inet6 addr: fe80::266e:96ff:fe44:acd0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:22768 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1201522 (1.1 MiB)  TX bytes:578 (578.0 B)

vmbr1v1   Link encap:Ethernet  HWaddr 24:6e:96:44:ac:d0  
          inet6 addr: fe80::266e:96ff:fe44:acd0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:305 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:52086 (50.8 KiB)  TX bytes:468 (468.0 B)

iptables
Code: 
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Example command line from ps:
Code: 
root        4569       1  4 11:42 ?        00:04:04 /usr/bin/kvm -id 101 -chardev socket,id=qmp,path=/var/run/qemu-server/101.qmp,server,nowait -mon chardev=qmp,mode=control -vnc unix:/var/run/qemu-server/101.vnc,x509,password -pidfile /var/run/qemu-server/101.pid -daemonize -smbios type=1,uuid=14219dbf-eee9-4940-9a57-aaccf5089546 -name pm01-node01 -smp 2,sockets=1,cores=2,maxcpus=2 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000 -vga std -no-hpet -cpu kvm64,hv_spinlocks=0xffff,hv_relaxed,+lahf_lm,+x2apic,+sep -m 2048 -k en-us -cpuunits 1000 -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=uhci.0,port=1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:5540986589c0 -drive if=none,id=drive-ide2,media=cdrom,aio=native -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -drive file=/proxmox/images/101/vm-101-disk-1.raw,if=none,id=drive-ide0,format=raw,cache=writeback,aio=threads,detect-zeroes=on -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=100 -netdev type=tap,id=net0,ifname=tap101i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown -device e1000,mac=32:49:A8:79:97:55,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 -rtc driftfix=slew,base=localtime -global kvm-pit.lost_tick_policy=discard

On the host, something like wget google (no actual link, because new-user restriction) works. On the guests, can't even "ping 8.8.8.8". Also on the guests, the "netstat -nr" output looks like it's not even reaching a DHCP server.

What should I be looking at here?
 
For fun, I tried a tcpdump on one of the tap interfaces

Code: 
17:40:53.696914 ARP, Request who-has 10.20.20.108 tell 10.20.20.10, length 46
17:40:53.719117 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 3e:cf:8c:0a:6b:4d (oui Unknown), length 300
17:40:53.819455 IP 10.20.10.2.1985 > all-routers.mcast.net.1985: HSRPv0-hello 20: state=active group=10 addr=10.20.10.1
17:40:53.859722 IP6 fe80::d09e:af12:32ba:5693.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
17:40:53.984717 IP 169.254.161.242.62911 > 239.255.255.250.1900: UDP, length 133
17:40:54.187901 IP6 fe80::8c09:8167:78f:5258.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
17:40:54.234650 IP 169.254.82.114.61854 > 239.255.255.250.1900: UDP, length 133
17:40:54.343630 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 800a.74:26:ac:8d:20:3c.8084, length 42
17:40:54.343801 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 8014.74:26:ac:8d:20:3c.8084, length 42
17:40:54.344041 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 803c.74:26:ac:8d:20:3c.8084, length 42
17:40:54.469215 IP6 fe80::2584:663a:e932:e341.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
17:40:54.696241 ARP, Request who-has 10.20.20.108 tell 10.20.20.10, length 46
17:40:55.443353 IP 10.20.10.3.1985 > all-routers.mcast.net.1985: HSRPv0-hello 20: state=standby group=10 addr=10.20.10.1
17:40:55.696518 ARP, Request who-has 10.20.20.108 tell 10.20.20.10, length 46
17:40:55.859721 IP6 fe80::d09e:af12:32ba:5693.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
17:40:55.998263 IP 10.20.60.2.1985 > all-routers.mcast.net.1985: HSRPv0-hello 20: state=active group=60 addr=10.20.60.1
17:40:55.998437 IP 10.20.20.2.1985 > all-routers.mcast.net.1985: HSRPv0-hello 20: state=active group=20 addr=10.20.20.1
17:40:56.000460 IP6 fe80::4495:73ad:d24b:6220.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
17:40:56.094139 IP6 fe80::ecdb:cb41:8a00:cf11.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
17:40:56.259110 IP 10.20.60.3.1985 > all-routers.mcast.net.1985: HSRPv0-hello 20: state=standby group=60 addr=10.20.60.1
17:40:56.294214 IP 10.20.20.3.1985 > all-routers.mcast.net.1985: HSRPv0-hello 20: state=standby group=20 addr=10.20.20.1
17:40:56.343609 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 800a.74:26:ac:8d:20:3c.8084, length 42
17:40:56.343794 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 8014.74:26:ac:8d:20:3c.8084, length 42
17:40:56.344034 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 803c.74:26:ac:8d:20:3c.8084, length 42
17:40:56.696420 ARP, Request who-has 10.20.20.108 tell 10.20.20.10, length 46
17:40:56.796444 IP 10.20.10.2.1985 > all-routers.mcast.net.1985: HSRPv0-hello 20: state=active group=10 addr=10.20.10.1
17:40:56.984669 IP 169.254.161.242.62911 > 239.255.255.250.1900: UDP, length 133
17:40:56.985499 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 12:12:a6:bf:b9:eb (oui Unknown), length 300
17:40:56.985792 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 0a:95:b8:98:f6:e6 (oui Unknown), length 300
17:40:57.234676 IP 169.254.82.114.61854 > 239.255.255.250.1900: UDP, length 133
17:40:57.696973 ARP, Request who-has 10.20.20.108 tell 10.20.20.10, length 46
17:40:58.176288 IP 10.20.10.3.1985 > all-routers.mcast.net.1985: HSRPv0-hello 20: state=standby group=10 addr=10.20.10.1
17:40:58.344236 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 800a.74:26:ac:8d:20:3c.8084, length 42
17:40:58.344311 IP6 fe80::fd48:3442:b04:5272.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
17:40:58.344444 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 8014.74:26:ac:8d:20:3c.8084, length 42
17:40:58.344667 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 803c.74:26:ac:8d:20:3c.8084, length 42
17:40:58.644387 IP 10.20.20.2.1985 > all-routers.mcast.net.1985: HSRPv0-hello 20: state=active group=20 addr=10.20.20.1
17:40:58.751739 IP 10.20.60.2.1985 > all-routers.mcast.net.1985: HSRPv0-hello 20: state=active group=60 addr=10.20.60.1
17:40:59.208764 IP 10.20.60.3.1985 > all-routers.mcast.net.1985: HSRPv0-hello 20: state=standby group=60 addr=10.20.60.1
17:40:59.242883 IP 10.20.20.3.1985 > all-routers.mcast.net.1985: HSRPv0-hello 20: state=standby group=20 addr=10.20.20.1
17:40:59.773190 IP 10.20.10.2.1985 > all-routers.mcast.net.1985: HSRPv0-hello 20: state=active group=10 addr=10.20.10.1
17:40:59.859739 IP6 fe80::d09e:af12:32ba:5693.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
17:41:00.344268 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 800a.74:26:ac:8d:20:3c.8084, length 42
17:41:00.344470 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 8014.74:26:ac:8d:20:3c.8084, length 42
17:41:00.344659 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 803c.74:26:ac:8d:20:3c.8084, length 42
17:41:00.631534 ARP, Request who-has 10.20.20.108 tell 10.20.20.10, length 46
17:41:00.908937 IP 10.20.10.3.1985 > all-routers.mcast.net.1985: HSRPv0-hello 20: state=standby group=10 addr=10.20.10.1
17:41:01.289494 IP 10.20.20.2.1985 > all-routers.mcast.net.1985: HSRPv0-hello 20: state=active group=20 addr=10.20.20.1
17:41:01.503861 IP 10.20.60.2.1985 > all-routers.mcast.net.1985: HSRPv0-hello 20: state=active group=60 addr=10.20.60.1
17:41:01.631007 ARP, Request who-has 10.20.20.108 tell 10.20.20.10, length 46
17:41:02.158996 IP 10.20.60.3.1985 > all-routers.mcast.net.1985: HSRPv0-hello 20: state=standby group=60 addr=10.20.60.1
17:41:02.191062 IP 10.20.20.3.1985 > all-routers.mcast.net.1985: HSRPv0-hello 20: state=standby group=20 addr=10.20.20.1
17:41:02.234982 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 9e:08:ed:4b:8a:96 (oui Unknown), length 300
17:41:02.343597 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 800a.74:26:ac:8d:20:3c.8084, length 42
17:41:02.343783 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 8014.74:26:ac:8d:20:3c.8084, length 42
17:41:02.344004 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 803c.74:26:ac:8d:20:3c.8084, length 42

Not sure if that adds anything.
 
I can ping gateway + world from the host system. I can ping nothing from the guest nodes.
 
For experimenting here, how do I configure nodes to be staticly IPed instead of using DHCP?
 
If you have changed nothing before this happened, I'd think that implies an upstream problem. It's possible that your ISP has introduced a change in the way they operate their network. This in fact happened to me recently with similar consequences.
 
JBB said:
post the contents of the /etc/network/interface for on one of the VM's and we can modify it accordingly.
Click to expand...

AKA "fix it on the node"? I hadn't tried that. Nodes are all Windows anyway, so no /etc/network/anything there.
 
Okay, I configured one of the nodes to have a static IP, but I'm not getting anything still. The tap device for that node still shows no IP:

Code: 
tap152i0  Link encap:Ethernet  HWaddr a6:a7:9c:41:7f:c4  
          inet6 addr: fe80::a4a7:9cff:fe41:7fc4/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:702 errors:0 dropped:0 overruns:0 frame:0
          TX packets:50 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:47591 (46.4 KiB)  TX bytes:3408 (3.3 KiB)

And tcpdump on the tap shows a never ending stream of "ARP, Request who-has {gatewayip} tell {nodeip}"
 
Well, replacing (=rebuild on new machine) he DHCP server seems to have fixed this. I don't know why there are tap devices in there, like I said, I got tasked with fixing this after everyone who knew anything about it left. (And left no documentation.)
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back