VMs can't get IP from DHCP after docker engine is installed on Proxmox

dave_ · Dec 12, 2019

I have been experimenting Proxmox for a few days, everything worked until I installed docker (docker-ce v19.03) on pve, then the VMs can't get IP from DHCP anymore.
it came back to normal when I uninstalled docker on pve, so I'm pretty sure it's because of the networking configuration changes docker made on pve, but I'm not a network guy and can't really figure out how to fix it. Your help is greatly appreciated!

Off topic here. The reason I needed to run a separate docker engine on pve is because I need to run `Nvidia-docker` and get GPU acceleration. I've tried PCI passthrough to VM but it didn't work as expected. I've also tried LXC on pve but it's like running docker in another container, and it doesn't really work for the Nvidia-container docker runtime.

So....

My homelab HW is just one box with one gigabit NIC that connects to the internet home router. The router provides DHCP and goes to the internet directly.
Internet Router: 192.168.2.1
Proxmox: 192.168.2.103 (static ip allocated by Router)
Ubuntu VM: 192.168.2.106 (static ip allocated by Router), on the VM it runs a separate Docker engine as well (a separate one from the docker engine I intended to run on pve host)

1. When Docker is NOT installed on pve and ubuntu VM is running, see below output on pve and screenshot for ubuntu vm network config:

Code:

When ubuntu VM is RUNNING, on the pve host:

pve 12|16:46 [~] ip route
default via 192.168.2.1 dev vmbr0 onlink
192.168.2.0/24 dev vmbr0 proto kernel scope link src 192.168.2.103

pve 12|16:54 [~] ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether 04:d9:f5:1c:03:f9 brd ff:ff:ff:ff:ff:ff
3: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 04:d9:f5:1c:03:f9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.103/24 brd 192.168.2.255 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::6d9:f5ff:fe1c:3f9/64 scope link
       valid_lft forever preferred_lft forever
43: tap106i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr106i0 state UNKNOWN group default qlen 1000
    link/ether 7a:57:6e:7f:b9:8c brd ff:ff:ff:ff:ff:ff
44: fwbr106i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 9e:56:0a:15:15:05 brd ff:ff:ff:ff:ff:ff
45: fwpr106p0@fwln106i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 2a:35:12:e1:7d:66 brd ff:ff:ff:ff:ff:ff
46: fwln106i0@fwpr106p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr106i0 state UP group default qlen 1000
    link/ether 9e:56:0a:15:15:05 brd ff:ff:ff:ff:ff:ff

pve 12|16:55 [~] ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether 04:d9:f5:1c:03:f9 brd ff:ff:ff:ff:ff:ff
3: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 04:d9:f5:1c:03:f9 brd ff:ff:ff:ff:ff:ff
43: tap106i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr106i0 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 7a:57:6e:7f:b9:8c brd ff:ff:ff:ff:ff:ff
44: fwbr106i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 9e:56:0a:15:15:05 brd ff:ff:ff:ff:ff:ff
45: fwpr106p0@fwln106i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether 2a:35:12:e1:7d:66 brd ff:ff:ff:ff:ff:ff
46: fwln106i0@fwpr106p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr106i0 state UP mode DEFAULT group default qlen 1000
    link/ether 9e:56:0a:15:15:05 brd ff:ff:ff:ff:ff:ff


Inside Ubuntu VM, see below for `ifconfig -a` output:
ubuntu 12|16:53 [~] ifconfig -a
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        inet6 fe80::42:1ff:fe39:a0f5  prefixlen 64  scopeid 0x20<link>
        ether 02:42:01:39:a0:f5  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 28  bytes 2476 (2.4 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens18: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.2.106  netmask 255.255.255.0  broadcast 192.168.2.255
        inet6 fe80::5ca6:7cff:fe26:47d4  prefixlen 64  scopeid 0x20<link>
        ether 5e:a6:7c:26:47:d4  txqueuelen 1000  (Ethernet)
        RX packets 807  bytes 75521 (75.5 KB)
        RX errors 0  dropped 20  overruns 0  frame 0
        TX packets 170  bytes 21155 (21.1 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 84  bytes 6324 (6.3 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 84  bytes 6324 (6.3 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

dave_ · Dec 12, 2019

2. When Docker is installed and running on pve and ubuntu VM is also running with its own Docker engine:

Code:

on pve host, 

pve 12|17:29 [~] ip route
default via 192.168.2.1 dev vmbr0 onlink
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.18.0.0/16 dev docker_gwbridge proto kernel scope link src 172.18.0.1
192.168.2.0/24 dev vmbr0 proto kernel scope link src 192.168.2.103

pve 12|17:30 [~] ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether 04:d9:f5:1c:03:f9 brd ff:ff:ff:ff:ff:ff
3: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 04:d9:f5:1c:03:f9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.103/24 brd 192.168.2.255 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::6d9:f5ff:fe1c:3f9/64 scope link
       valid_lft forever preferred_lft forever
55: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:d3:67:1c:85 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
56: docker_gwbridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:cc:ae:00:fb brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global docker_gwbridge
       valid_lft forever preferred_lft forever
    inet6 fe80::42:ccff:feae:fb/64 scope link
       valid_lft forever preferred_lft forever
65: veth19c7b99@if64: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker_gwbridge state UP group default
    link/ether 62:95:25:89:27:2a brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::6095:25ff:fe89:272a/64 scope link
       valid_lft forever preferred_lft forever
72: veth801f2e9@if71: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker_gwbridge state UP group default
    link/ether 32:5a:17:93:06:9f brd ff:ff:ff:ff:ff:ff link-netnsid 4
    inet6 fe80::305a:17ff:fe93:69f/64 scope link
       valid_lft forever preferred_lft forever
73: tap106i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr106i0 state UNKNOWN group default qlen 1000
    link/ether 4e:28:20:85:bd:8d brd ff:ff:ff:ff:ff:ff
74: fwbr106i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f6:1e:d2:6f:21:e8 brd ff:ff:ff:ff:ff:ff
75: fwpr106p0@fwln106i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 96:c2:4b:bd:ec:5e brd ff:ff:ff:ff:ff:ff
76: fwln106i0@fwpr106p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr106i0 state UP group default qlen 1000
    link/ether f6:1e:d2:6f:21:e8 brd ff:ff:ff:ff:ff:ff

pve 12|17:32 [~] ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether 04:d9:f5:1c:03:f9 brd ff:ff:ff:ff:ff:ff
3: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 04:d9:f5:1c:03:f9 brd ff:ff:ff:ff:ff:ff
55: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
    link/ether 02:42:d3:67:1c:85 brd ff:ff:ff:ff:ff:ff
56: docker_gwbridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
    link/ether 02:42:cc:ae:00:fb brd ff:ff:ff:ff:ff:ff
65: veth19c7b99@if64: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker_gwbridge state UP mode DEFAULT group default
    link/ether 62:95:25:89:27:2a brd ff:ff:ff:ff:ff:ff link-netnsid 2
72: veth801f2e9@if71: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker_gwbridge state UP mode DEFAULT group default
    link/ether 32:5a:17:93:06:9f brd ff:ff:ff:ff:ff:ff link-netnsid 4
73: tap106i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr106i0 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 4e:28:20:85:bd:8d brd ff:ff:ff:ff:ff:ff
74: fwbr106i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether f6:1e:d2:6f:21:e8 brd ff:ff:ff:ff:ff:ff
75: fwpr106p0@fwln106i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether 96:c2:4b:bd:ec:5e brd ff:ff:ff:ff:ff:ff
76: fwln106i0@fwpr106p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr106i0 state UP mode DEFAULT group default qlen 1000
    link/ether f6:1e:d2:6f:21:e8 brd ff:ff:ff:ff:ff:ff

pve 12|17:32 [~] iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy DROP)
target     prot opt source               destination
DOCKER-USER  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain DOCKER (2 references)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             172.18.0.3           tcp dpt:9001

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-USER (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

on Ubuntu VM, it can't get IP from DHCP when pve docker is running. From the router, I can see that it does not receive DHCP request from ubuntu

t.lamprecht · Dec 13, 2019

dave_ said:
I have been experimenting Proxmox for a few days, everything worked until I installed docker (docker-ce v19.03) on pve, then the VMs can't get IP from DHCP anymore.

Running Docker CE along side Proxmox VE on the host is not supported.

dave_ said:
I've tried PCI passthrough to VM but it didn't work as expected.

How so? Can't you do PCI(e) pass-through and run docker-ce with that nvidia stuff in the VM?

dave_ said:
When Docker is installed and running on pve and ubuntu VM is also running with its own Docker engine:

Does the VM has the same addresses and routes? Then this probably a address conflict?
Maybe change the CIDR network ranges from one docker installation:
https://docs.docker.com/v17.09/engine/userguide/networking/default_network/custom-docker0/

IMO, this has nothing to do with Proxmox VE, it would be probably better to ask on Docker support channels.
I'd say that this would happen with any nested setup, e.g., a plain ubuntu with a ubuntu VM and docker in both.

dave_ · Jan 1, 2020

Thanks [B]t.lamprecht[/B] for your comments!
The issue was due to multiple docker instances have conflicting CIDR ranges.
After changing the CIDR settings, everything is running fine now, thanks again

Search

Search

VMs can't get IP from DHCP after docker engine is installed on Proxmox

dave_

New Member

Attachments

dave_

New Member

t.lamprecht

Proxmox Staff Member

dave_

New Member

We value your privacy