[SOLVED] Why is this simple bridge network not receiving ARP replies correctly?

[meitar]

Hi. This is my first post. I've been experimenting with Proxmox and have been scratching my head trying to solve this issue for the last day or so.

The short version: I have three devices. There is the Proxmox VE host itself (let's call it pve, running Proxmox VE 6.2-11), a single virtualized guest running inside the Proxmox VE hypervisor (let's call it VM100), and another Linux computer that is completely independent of Proxmox (let's call it laptop). I want to set things up such that all of these three devices are connected on the same LAN, a typical "default configuration using a bridge" as described by the Proxmox wiki. The catch is that all of these devices are running inside VirtualBox 6.1.14 (the latest stable version as of this writing) because, as I said, I've just begun experimenting with Proxmox and am not ready to commit hardware to it.

What I'm experiencing is a lack of connectivity between VM100 (the Proxmox guest) and laptop (the computer outside Proxmox but on the same LAN as the Proxmox VE host), despite perfect connectivity between pve (the Proxmox host) and laptop. Specifically, if I try to ping laptop from VM100, I observe the following sequence of events on the Proxmox host (pve).

1. ARP request is sent from VM100 (Proxmox guest).
2. Proxmox host sees the request being sent.
3. laptop sees the ARP request intended for it ("who has…?"), and sends a reply ("is at…") intended for VM100.
4. Proxmox host (pve) doesn't see the ARP reply at all, for some reason, and so neither does its guest, resulting in a failure to connect.

It's that last bit I'm confused by because, again, there's no other connectivity issue between the laptop and the pve node at all.

The configuration details follow.

Here's how I've set up my Proxmox host's /etc/network/interfaces file:

root@pve:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual

auto vmbr0
iface vmbr0 inet dhcp
    bridge-ports eth0
    bridge-stp off
    bridge-fd 0
#DHCP to Internet

auto eth1
iface eth1 inet manual

auto vmbr1
iface vmbr1 inet static
    address 10.10.10.2/24
    bridge-ports eth1
    bridge-stp off
    bridge-fd 0
#Static IPs to hostonly network

In other words, the Proxmox host has two NICs. The first, eth0 is the NAT'ed connection to the Internet configured with DHCP. The second, eth1 is attached to the VirtualBox hostonly network. This also follows the direction of the Proxmox wiki's "Proxmox VE inside VirtualBox" page.

Both of these connections are bridged (to vmbr0 and vmbr1, respectively), so that Proxmox's guests can also have the same setup: one NIC connected to the Internet via NAT and DHCP auto-configuration, and another connected to a private LAN.

And here's how I've set up my single test VM (VM100) hosted by Proxmox:

root@pve:~# cat /etc/pve/nodes/pve/qemu-server/100.conf
acpi: 1
agent: enabled=0,fstrim_cloned_disks=0,type=virtio
arch: x86_64
balloon: 0
bios: seabios
boot: cd
bootdisk: scsi0
#cipassword: [REDACTED]
ciuser: ubuntu
cores: 1
cpu: cputype=qemu64
cpuunits: 1024
ide2: local-lvm:vm-100-cloudinit,media=cdrom
ipconfig0: ip=dhcp
ipconfig1: ip=10.10.10.50/24
keyboard: en-us
memory: 512
name: testvm
net0: e1000=D2:91:C2:04:67:90,bridge=vmbr0
net1: e1000=66:B6:78:6E:92:42,bridge=vmbr1
ostype: other
scsi0: local-lvm:vm-100-disk-0,size=8G
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=19f3a52c-e6a6-4321-b909-fa9f548adea1
sockets: 1
#sshkeys: [REDACTED]
tablet: 1
template: 0
vga: memory=16,type=std
vmgenid: 61746464-665e-4788-af0a-b6652f36d99d

Notice:

• I was careful to use e1000 network device emulation because, again, Proxmox VE itself is a VirtualBox guest. The Proxmox guest is thus nested inside a VirtualBox hypervisor and the private LAN to which net1 is attached (via bridge=vmbr1) is a VirtualBox hostonly network.
• The IP address for this VM on the hostonly network that is causing trouble is 10.10.10.50.

This configuration, set up via cloud-init, is active:

ubuntu@testvm:~$ ip a show eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 66:b6:78:6e:92:42 brd ff:ff:ff:ff:ff:ff
    inet 10.10.10.50/24 brd 10.10.10.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::64b6:78ff:fe6e:9242/64 scope link
       valid_lft forever preferred_lft forever

So, let's try pinging the guest from either another Ubuntu Linux machine (a VirtualBox guest attached to the same hostonly interface as the Proxmox VE host), or from my actual physical host. In either case, my expectation is that since the Proxmox VE guest (VM100) is bridged to a VirtualBox Host-Only network, I should be able to contact it. However, I can't:

vagrant@10.10.10.20:~$ ping -c 4 10.10.10.50
PING 10.10.10.50 (10.10.10.50) 56(84) bytes of data.

--- 10.10.10.50 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3328ms

What's going on? Here's output of tcpdump -i vmbr1 -v host 10.10.10.20 or host 10.10.10.50 running on pve (the Proxmox VE node) while doing the above ping:

root@pve:~# tcpdump -i vmbr1 -v host 10.10.10.50 or 10.10.10.20
tcpdump: listening on vmbr1, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel

As you can see, there is absolutely no traffic at all captured. Weird, right? So let's try the other direction: pinging from the Proxmox guest, through the bridge, to laptop. Here's what pve sees this time:

root@pve:~# tcpdump -i vmbr1 -v host 10.10.10.50 or 10.10.10.20
tcpdump: listening on vmbr1, link-type EN10MB (Ethernet), capture size 262144 bytes
07:31:16.369083 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.10.10.20 tell 10.10.10.50, length 46
07:31:17.375724 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.10.10.20 tell 10.10.10.50, length 46
07:31:18.399932 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.10.10.20 tell 10.10.10.50, length 46
07:31:19.424524 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.10.10.20 tell 10.10.10.50, length 46
07:31:20.447462 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.10.10.20 tell 10.10.10.50, length 46
07:31:21.471552 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.10.10.20 tell 10.10.10.50, length 46

Okay! So the Proxmox guest is sending out ARP requests. Is the laptop seeing those requests? Here's a sudo tcpdump -i enp0s8 -n -v host 10.10.10.50 or 10.10.10.20 on the laptop while pinging it from VM100:

vagrant@10.10.10.20:~$ sudo tcpdump -i enp0s8 -n -v host 10.10.10.50 or host 10.10.10.20
tcpdump: listening on enp0s8, link-type EN10MB (Ethernet), capture size 262144 bytes
06:35:51.749683 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.10.10.20 tell 10.10.10.50, length 46
06:35:51.749712 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.10.10.20 is-at 08:00:27:02:b2:48, length 28
06:35:52.774498 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.10.10.20 tell 10.10.10.50, length 46
06:35:52.774539 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.10.10.20 is-at 08:00:27:02:b2:48, length 28
06:35:53.800175 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.10.10.20 tell 10.10.10.50, length 46
06:35:53.800225 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.10.10.20 is-at 08:00:27:02:b2:48, length 28
06:35:54.821756 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.10.10.20 tell 10.10.10.50, length 46
06:35:54.821776 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.10.10.20 is-at 08:00:27:02:b2:48, length 28
06:35:55.867627 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.10.10.20 tell 10.10.10.50, length 46
06:35:55.867652 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.10.10.20 is-at 08:00:27:02:b2:48, length 28
06:35:56.877819 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.10.10.20 tell 10.10.10.50, length 46
06:35:56.877842 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.10.10.20 is-at 08:00:27:02:b2:48, length 28

Okay! So ARP requests are able to traverse the bridge as expected. Moreover, ARP replies are being emitted, as well.

But why aren't those replies being received by the Proxmox VE host? Any guidance will be appreciated.

 

[spirit]

Hi, if proxmox don't see arp replies, it's because it don't receive them.

your laptop is correctly replying.

maybe try to tcpdump on the virtualbox server, something is maybe filtered here.

 

[meitar]

Hi, if proxmox don't see arp replies, it's because it don't receive them.

your laptop is correctly replying.

maybe try to tcpdump on the virtualbox server, something is maybe filtered here.

Thanks for the reply.

I don't completely understand your suggestion. The packet captures I posted demonstrate the issue from the perspective of the Proxmox VE host (pve) as well as from the workstation connected to the same VirtualBox host-only network (called laptop). In this configuration, my expectation is that the machine running VirtualBox would simply see the same exact ARP broadcast traffic as the laptop VM because it is directly attached to the same virtual switch, i.e., VirtualBox's internal switch.

Sure enough, this is exactly what I see. Here's a packet capture from the perspective of the VirtualBox host computer itself while the VM100 machine (the guest inside Proxmox) is pinging the laptop (VirtualBox-hosted guest workstation). Note that the packet capture is listening on interface vboxnet3, which is the interface for the VirtualBox Host-Only network in question.

$ sudo tcpdump -i vboxnet3 -n host 10.10.10.50 or host 10.10.10.20
Password:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vboxnet3, link-type EN10MB (Ethernet), capture size 262144 bytes
23:25:31.396266 ARP, Request who-has 10.10.10.20 tell 10.10.10.50, length 46
23:25:31.396270 ARP, Request who-has 10.10.10.20 tell 10.10.10.50, length 46
23:25:31.396537 ARP, Reply 10.10.10.20 is-at 08:00:27:02:b2:48, length 46
23:25:32.410290 ARP, Request who-has 10.10.10.20 tell 10.10.10.50, length 46
23:25:32.410298 ARP, Request who-has 10.10.10.20 tell 10.10.10.50, length 46
23:25:32.410702 ARP, Reply 10.10.10.20 is-at 08:00:27:02:b2:48, length 46
23:25:33.433883 ARP, Request who-has 10.10.10.20 tell 10.10.10.50, length 46
23:25:33.433890 ARP, Request who-has 10.10.10.20 tell 10.10.10.50, length 46
23:25:33.434295 ARP, Reply 10.10.10.20 is-at 08:00:27:02:b2:48, length 46

This shows the same problem as described in the original post: that the VirtualBox host machine is also seeing the relevant ARP queries and the replies, but that the ARP requests are being repeated in very quick succession presumably because the replies are never being received by the machine requesting them.

Is this a Proxmox bug? Is it a VirtualBox bug? Is it a Proxmox-in-VirtualBox bug? I'm not trying to do anything complicated with this…right? Like, this should be pretty straightforward, yes?

Thanks again for having a look.

 

[spirit]

Hi, this is strange indeed.

I don't known if it's possible in virtualbox, too see mac address table of the virtualbox bridge, and see if you correctly see the mac address of the vm mapped to the proxmox vm ?

 

[meitar]

I've solved the problem.

It turns out that VirtualBox's default is to disallow its VMs from being able to run in promiscuous mode. This means that the Proxmox VE host pve was the only machine to have a registered MAC address in the VirtualBox internal swtich. No traffic in Ethernet frames whose destination MAC address different from the Proxmox VE's MAC address (generated by VirtualBox upon creation of the VM) would be passed to the Proxmox VE guest. This explains why the Proxmox VE machine itself was accessible from other VMs, and why it was passing traffic to other devices on the VirtualBox Host-Only network LAN, but why the Proxmox VE's own guests (nested inside VirtualBox) were unable to receive any traffic destined for them.

The setting is buried in the VirtualBox GUI (VM Settings -> Network -> Adapter N [where N is the adaper number] -> Advanced -> Promiscuous Mode) or can be modified using the VirtualBox CLI like so:

vboxmanage modifyvm <uuid|name> --nicpromiscN <allow-vms|allow-all>

In my scenario, above, the allow-vms option is sufficient, but it would prevent the VirtualBox host itself from being able to communicate with the Proxmox VE guests. If you want to speak to the nested VMs inside Proxmox VE from the VirtualBox host itself, the allow-all value is needed.

Thanks for pointing me in the right direction, spirit.

 

[spirit]

Great ! Thanks for the report !