Unable to ping or ssh from secondary pve host to primary pve host

[Dwain]

Hi,

I'm trying to set up a cluster with two servers that are on the same subnet with no luck. Firewall isn't enabled on any of the servers. I've checked my internal firewall and confirmed none of the traffic is being blocked by router. I need to remove the block to my Primary server that only resides on the same subnet.

Below is a list of what I'm working with.

1. Primary Server: Supermicro X8DT6:
a. Few LXCs and VMs
b. pve version 6.3-3
c. Couple four port nics bonded with LACP on vmbr1 & 2
d. Two on board single nics vmbr0 & 3

2. Secondary Server: Dell PowerEdge R630XL
a. New PVE Install
b. No LXCs or VMs
c. pve version 6.3-3

3. Pfsense router:
a. Multiple subnets
b. version 2.4.5

4. Cisco Catalyst 3560E -48 PoE Switch:
a. DHCP disabled
b. Default gateway <IP of subnet>
c. VLAN = 1

Issues:
1. I'm unable to PING or SSH from the Secondary server to the Primary server using either IP or hostname.
a. Secondary to Primary PING returns no error just hangs indefinitely until I cancel it.
b. Secondary to Primary SSH returns no error just hangs until I cancel it.
c. The same result is present while trying to PING or SSH from any of the LXCs or VMs on to the Primary server to the Primary server host.
d. I can SSH or PING the Primary server from any other subnet except the subnet it resides on.
e. traceroute from Secondary server to Primary server returns * * * from all 30 hops max

2. I'm unable to ping or ssh from the Primary server to the Secondary server using either IP or hostname.
a. Primary to Secondary PING returns Destination Host Unreachable.
b. Primary to Seondary SSH returns ssh: connect to host <ipaddress or hostname> port 22: No route to host.
c. Primary to Secondary traceroute returns one entry: x.x.x.x (x.x.x.x) 3071.462 ms !H 3071.447 ms !H 3071.441 ms !H
d. I can SSH or PING the Secondary server from any subnet in my network.
e. I can SSH or PING any other host including LXC or VM in my network from the Secondary server.

3. I can create a cluster on either server but unable to join TASK ERROR: 500 Can't connect to <ipaddress>:8006 (Connection timed out)

Primary Server

pveversion -v
proxmox-ve: 6.3-1 (running kernel: 5.4.78-1-pve)
pve-manager: 6.3-3 (running version: 6.3-3/eee5f901)
pve-kernel-5.4: 6.3-3
pve-kernel-helper: 6.3-3
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.78-1-pve: 5.4.78-1
pve-kernel-5.4.73-1-pve: 5.4.73-1
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-4.10.11-1-pve: 4.10.11-9
ceph-fuse: 14.2.15-pve3
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libproxmox-backup-qemu0: 1.0.2-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.3-3
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.5-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-3
pve-cluster: 6.2-1
pve-container: 3.3-1
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-7
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-2
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.5-pve1


Secondary Server

pveversion -v
proxmox-ve: 6.3-1 (running kernel: 5.4.78-2-pve)
pve-manager: 6.3-3 (running version: 6.3-3/eee5f901)
pve-kernel-5.4: 6.3-3
pve-kernel-helper: 6.3-3
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libproxmox-backup-qemu0: 1.0.2-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.3-3
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.5-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-3
pve-cluster: 6.2-1
pve-container: 3.3-1
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-7
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-2
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.5-pve1[/B]

 

[nktech1135]

Hi.
Could you post the contents of /etc/network/interfaces from both hosts?
Also, log into both hosts and varify firewall is disabled, it is by default so unless you ever enabled it it should still be disabled.

Thanks.

 

[Dwain]

Hi nktech1135,

1. I just checked the firewall on the Primary server and I found one entry (unknown to me but not surprised because I installed the Nagios agent on that server).

root@dtrinibs:~# pve-firewall status
Status: disabled/running


2. The secondary server's firewall has no entries.

root@mains:~# pve-firewall status
Status: disabled/running


3. Primary server:

root@dtrinibs:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface enp9s0f0 inet manual

iface enp9s0f1 inet manual

iface enp10s0f0 inet manual

iface enp10s0f1 inet manual

iface enp6s0f0 inet manual

iface enp3s0 inet manual

iface enp6s0f1 inet manual

iface enp4s0 inet manual

iface enp6s0f2 inet manual

iface enp6s0f3 inet manual

auto bond1
iface bond1 inet static
address 10.1.3.34
netmask 24
bond-slaves enp6s0f0 enp6s0f1 enp6s0f2 enp6s0f3
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3
#Quad Port PCIe NIC1

auto bond2
iface bond2 inet static
address 10.1.3.35
netmask 24
bond-slaves enp10s0f0 enp10s0f1 enp9s0f0 enp9s0f1
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3
#Quad Port PCIe NIC2

auto vmbr0
iface vmbr0 inet static
address 10.1.3.30
netmask 255.255.255.0
gateway 10.1.3.1
bridge-ports enp3s0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#Main NIC

auto vmbr1
iface vmbr1 inet static
address 10.1.3.32
netmask 255.255.255.0
bridge-ports bond2
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#FreeNAS NICs

auto vmbr2
iface vmbr2 inet static
address 10.1.3.33
netmask 255.255.255.0
bridge-ports bond1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#Media NICs

auto vmbr3
iface vmbr3 inet static
address 10.1.3.31
netmask 255.255.255.0
bridge-ports enp4s0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#Other NIC


4. Secondary Server:

root@mains:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno3 inet manual

iface eno4 inet manual

iface eno2 inet manual

auto bond0
iface bond0 inet manual
bond-slaves eno2 eno3 eno4
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3
#Tri Port Onboard NICs

auto vmbr0
iface vmbr0 inet static
address 10.1.3.40/24
gateway 10.1.3.1
bridge-ports eno1
bridge-stp off
bridge-fd 0

auto vmbr1
iface vmbr1 inet static
address 10.1.3.41/24
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#Three Onboard NICs

 

[Dwain]

I'm guessing this is a unique case and not very common.

 

[nktech1135]

Hi.
Question, from the main host can you ping any of the vmbr interface addresses? 10.1.3.30, 10.1.3.31?
Also, why is the bond config different between the hosts?
From what you posted this should work. Using something like tcpdump to see the packets might be needed.

Thanks.

 

[Dwain]

Yes I can ping all the other IP interfaces from the main host:

root@dtrinibs:/var/log# ping localhost
PING localhost.localdomain (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost.localdomain (127.0.0.1): icmp_seq=1 ttl=64 time=0.033 ms
64 bytes from localhost.localdomain (127.0.0.1): icmp_seq=2 ttl=64 time=0.024 ms
^C
--- localhost.localdomain ping statistics ---
2 packets transmitted, 3 received, 0% packet loss, time 77ms
rtt min/avg/max/mdev = 0.024/0.033/0.041/0.009 ms

root@dtrinibs:/var/log# traceroute localhost
traceroute to localhost (127.0.0.1), 30 hops max, 60 byte packets
1 localhost.localdomain (127.0.0.1) 0.280 ms 0.245 ms 0.231 ms

root@dtrinibs:/var/log# ping localhost
PING localhost.localdomain (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost.localdomain (127.0.0.1): icmp_seq=1 ttl=64 time=0.033 ms
64 bytes from localhost.localdomain (127.0.0.1): icmp_seq=2 ttl=64 time=0.037 ms
^C
--- localhost.localdomain ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 58ms
rtt min/avg/max/mdev = 0.025/0.031/0.037/0.008 ms

root@dtrinibs:/var/log# ping 10.1.3.21
PING 10.1.3.21 (10.1.3.21) 56(84) bytes of data.
cFrom 10.1.3.34 icmp_seq=1 Destination Host Unreachable
From 10.1.3.34 icmp_seq=2 Destination Host Unreachable
From 10.1.3.34 icmp_seq=3 Destination Host Unreachable
^[[A^C
--- 10.1.3.21 ping statistics ---
5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 77ms
pipe 4

root@dtrinibs:/var/log# ping 10.1.3.31
PING 10.1.3.31 (10.1.3.31) 56(84) bytes of data.
64 bytes from 10.1.3.31: icmp_seq=1 ttl=64 time=0.025 ms
64 bytes from 10.1.3.31: icmp_seq=2 ttl=64 time=0.045 ms
64 bytes from 10.1.3.31: icmp_seq=3 ttl=64 time=0.024 ms
^C
--- 10.1.3.31 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 26ms
rtt min/avg/max/mdev = 0.024/0.031/0.045/0.010 ms

root@dtrinibs:/var/log# ping 10.1.3.32
PING 10.1.3.32 (10.1.3.32) 56(84) bytes of data.
64 bytes from 10.1.3.32: icmp_seq=1 ttl=64 time=0.036 ms
64 bytes from 10.1.3.32: icmp_seq=2 ttl=64 time=0.025 ms
^C
--- 10.1.3.32 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 26ms
rtt min/avg/max/mdev = 0.025/0.030/0.036/0.007 ms

root@dtrinibs:/var/log# ping 10.1.3.33
PING 10.1.3.33 (10.1.3.33) 56(84) bytes of data.
64 bytes from 10.1.3.33: icmp_seq=1 ttl=64 time=0.034 ms
64 bytes from 10.1.3.33: icmp_seq=2 ttl=64 time=0.044 ms
^C
--- 10.1.3.33 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 32ms
rtt min/avg/max/mdev = 0.034/0.039/0.044/0.005 ms

root@dtrinibs:/var/log# ping 10.1.3.34
PING 10.1.3.34 (10.1.3.34) 56(84) bytes of data.
64 bytes from 10.1.3.34: icmp_seq=1 ttl=64 time=0.023 ms
64 bytes from 10.1.3.34: icmp_seq=2 ttl=64 time=0.025 ms
64 bytes from 10.1.3.34: icmp_seq=3 ttl=64 time=0.025 ms
^C
--- 10.1.3.34 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 29ms
rtt min/avg/max/mdev = 0.023/0.024/0.025/0.004 ms

root@dtrinibs:/var/log# ping 10.1.3.35
PING 10.1.3.35 (10.1.3.35) 56(84) bytes of data.
64 bytes from 10.1.3.35: icmp_seq=1 ttl=64 time=0.034 ms
64 bytes from 10.1.3.35: icmp_seq=2 ttl=64 time=0.026 ms
^C
--- 10.1.3.35 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 14ms
rtt min/avg/max/mdev = 0.026/0.030/0.034/0.004 ms

 

[Dwain]

Note:
I tried accessing accessing the GUI from the same subnet, for the first time since setting everything up and I can't access it.
I've never had to access it from the same subnet because my sever is on another floor in my home.

However I can access the GUI for the Secondary server from the same subnet or any other subnet in my network.

 

[Dwain]

The Bond config on Primary host vs Secondary host is different because during the Secondary host bond creation when I enter IPv4/CIDR I get an error. When I try remove that config on the Primary host I get and error as well.

 

[Dwain]

I just need to be pointed in the right direction to removing whatever is blocking communication to and from the same subnet on the Primary server.

 

[Dwain]

While trying to make changes to the network config on any of the interfaces I was presented with an error related to bond1 or bond2
iface bond1 - ip address can't be set on interface if bridged in vmbr2 (500)

I decided to do the following which resolved the issue:
1. Delete vmbr2 and vmbr3.
2. Remove the IPs assigned to bond1 and bond2 (This was assignment was allowed during pve version 5.x build and version 6.x and removed the capability to do this.)
3. Re-create vmbr2 and vmbr3 with the respected bonds.

Thanks to

nktech1135​

For asking the question about why Primary and Secondary bond config being different, that set me on the path to figuring out the issue.