ProxMox with 3 Node Cluster - Network error

Sarlis Dimitris

Active Member
Oct 19, 2018
26
2
43
43
Hi all, we started a new project of 3 node machine cluster to be used for about 6 users in a simple small network.
We already setup all machines with pending matter to join them into cluster but first i want to enable the community subscription purchased, make the updates (setup ceph-nautilus) and proceed to cluster connection.

Thing is that i get errors while trying to activate the subscription and while trying to update the server.
Maybe cause of the setup of network? From the installation we added the network and gateway of this network and not in my office.
Should this cause a problem? I mean when the system use a bridged network, how can i check that server (not VMs, VMs are ok with internet) goes to internet?

Here is System info from network in last (3rd) node:
Code:
==== info about network ====

# ip -details -statistics address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 minmtu 0 maxmtu 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
    RX: bytes  packets  errors  dropped overrun mcast   
    213732     790      0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns
    213732     790      0       0       0       0       
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000
    link/ether 9c:b6:54:ed:7b:e1 brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 60 maxmtu 9200
    bridge_slave state forwarding priority 32 cost 4 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.9c:b6:54:ed:7b:e1 designated_root 8000.9c:b6:54:ed:7b:e1 hold_timer    0.00 message_age_timer    0.00 forward_delay_timer    0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
    RX: bytes  packets  errors  dropped overrun mcast   
    8757833    32333    0       0       0       1362   
    TX: bytes  packets  errors  dropped carrier collsns
    12320226   17352    0       0       0       0       
3: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 9c:b6:54:ed:7b:e1 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535
    bridge forward_delay 0 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q bridge_id 8000.9c:b6:54:ed:7b:e1 designated_root 8000.9c:b6:54:ed:7b:e1 root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer    0.00 tcn_timer    0.00 topology_change_timer    0.00 gc_timer   89.17 vlan_default_pvid 1 vlan_stats_enabled 0 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 1 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 16 mcast_hash_max 4096 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3124 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
    inet 192.168.2.203/24 brd 192.168.2.255 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::9eb6:54ff:feed:7be1/64 scope link
       valid_lft forever preferred_lft forever
    RX: bytes  packets  errors  dropped overrun mcast   
    8304820    32332    0       0       0       0       
    TX: bytes  packets  errors  dropped carrier collsns
    12320226   17352    0       0       0       0       

# ip -details -4 route show
unicast 192.168.2.0/24 dev vmbr0 proto kernel scope link src 192.168.2.203

# ip -details -6 route show
unicast ::1 dev lo proto kernel scope global metric 256 pref medium
unicast fe80::/64 dev vmbr0 proto kernel scope global metric 256 pref medium

# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface enp2s0 inet static
    address  192.168.108.30
    netmask  24
    gateway  192.168.108.254

auto vmbr0
iface vmbr0 inet static
    address  192.168.2.203
    netmask  24
    bridge-ports enp2s0
    bridge-stp off
    bridge-fd 0


Error message while suubscription check:
Invalid response from server: 500 Can't connect to shop.maurer-it.com:443 (Temporary failure in name resolution) (500)

Error while trying to update:

starting apt-get update
Err:1 http://ftp.uk.debian.org/debian buster InRelease
Temporary failure resolving 'ftp.uk.debian.org'
Err:2 http://security.debian.org buster/updates InRelease
Temporary failure resolving 'security.debian.org'
Err:3 http://download.proxmox.com/debian/ceph-nautilus buster InRelease
Temporary failure resolving 'download.proxmox.com'
Err:4 https://enterprise.proxmox.com/debian/pve buster InRelease
Temporary failure resolving 'enterprise.proxmox.com'
Err:5 http://ftp.uk.debian.org/debian buster-updates InRelease
Temporary failure resolving 'ftp.uk.debian.org'
Reading package lists...
W: Failed to fetch http://ftp.uk.debian.org/debian/dists/buster/InRelease Temporary failure resolving 'ftp.uk.debian.org'
W: Failed to fetch http://ftp.uk.debian.org/debian/dists/buster-updates/InRelease Temporary failure resolving 'ftp.uk.debian.org'
W: Failed to fetch http://security.debian.org/dists/buster/updates/InRelease Temporary failure resolving 'security.debian.org'
W: Failed to fetch http://download.proxmox.com/debian/ceph-nautilus/dists/buster/InRelease Temporary failure resolving 'download.proxmox.com'
W: Failed to fetch https://enterprise.proxmox.com/debian/pve/dists/buster/InRelease Temporary failure resolving 'enterprise.proxmox.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.
TASK OK

So any help to clarify what is wrong will be greatuful accepted.



Dimitris
 
I had the 8.8.8.8
8.8.8.1

but anyhow i changed the IP inside vmbr0 to my local network and I can see that problem is resolved,
In the other 2 nodes, I have 2 network cards so may I use one for getting out to web for updates etc and the other for my internal network?
I will do so with new bonds?
 
glad you found a solution! (I just noticed now that you did not have a default route for ipv4 in the pasted configuration - that likely was the root of the problem)

In the other 2 nodes, I have 2 network cards so may I use one for getting out to web for updates etc and the other for my internal network?
I will do so with new bonds?

if you have 2 network cards you can create 1 bond with them?
You can also use them individually and configure one for your internal network and the other with another network where the default route is configured (that will be used for network traffic)

check out the reference documentation about the network setup: https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysadmin_network_configuration

I hope this helps!
 
I will keep open the thread cause i need to run some extra actions to check how it will go. I will revert once eveything is settled to confirm operational status.
 
  • Like
Reactions: Stoiko Ivanov

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!