Strange behaviour with cidr and cluster

[GvdL]

I currently have 2 machines in a cluster and wanted to add the 3rd. however I have an issue with the networking part.

on hosts 1 and 2 the networking is as followed:

Node 1

Spoiler: Spoiled for readablility

auto lo
iface lo inet loopback

iface enp3s0f0 inet manual

iface enp3s0f1 inet manual

iface enp4s0f0 inet manual

auto enp4s0f1
iface enp4s0f1 inet static
        address  10.254.0.1
        netmask  24
#Cluster Interconnect

auto bond0
iface bond0 inet manual
        bond-slaves enp3s0f0 enp3s0f1 enp4s0f0
        bond-miimon 100
        bond-mode balance-alb

auto vmbr0
iface vmbr0 inet manual
        address  192.168.1.204
        netmask  24
        gateway  192.168.1.1
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

Node 2

Spoiler: Spoiled for readablility

auto lo
iface lo inet loopback

iface enp3s0f0 inet manual

iface enp3s0f1 inet manual

iface enp4s0f0 inet manual

auto enp4s0f1
iface enp4s0f1 inet static
        address  10.254.0.2
        netmask  24
#Cluster Interconnect

auto bond0
iface bond0 inet manual
        bond-slaves enp3s0f0 enp3s0f1 enp4s0f0
        bond-miimon 100
        bond-mode balance-alb
        mtu 1500

auto vmbr0
iface vmbr0 inet static
        address  192.168.1.203
        netmask  24
        gateway  192.168.1.1
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 1500

Node 3 <- this one needs to be added

Spoiler: Spoiled for readablility

auto lo
iface lo inet loopback

iface enp2s0f0 inet manual

auto enp2s0f1
iface enp2s0f1 inet static
        address 10.254.0.3
        netmask  24
#Cluster Interconnect

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.206
        netmask  24
        gateway 192.168.1.254
        bridge-ports enp2s0f0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 1500

However if i want to add node 3 to the cluster I get the following message:





what am I missing here? I cannot find the problem?

 

[GvdL]

Did some more testing, appears that the complete CIDR notation is messed up. A running machine with IP addressess filled in in the gui lose their /24 (in this case) this appears to be a bug in the software.



quick IP statistics that shows that its now magically a /32:

2: enp2s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether d8:d3:85:e0:e7:20 brd ff:ff:ff:ff:ff:ff
3: enp2s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
link/ether d8:d3:85:e0:e7:22 brd ff:ff:ff:ff:ff:ff
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether d8:d3:85:e0:e7:22 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.206/32 scope global vmbr0
valid_lft forever preferred_lft forever
inet6 fe80::dad3:85ff:fee0:e720/64 scope link
valid_lft forever preferred_lft forever

this goes for the 10.254.0.3 interface too. Can someone confirm with the laterst pve public download?

 

[aaron]

Can you please send the output of pveversion -v?
Is this installed from our installer or on top of Debian?

 

[GvdL]

this is installed with your iso proxmox-ve_6.1-1.iso and then upgraded to the latest version

the pveversion -v output

proxmox-ve: 6.1-2 (running kernel: 5.3.18-2-pve)
pve-manager: 6.1-7 (running version: 6.1-7/13e58d5e)
pve-kernel-helper: 6.1-7
pve-kernel-5.3: 6.1-5
pve-kernel-5.3.18-2-pve: 5.3.18-2
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph: 14.2.6-pve1
ceph-fuse: 14.2.6-pve1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 2.0.1-1+pve4
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libpve-access-control: 6.0-6
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.0-13
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-4
libpve-storage-perl: 6.1-5
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-3
pve-cluster: 6.1-4
pve-container: 3.0-21
pve-docs: 6.1-6
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.0-10
pve-firmware: 3.0-6
pve-ha-manager: 3.0-8
pve-i18n: 2.0-4
pve-qemu-kvm: 4.1.1-3
pve-xtermjs: 4.3.0-1
qemu-server: 6.1-6
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1

Though I have got the cluster working with the cli installer but I'm scared to change any network setting at this moment.

 

[t.lamprecht]

we're onto it, thanks for your report!

 

[t.lamprecht]

A fix is available on pvetest in libpve-common-perl version 6.0-16.

Could you please try it, either enable the pvetest repository: https://pve.proxmox.com/wiki/Package_Repositories#sysadmin_test_repo

Or download that specific version manually:

wget http://download.proxmox.com/debian/pve/dists/buster/pvetest/binary-amd64/libpve-common-perl_6.0-16_all.deb
apt install ./libpve-common-perl_6.0-16_all.deb

 

[GvdL]

as far as I can test now (like I mentioned that the server now is in a cluster) the opening of the network interface at least gives me the correct cidr notation



and when changing the cidr notation of that interface the correct things are applied in the preview

--- /etc/network/interfaces    2020-03-12 10:20:39.040249951 +0100
+++ /etc/network/interfaces.new    2020-03-12 16:52:06.604379306 +0100
@@ -16,14 +16,12 @@
 
 auto enp2s0f1
 iface enp2s0f1 inet static
-    address 10.254.0.3
-        netmask  24
+    address 10.254.0.3/23
 #Cluster Interconnect
 
 auto vmbr0
 iface vmbr0 inet static
-    address 192.168.1.206
-        netmask  24
+    address 192.168.1.206/24
     gateway 192.168.1.254
     bridge-ports enp2s0f0
     bridge-stp off

 

[t.lamprecht]

OK, thanks for your testing and feedback!

 

[dropndestroy]

@t.lamprecht Just wanted to report that the CIDR issue is resolved for me as well with the test package. Thanks!

 

[t.lamprecht]

Thanks for the additionall feedback, I move the version over to pve-no-subscription, so the newest version rolled out there should be ok.

 

[dropndestroy]

Though the root issue from OP is still there: creating new clusters fails due the format of the ip/cidr. See the error below:

Parameter verification failed. (400)

link0: invalid format - format error link0.address: invalid format - value does not look like a valid address: 172.16.8.11/16
link1: invalid format - format error link1.address: invalid format - value does not look like a valid address: 172.16.9.11/16

 

[spirit]

Though the root issue from OP is still there: creating new clusters fails due the format of the ip/cidr. See the error below:

Parameter verification failed. (400)

link0: invalid format - format error link0.address: invalid format - value does not look like a valid address: 172.16.8.11/16
link1: invalid format - format error link1.address: invalid format - value does not look like a valid address: 172.16.9.11/16

I think that it's expecting an ip address without netmask. does it work if you simply remove "/16" from the gui ?

 

[spirit]

seem to come from networkselector, calling the api /node/network to list interface (same than the main list of interface in network section) , and take the "address" field which is now a cidr.

I'll try to look if we couldn't use a format for networkselector (address|cidr|networkaddress), as if I remember, they was also old problem with ceph form, when the address expect a network address, and not a host address.

 

[t.lamprecht]

IMO, address/address6 should stay adress I think to much expects it and we have the cider field already, so maybe it's better to keep those cleanly separated (address+nemask from cidr) ?

 

[spirit]

IMO, address/address6 should stay adress I think to much expects it and we have the cider field already, so maybe it's better to keep those cleanly separated (address+nemask from cidr) ?

yes, I think the api should return "address" as an ip only to avoid breaking

something like:

API2/Network.pm

__PACKAGE__->register_method({
    name => 'index',
    path => '',
    method => 'GET',
..

       my $tmp = PVE::INotify::read_file('interfaces', 1);
        my $config = $tmp->{data};
        my $changes = $tmp->{changes};

        $rpcenv->set_result_attrib('changes', $changes) if $changes;

        my $ifaces = $config->{ifaces};

        delete $ifaces->{lo}; # do not list the loopback device

        #rewrite "address" field from cidr to ip
        foreach my $k (keys %$ifaces) {
                 my $iface = $ifaces->{$k};
                 ($iface->{address}, undef) = $iface->{cidr} =~ m!^(\S+?)/(\S+)$! if $iface->{cidr};
                 ($iface->{address6}, undef) = $iface->{cidr6} =~ m!^(\S+?)/(\S+)$! if $iface->{cidr6};

        }

(could be interesting to return also a "network_address" for ceph wizard)

 

[t.lamprecht]

I'll probably do it in INotify, i.e., cidr is always cidr, address is always address and netmask is always mask (for reading out, write should accept all valid variants and write that as you recently changed - that's OK).
Let's see on pve-devel

 

[dropndestroy]

Since I'm already rebuilding my cluster, I can test out the devel package. How can I help (more importantly how ) ?

 

[t.lamprecht]

Since I'm already rebuilding my cluster, I can test out the devel package. How can I help (more importantly how ) ?

A updated verison is available, now also ensuring the read directions stays comaptible (with more tests)

wget http://download.proxmox.com/debian/pve/dists/buster/pvetest/binary-amd64/libpve-common-perl_6.0-17_all.deb
apt install ./libpve-common-perl_6.0-17_all.deb

 

[kwinz]

A updated verison is available, now also ensuring the read directions stays comaptible (with more tests)

wget http://download.proxmox.com/debian/pve/dists/buster/pvetest/binary-amd64/libpve-common-perl_6.0-17_all.deb
apt install ./libpve-common-perl_6.0-17_all.deb

I had the same issues. I am new to Proxmox and just started evaluating Proxmox for the first time, and immediately got bitten by this.
I can confirm the update with above libpve-common-perl_6.0-17_all.deb fixed it for me.

Does APT now still automatically update that package or will I be stuck on that version because it thinks I have manually installed that version and want to keep it?

[Edit]: It's not quite fixed. The cluster can be created and configured but when I want to add a second Ceph monitor I now get a new error message:
No active IP found for the requested ceph public network '192.168.0.50/32' on node 'pve2' (500)

[Edit2]: Ok so manually reentering all the network IPs including netmask /24 for all interfaces via GUI, reboot, and subsequently
changing all /32 to /24 in /etc/ceph/ceph.conf fixed it for me. I can now also create monitors!

 

[t.lamprecht]

Does APT now still automatically update that package or will I be stuck on that version because it thinks I have manually installed that version and want to keep it?

with plain apt install it's the same for apt as a normal package update over the repositories got installed, the single thing it may change for packages is, that the package gets marked as "manually selected". This matters only for packages which got indirectly pulled in due to being a dependency for an explicitly installed package, and the only thing it changes is that said marked manual installed package won't get autoremoved when using apt autoremove, that doesn'T matters here but telling for sake of completeness.

Package pinning is possible, but that feature would need to be done explicitly, i.e., via manually specifiying some version/release in /etc/apt/preferences.d.

Glad you could make it work.