CEPH on node upgrade issue

GoZippy

Active Member
Nov 27, 2020
113
2
38
45
www.gozippy.com
Since update on one of my nodes it has become a bit unstable.. well 3 of my nodes - but they have different issues. Some dealing with network problems...
For node 5 - I have ceph monitor that is showing as "undefined" in the list of monitors on any node I connect to the gui with.

root@node5:~# ceph mon stats no valid command found; 10 closest matches: mon ok-to-stop <ids>... mon ok-to-add-offline mon ok-to-rm <id> mon scrub mon metadata [<id>] mon count-metadata <property> mon versions mon dump [<epoch:int>] mon stat mon getmap [<epoch:int>] Error EINVAL: invalid command root@node5:~# ceph osd status ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE 0 stack1 4505M 927G 0 0 0 0 exists,up 1 node2 81.9G 849G 0 0 0 0 exists,up 2 node3 83.6G 847G 0 2457 0 0 exists,up 3 node4 85.4G 846G 0 0 0 0 exists,up 4 0 0 0 0 0 0 autoout,exists 5 0 0 0 0 0 0 exists 6 0 0 0 0 0 0 exists root@node5:~# ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -2 hdd 6.36789 root default~hdd -6 hdd 0.90970 host node2~hdd 1 hdd 0.90970 osd.1 -8 hdd 0.90970 host node3~hdd 2 hdd 0.90970 osd.2 -10 hdd 0.90970 host node4~hdd 3 hdd 0.90970 osd.3 -12 hdd 0.90970 host node5~hdd 4 hdd 0.90970 osd.4 -16 hdd 0.90970 host node6~hdd 6 hdd 0.90970 osd.6 -14 hdd 0.90970 host node8~hdd 5 hdd 0.90970 osd.5 -4 hdd 0.90970 host stack1~hdd 0 hdd 0.90970 osd.0 -1 6.36789 root default -5 0.90970 host node2 1 hdd 0.90970 osd.1 -7 0.90970 host node3 2 hdd 0.90970 osd.2 -9 0.90970 host node4 3 hdd 0.90970 osd.3 -11 0.90970 host node5 4 hdd 0.90970 osd.4 -15 0.90970 host node6 6 hdd 0.90970 osd.6 -13 0.90970 host node8 5 hdd 0.90970 osd.5 -3 0.90970 host stack1 0 hdd 0.90970 osd.0 root@node5:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp2s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000 link/ether 00:0e:b6:5c:a3:e8 brd ff:ff:ff:ff:ff:ff 3: enp2s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:0e:b6:5c:a3:e9 brd ff:ff:ff:ff:ff:ff 4: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:0e:b6:a8:54:b3 brd ff:ff:ff:ff:ff:ff 5: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:0e:b6:a8:54:b4 brd ff:ff:ff:ff:ff:ff 6: enp1s0f2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:0e:b6:a8:54:b5 brd ff:ff:ff:ff:ff:ff 7: enp1s0f3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:0e:b6:a8:54:b6 brd ff:ff:ff:ff:ff:ff 8: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 00:0e:b6:5c:a3:e8 brd ff:ff:ff:ff:ff:ff inet 10.0.1.5/16 scope global vmbr0 valid_lft forever preferred_lft forever inet6 fe80::20e:b6ff:fe5c:a3e8/64 scope link valid_lft forever preferred_lft forever root@node5:~#
1609946020018.png
1609946684387.png



1609946654884.png
Any ideas where to start?

I did restart each node... reboot.. but it still acts like the old version installed... any command line interface commands I can run to get some more intel on what is going on?
 
Last edited:
Some dealing with network problems...
First get the network reliably working. Then the rest of the issue might fall in line.
 
Any suggestions for setup?

Dell r210ii server
2 on board nic
4 on card nic
(see above)...

I have had a fit with trying to figure out best setup to stack these and use ceph or management bond over a pair to speed up data or separate the management from the public/behind router private lan...

Anyhow - I have a couple nodes that are just not working anymore on the network side of things... need to figure out how to identify which port the computer thinks the physical cable is attached to... I have identical server and all plugged in on one port on the main board - most boxes work if I configure eno1 and bridge to eno1 on proxmox setup - but some seem to only respond if I use enps0f1 or something... anyhow - just a little lost on the network setup on a couple of them (node 7 and 8 I can't seem to get going again after update)... no network... but this node above, node 5, is just giving me ceph issues...
 
so how do I do that with the above? any setup you recommend?
As you stated yourself:
Anyhow - I have a couple nodes that are just not working anymore on the network side of things...
Do this first and check that every node can ping each other on every interface/network configured. Including MTU.
 
So I guess my questions is do I configure eno1 as the node IP and then vmbr as the virtual machine requested IP?

What I want is each node to be 10.0.1.1 thru 10.0.1.8 and each VM to be 10.0.1.101 thru XXX

I lose connection to the node if I set eno1 to a specific IP like 10.0.1.5 for node 5 but when I reload and log in to console from the physical machine connection and reset the config I can get proxmox gui back and this seems to be the only thing that works... setting vmbr1 as a specific static IP...

Then in the VM I add VMBR1 and set the vm to static requesting a specific IP like 10.0.1.103

It works - sorta... but not seeing it from outside my own network (probably a DDWRT routing issue on my own side) but can get to it from behind the DDWRT on the lan 10.0.x.x net

Just lost on where to be setting the NODE IP and configure the VM IP with 6 NIC cards... all physical connections are removed except for eno1

Oddly - some nodes report enpsof1 as the link that is up even though the physical connection is on the on-board nic not the 4 port card installed. Odd things indeed...

1611538776123.png
 
Last edited:
Yeah - I can get bridge to work for private net ip ranges... but not figured out how to send the public ip address thru to the VM...

I have ISP Fiber modem - switch - DDWRT - SWITCH - Cluster nodes
ISP Modem in passthru (but also serves own 192.168.1.x clients direct connected )
DDWRT WAN is assigned public IP
I have routing thru DDWRT for the public IPs (I think)

StartupWANIF=`nvram get wan_iface`
WANMASK=`nvram get wan_netmask`
ifconfig $WANIF:1 104.8.233.106 netmask $WANMASK broadcast 104.8.233.111
ifconfig $WANIF:2 104.8.233.107 netmask $WANMASK broadcast 104.8.233.111
ifconfig $WANIF:3 104.8.233.108 netmask $WANMASK broadcast 104.8.233.111
ifconfig $WANIF:4 104.8.233.109 netmask $WANMASK broadcast 104.8.233.111




Firewall# WAN .106 -> LAN .6
iptables -t nat -I PREROUTING -d 104.8.233.106 -j DNAT --to 10.0.1.6
iptables -t nat -I POSTROUTING -s 10.0.1.6 -j SNAT --to 104.8.233.106
iptables -I FORWARD -d 10.0.1.6 -j ACCEPT

# WAN .107 -> LAN .7
iptables -t nat -I PREROUTING -d 104.8.233.1067-j DNAT --to 10.0.1.7
iptables -t nat -I POSTROUTING -s 10.0.1.7 -j SNAT --to 104.8.233.107
iptables -I FORWARD -d 10.0.1.7 -j ACCEPT

# WAN .108 -> LAN .8
iptables -t nat -I PREROUTING -d 104.8.233.108 -j DNAT --to 10.0.1.8
iptables -t nat -I POSTROUTING -s 10.0.1.8 -j SNAT --to 104.8.233.108
iptables -I FORWARD -d 10.0.1.8 -j ACCEPT

# WAN .109 -> LAN .9
iptables -t nat -I PREROUTING -d 104.8.233.109 -j DNAT --to 10.0.1.9
iptables -t nat -I POSTROUTING -s 10.0.1.9 -j SNAT --to 104.8.233.109
iptables -I FORWARD -d 10.0.1.9 -j ACCEPT

------------------
------------------

I have the DDRWT LAN side attached to switch going to servers on board eno1 (though some are reporting as though physically attached to enpsof range... odd).

anyhow so I assign eno1 and vmbridge vmbr0 on eno1 as 10.0.1.x range

I have vmbr0 as the primary net hardware for eacn VM on that box for now.

I ask vm running ubuntu to pull a public IP for instance 104.8.233.109 and it does not seem to work. If I ask it to pull in the 10.0.x.x (/16) I have setup on DDWRT and other net it works fine... but I need to be able to use those IP and all port ranges so I am not messing up port forwarding on the ranges available only on the one IP picked up by the DDWRT...

Ideas welcome...

Was reading many forums for network setup but none show a public IP to a private router with private lan mixed with forwarding public IP thru server to a VM picking up that public IP behind a private net... was hopeful the passthru mode on the ISP router/modem with DDWRT box pre and post routing would make it exposed straight to those servers so the instance VM can bridge right thru and pickup...

Problem I see is gateway and masquerading issues ...

SO if the vmbr0 is on 10.0.1.100 with gateway set to 10.0.0.1 (DDWRT) I am on that subnet and not able to pull the 104.8.233.x public ip ranges...

I added vmbr1 tied to another port (each server has 2 onboard and 4 on card nic 1gb)

Still trying to sort out what is best setting for network on each proxmox box to ensure I can assign public IP's to a specific instance and have it follow across the cluster without intervention for HA failover as needed.

My VMs are on ceph osd's. With the 1GB link between servers I am getting 22 sec to recovery from fail to live migration to available node. Not bad. I would like to bond 2 of the 4 lines to each machine thru dual ports on 2 different GB switches for true 2 GB bonded link.. I think... so failover recovery can be faster with migrations and data channel can be prioritized on those links.

Each server has 2 on board 1gb and 4 1gb on a card inserter. so plenty of ports to make it happen even if I need to do jumper between ports for simple routing...

Ideas very much welcomed.

1611600029345.png


1611600097792.png


1611600214489.png

anyhow - if there is a solution I am not seeing it rightly here - just been staring at it too long I think..
 
I have routing thru DDWRT for the public IPs (I think)
Judging from the iptables lines, its NAT. All public IPs terminate on the router. The guests behind the router will only see the internal IP. The router is rewriting it according to the iptable rules. Best to only forward the ports really needed.

Routing the network will mean to use a IP of the public subnet on the router as GW. And provide the upstream router with routing information (if not already done so by the provider). This should also be separated into a different security zone.
 
my goal is to get rid of the ddwrt and replace with 2 node failover router setup of some sort.. I have all those dell r210ii servers with 6 nic cards I was hoping to go bare metal pf/opnsense or some firewall like option or something with better failover and threat detenction ClearOs... who knows... anyhow ideas welcome - but for what I have with the single point of failure being the DDWRT between the proxmox nodes and the uplink WAN to ISP I need to sort it out... I believe the DDWRT router is set to simply forward and translate everything fine with pre and post in and out... that may be another discussion anyhow... I would be happy to just keep it out of the picture and use only as wifi for the house devices and separate the nodes under another option...
I have 5 public IP's that I need to assign to certain VM's and setup domain certs for subdomains I will host on them locally whereas I have most other services, (mail, mysql services, webhost and other domain services) on a remote hosted server at a colocation facility.
Question is how to get the VM to pickup the public IP directly... when you route from internet to those IPs I have it dies at the DDWRT and cannot go further. I have the DDWRT set as one of them and that works fine and I can of course forward any port I need but it does not help with my larger issue of passing the ip traffic for the public IP based on each IP to a specific VM
 
I have 5 public IP's that I need to assign to certain VM's
Either use NAT or (if you have a subnet) route the public IPs. That's much like routing private IP networks.

when you route from internet to those IPs I have it dies at the DDWRT and cannot go further.
I am not certain what you mean by 'dies' but the dd-wrt will be the gateway for those guests.

my goal is to get rid of the ddwrt and replace with 2 node failover router setup of some sort..
Do you have more than one internet connection? Basically, when the modem dies, than the failover will not be of much use (if that's the sole purpose). A less complex solution would be to have a cold standby server, that is easily replaced.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!