CEPH on node upgrade issue

[GoZippy]

Since update on one of my nodes it has become a bit unstable.. well 3 of my nodes - but they have different issues. Some dealing with network problems...
For node 5 - I have ceph monitor that is showing as "undefined" in the list of monitors on any node I connect to the gui with.

root@node5:~# ceph mon stats no valid command found; 10 closest matches: mon ok-to-stop <ids>... mon ok-to-add-offline mon ok-to-rm <id> mon scrub mon metadata [<id>] mon count-metadata <property> mon versions mon dump [<epoch:int>] mon stat mon getmap [<epoch:int>] Error EINVAL: invalid command root@node5:~# ceph osd status ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE 0 stack1 4505M 927G 0 0 0 0 exists,up 1 node2 81.9G 849G 0 0 0 0 exists,up 2 node3 83.6G 847G 0 2457 0 0 exists,up 3 node4 85.4G 846G 0 0 0 0 exists,up 4 0 0 0 0 0 0 autoout,exists 5 0 0 0 0 0 0 exists 6 0 0 0 0 0 0 exists root@node5:~# ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -2 hdd 6.36789 root default~hdd -6 hdd 0.90970 host node2~hdd 1 hdd 0.90970 osd.1 -8 hdd 0.90970 host node3~hdd 2 hdd 0.90970 osd.2 -10 hdd 0.90970 host node4~hdd 3 hdd 0.90970 osd.3 -12 hdd 0.90970 host node5~hdd 4 hdd 0.90970 osd.4 -16 hdd 0.90970 host node6~hdd 6 hdd 0.90970 osd.6 -14 hdd 0.90970 host node8~hdd 5 hdd 0.90970 osd.5 -4 hdd 0.90970 host stack1~hdd 0 hdd 0.90970 osd.0 -1 6.36789 root default -5 0.90970 host node2 1 hdd 0.90970 osd.1 -7 0.90970 host node3 2 hdd 0.90970 osd.2 -9 0.90970 host node4 3 hdd 0.90970 osd.3 -11 0.90970 host node5 4 hdd 0.90970 osd.4 -15 0.90970 host node6 6 hdd 0.90970 osd.6 -13 0.90970 host node8 5 hdd 0.90970 osd.5 -3 0.90970 host stack1 0 hdd 0.90970 osd.0 root@node5:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp2s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000 link/ether 00:0e:b6:5c:a3:e8 brd ff:ff:ff:ff:ff:ff 3: enp2s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:0e:b6:5c:a3:e9 brd ff:ff:ff:ff:ff:ff 4: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:0e:b6:a8:54:b3 brd ff:ff:ff:ff:ff:ff 5: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:0e:b6:a8:54:b4 brd ff:ff:ff:ff:ff:ff 6: enp1s0f2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:0e:b6:a8:54:b5 brd ff:ff:ff:ff:ff:ff 7: enp1s0f3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:0e:b6:a8:54:b6 brd ff:ff:ff:ff:ff:ff 8: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 00:0e:b6:5c:a3:e8 brd ff:ff:ff:ff:ff:ff inet 10.0.1.5/16 scope global vmbr0 valid_lft forever preferred_lft forever inet6 fe80::20e:b6ff:fe5c:a3e8/64 scope link valid_lft forever preferred_lft forever root@node5:~#






Any ideas where to start?

I did restart each node... reboot.. but it still acts like the old version installed... any command line interface commands I can run to get some more intel on what is going on?

 

[Alwin]

Some dealing with network problems...

First get the network reliably working. Then the rest of the issue might fall in line.

 

[GoZippy]

Any suggestions for setup?

Dell r210ii server
2 on board nic
4 on card nic
(see above)...

I have had a fit with trying to figure out best setup to stack these and use ceph or management bond over a pair to speed up data or separate the management from the public/behind router private lan...

Anyhow - I have a couple nodes that are just not working anymore on the network side of things... need to figure out how to identify which port the computer thinks the physical cable is attached to... I have identical server and all plugged in on one port on the main board - most boxes work if I configure eno1 and bridge to eno1 on proxmox setup - but some seem to only respond if I use enps0f1 or something... anyhow - just a little lost on the network setup on a couple of them (node 7 and 8 I can't seem to get going again after update)... no network... but this node above, node 5, is just giving me ceph issues...

 

[GoZippy]

First get the network reliably working. Then the rest of the issue might fall in line.

so how do I do that with the above? any setup you recommend?

 

[Alwin]

so how do I do that with the above? any setup you recommend?

As you stated yourself:

Anyhow - I have a couple nodes that are just not working anymore on the network side of things...

Do this first and check that every node can ping each other on every interface/network configured. Including MTU.

 

[GoZippy]

So I guess my questions is do I configure eno1 as the node IP and then vmbr as the virtual machine requested IP?

What I want is each node to be 10.0.1.1 thru 10.0.1.8 and each VM to be 10.0.1.101 thru XXX

I lose connection to the node if I set eno1 to a specific IP like 10.0.1.5 for node 5 but when I reload and log in to console from the physical machine connection and reset the config I can get proxmox gui back and this seems to be the only thing that works... setting vmbr1 as a specific static IP...

Then in the VM I add VMBR1 and set the vm to static requesting a specific IP like 10.0.1.103

It works - sorta... but not seeing it from outside my own network (probably a DDWRT routing issue on my own side) but can get to it from behind the DDWRT on the lan 10.0.x.x net

Just lost on where to be setting the NODE IP and configure the VM IP with 6 NIC cards... all physical connections are removed except for eno1

Oddly - some nodes report enpsof1 as the link that is up even though the physical connection is on the on-board nic not the 4 port card installed. Odd things indeed...

 

[Alwin Antreich]

So I guess my questions is do I configure eno1 as the node IP and then vmbr as the virtual machine requested IP?

IPs for the VMs are configured inside the VM. And either the interface has an IP (without a bridge) or the bridge.

 

[GoZippy]

Yeah - I can get bridge to work for private net ip ranges... but not figured out how to send the public ip address thru to the VM...

I have ISP Fiber modem - switch - DDWRT - SWITCH - Cluster nodes
ISP Modem in passthru (but also serves own 192.168.1.x clients direct connected )
DDWRT WAN is assigned public IP
I have routing thru DDWRT for the public IPs (I think)

StartupWANIF=`nvram get wan_iface`
WANMASK=`nvram get wan_netmask`
ifconfig $WANIF:1 104.8.233.106 netmask $WANMASK broadcast 104.8.233.111
ifconfig $WANIF:2 104.8.233.107 netmask $WANMASK broadcast 104.8.233.111
ifconfig $WANIF:3 104.8.233.108 netmask $WANMASK broadcast 104.8.233.111
ifconfig $WANIF:4 104.8.233.109 netmask $WANMASK broadcast 104.8.233.111




Firewall# WAN .106 -> LAN .6
iptables -t nat -I PREROUTING -d 104.8.233.106 -j DNAT --to 10.0.1.6
iptables -t nat -I POSTROUTING -s 10.0.1.6 -j SNAT --to 104.8.233.106
iptables -I FORWARD -d 10.0.1.6 -j ACCEPT

# WAN .107 -> LAN .7
iptables -t nat -I PREROUTING -d 104.8.233.1067-j DNAT --to 10.0.1.7
iptables -t nat -I POSTROUTING -s 10.0.1.7 -j SNAT --to 104.8.233.107
iptables -I FORWARD -d 10.0.1.7 -j ACCEPT

# WAN .108 -> LAN .8
iptables -t nat -I PREROUTING -d 104.8.233.108 -j DNAT --to 10.0.1.8
iptables -t nat -I POSTROUTING -s 10.0.1.8 -j SNAT --to 104.8.233.108
iptables -I FORWARD -d 10.0.1.8 -j ACCEPT

# WAN .109 -> LAN .9
iptables -t nat -I PREROUTING -d 104.8.233.109 -j DNAT --to 10.0.1.9
iptables -t nat -I POSTROUTING -s 10.0.1.9 -j SNAT --to 104.8.233.109
iptables -I FORWARD -d 10.0.1.9 -j ACCEPT

------------------
------------------

I have the DDRWT LAN side attached to switch going to servers on board eno1 (though some are reporting as though physically attached to enpsof range... odd).

anyhow so I assign eno1 and vmbridge vmbr0 on eno1 as 10.0.1.x range

I have vmbr0 as the primary net hardware for eacn VM on that box for now.

I ask vm running ubuntu to pull a public IP for instance 104.8.233.109 and it does not seem to work. If I ask it to pull in the 10.0.x.x (/16) I have setup on DDWRT and other net it works fine... but I need to be able to use those IP and all port ranges so I am not messing up port forwarding on the ranges available only on the one IP picked up by the DDWRT...

Ideas welcome...

Was reading many forums for network setup but none show a public IP to a private router with private lan mixed with forwarding public IP thru server to a VM picking up that public IP behind a private net... was hopeful the passthru mode on the ISP router/modem with DDWRT box pre and post routing would make it exposed straight to those servers so the instance VM can bridge right thru and pickup...

Problem I see is gateway and masquerading issues ...

SO if the vmbr0 is on 10.0.1.100 with gateway set to 10.0.0.1 (DDWRT) I am on that subnet and not able to pull the 104.8.233.x public ip ranges...

I added vmbr1 tied to another port (each server has 2 onboard and 4 on card nic 1gb)

Still trying to sort out what is best setting for network on each proxmox box to ensure I can assign public IP's to a specific instance and have it follow across the cluster without intervention for HA failover as needed.

My VMs are on ceph osd's. With the 1GB link between servers I am getting 22 sec to recovery from fail to live migration to available node. Not bad. I would like to bond 2 of the 4 lines to each machine thru dual ports on 2 different GB switches for true 2 GB bonded link.. I think... so failover recovery can be faster with migrations and data channel can be prioritized on those links.

Each server has 2 on board 1gb and 4 1gb on a card inserter. so plenty of ports to make it happen even if I need to do jumper between ports for simple routing...

Ideas very much welcomed.









anyhow - if there is a solution I am not seeing it rightly here - just been staring at it too long I think..

 

[Alwin Antreich]

I have routing thru DDWRT for the public IPs (I think)

Judging from the iptables lines, its NAT. All public IPs terminate on the router. The guests behind the router will only see the internal IP. The router is rewriting it according to the iptable rules. Best to only forward the ports really needed.

Routing the network will mean to use a IP of the public subnet on the router as GW. And provide the upstream router with routing information (if not already done so by the provider). This should also be separated into a different security zone.

 

[GoZippy]

my goal is to get rid of the ddwrt and replace with 2 node failover router setup of some sort.. I have all those dell r210ii servers with 6 nic cards I was hoping to go bare metal pf/opnsense or some firewall like option or something with better failover and threat detenction ClearOs... who knows... anyhow ideas welcome - but for what I have with the single point of failure being the DDWRT between the proxmox nodes and the uplink WAN to ISP I need to sort it out... I believe the DDWRT router is set to simply forward and translate everything fine with pre and post in and out... that may be another discussion anyhow... I would be happy to just keep it out of the picture and use only as wifi for the house devices and separate the nodes under another option...
I have 5 public IP's that I need to assign to certain VM's and setup domain certs for subdomains I will host on them locally whereas I have most other services, (mail, mysql services, webhost and other domain services) on a remote hosted server at a colocation facility.
Question is how to get the VM to pickup the public IP directly... when you route from internet to those IPs I have it dies at the DDWRT and cannot go further. I have the DDWRT set as one of them and that works fine and I can of course forward any port I need but it does not help with my larger issue of passing the ip traffic for the public IP based on each IP to a specific VM

 

[Alwin Antreich]

I have 5 public IP's that I need to assign to certain VM's

Either use NAT or (if you have a subnet) route the public IPs. That's much like routing private IP networks.

when you route from internet to those IPs I have it dies at the DDWRT and cannot go further.

I am not certain what you mean by 'dies' but the dd-wrt will be the gateway for those guests.

my goal is to get rid of the ddwrt and replace with 2 node failover router setup of some sort..

Do you have more than one internet connection? Basically, when the modem dies, than the failover will not be of much use (if that's the sole purpose). A less complex solution would be to have a cold standby server, that is easily replaced.