Recent content by ikogan

Admittedly, Claude found it. It was relevant to me because in my original version, it pointed to a bunch of OSDs that had directories but no services, and all of these were ones that the migration script was complaining about and weren't on any of my hosts. I had that function giving me the same...

So, apologies, I was unclear. This _does not_ fix my 500 error where the web UI tells me some OSDs don't exist on my hosts (when they do). This only cleared up the migration scripts errors. What's the output of that command on your hosts? One of mine gives me this: $VAR1 = { 'perun'...

Hi everyone, it looks like I'm having a different exciting issue than everyone else: root@perun:/usr/share/pve-manager/migrations# /usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-cluster-keys --verbose INFO: This is a dry run. No Ceph key, cipher setting, or daemon...

This past weekend I completed migrating my config management to Ansible and updated all PVE packages, rebooting each node as I did so. Afterwards, 4 of my 5 nodes started throwing errors that some Ceph client(s) using the client.admin client are unable to authenticate...

Is there any reason why this couldn't be placed in a docker image? Additionally, how do y'all feel about releasing an official image?

No, I'm just taking the performance hit for now.

According to the ceph documentation, at least as of Reef, the mgrs no longer export perf counters by default (https://docs.ceph.com/en/reef/mgr/prometheus/#id1) which I thought wouldn't be a big deal for me. However, some of these counters include OSD storage information, in particular...

So the node got fenced again yesterday and now I'm not seeing errors anymore. There doesn't seem to be anything more in `daemon.log`. It includes the usual flapping followed by what looks like startup logs. Monitoring corosync stats might be a good idea. I'll try and get that worked in there to...

I've attached the cmap stats for the other nodes here. Node 1 is the node that fenced Node 5. The metrics up there are from Node 5.

Lucky for me, I have that info. From one of the other nodes: 2023-03-31T12:50:05-04:00 service 'vm:112': state changed from 'fence' to 'recovery' 2023-03-31T12:50:05-04:00 service 'vm:109': state changed from 'fence' to 'recovery' 2023-03-31T12:50:05-04:00 node 'zorya': state changed...

There are 0 errors on all nodes... There are retries across all nodes but no node is an outlier. In the logs, it always seems to be link 1 that's failing, never link 0. Looking at the logs some more, every node claims that "host 5 joined", host 5 claims every other node joined. Since this node...

So this cluster has: 1. Intel Core i7-9700k w/Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) 2. Intel Xeon E3-1240L v5 w/Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) 3. Intel Xeon D-1521 w/Intel Corporation 82599ES 10-Gigabit SFI/SFP+...

The interfaces I have for my corosync rings are not bonded. This just started happening again today and I'm not sure what changed. I don't see any errors so far on the links but I'm seeing constant flapping. If this is related to mac address aging, why would it have such a weird pattern...

I'm having some performance issues on CephFS I'm trying to track down. I tried enabling stats following the information on https://docs.ceph.com/en/quincy/cephfs/cephfs-top/. The second I do ceph mgr module enable stats, all of my MDSes start to puke the following: 2023-02-19T21:19:31.164-0500...

Sorry, I've been on a trip for the past week or so. Anyway, thanks for the tip, I've disabled RSTP and LLDP on ports connected to the cluster nodes. We'll see if that helps.