Search results

This error keeps popping in my Proxmox GUI and everytime I have to manually scrub pgs with "ceph pg deep-scrub 7.da" , why is this happening and how can I prevent/fix it. pveversion --verbose proxmox-ve: 6.1-2 (running kernel: 5.3.18-3-pve) pve-manager: 6.1-8 (running version: 6.1-8/806edfe1)...

So I found the culprit, it was stack links which was having lots of packet drops which caused unstable cluster. Thank you for all you help @fabian . Really appreciate it. Is there any timer I can change to increase for node to wait for quorum before rebooting ?

But this was all a stable setup before reboot mess. :(

Its a separate network and separate vlan but same interfaces.

And the weirdness continues and nodes have started crashing again root@sm1:~# tail -f /var/log/messages Apr 20 03:15:51 sm1 kernel: [41625.332130] device tap536i1 entered promiscuous mode Apr 20 03:15:51 sm1 kernel: [41625.342800] vmbr1: port 3(tap536i1) entered blocking state Apr 20 03:15:51...

That all went fine though but....none of my VMs are booting up now, they are stuck just starting up for 15 minutes now, Is there any chance my all VMs are gone with data during these reboot cycles ? If so any way to recover it. ~# ceph status cluster: id...

That did it, Thank you @fabian . From where can I begin to troubleshoot it, any pointers on what should I be looking at to fix in order for such issue to not happen again like changing value of wait seconds before rebooting in case nodes don't all come up at the same time.

This is what I see on HA settings.

Thanks @fabian , The commands you mentioned helped recover the cluster, I can see nodes green now # pvecm status Cluster information ------------------- Name: proxmox-cluster Config Version: 22 Transport: knet Secure auth: on Quorum information ------------------...

root@dellr730-1:~# pveversion --verbose proxmox-ve: 6.1-2 (running kernel: 5.3.18-3-pve) pve-manager: 6.1-8 (running version: 6.1-8/806edfe1) pve-kernel-helper: 6.1-8 pve-kernel-5.3: 6.1-6 pve-kernel-5.3.18-3-pve: 5.3.18-3 pve-kernel-5.3.18-2-pve: 5.3.18-2 pve-kernel-5.3.10-1-pve...

I happen to reboot my 11 node cluster yesterday all together, big mistake ! It took almost 24 hours to come back up and get out of reboot loop. Now all nodes are up but no cluster or quorum. Nodes can ping each other fine but still no cluster. Logs/Debugs attached. root@dellr730-1:~#...

Did you ever resolve it ? Just bumped into same issue recently even with latest proxmox 6.1

Thank you @Alwin . That makes sense.

Hello Experts, I have recently setup new proxmox 6 environment with ceph, I removed 3 osds from one of the node and after that ceph has stopped recovering even though its shows there is Degraded Data Redundancy, I can't figure out why it wouldn't recover these. ~# pveversion...

Same boat !

Thank you for all your assistance and time.

nevermind it failed with its original intel firmware, and with default include firmware it kept on flapping, I think I will be replacing this card, I don't think there is any other option. [ 46.733756] Intel(R) 10GbE PCI Express Linux Network Driver - version 5.6.5 [ 46.733759] Copyright(c)...

Ok the ports came up, but it not consistent across reboot, how can make it consistent or run theses commands everytime after boot automatically root@voip:~# modprobe ixgbe root@voip:~# echo 8086 0001 > /sys/bus/pci/drivers/ixgbe/new_id root@voip:~# ip link set enp132s0f1 up root@voip:~# ip link...

The problem is consistent on all 4 servers having same NIC card, likelihood of all NIC cards broken is less, I did try installing the firmware didn't help.

Same thing in pve-kernel-5.4 after bootup, when I issue modprobe ixgbe only these new entries appear in dmesg [ 81.471183] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 5.1.0-k [ 81.471184] ixgbe: Copyright (c) 1999-2016 Intel Corporation. However If I manually bind it...