Search results

Just adding to the voices that I have observed this behavior as well. proxmox-ve: 7.1-1 (running kernel: 5.13.19-5-pve) pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe) pve-kernel-helper: 7.1-12 pve-kernel-5.13: 7.1-8 pve-kernel-5.11: 7.0-10 pve-kernel-5.3: 6.1-6 pve-kernel-5.13.19-5-pve...

Verified network infrastructure issue. RSTP was indeed forcing corosync and other traffic along a route it had no business on. Adjusted RSTP values and have not had a node crash since.

My nodes are current. Further investigation suggests potential network infrastructure issue. Will post results. EDIT: Bad RSTP priority was bad, lowest performant switch in the stack became root. Switch stats show peaks of usage through the uplink and downlink ports soon before fencing events...

Not entirely sure if this is saying much of use: -- Journal begins at Thu 2021-07-08 11:26:31 PDT, ends at Fri 2021-10-08 08:05:08 PDT.> Oct 08 01:35:56 pvenode1 systemd[1]: Starting The Proxmox VE cluster filesystem... Oct 08 01:35:56 pvenode1 pmxcfs[6572]: [quorum] crit: quorum_initialize...

Further investigation suggests a corosync congestion issue with the current layout. Is there a good place to look for logging regarding corosync errors?

Every node running VMs suddenly reboots during our nightly backup task to a Proxmox Backup Server installation. Package Versions: proxmox-ve: 7.0-2 (running kernel: 5.11.22-5-pve) pve-manager: 7.0-11 (running version: 7.0-11/63d82f4e) pve-kernel-helper: 7.1-2 pve-kernel-5.11: 7.0-8...

@adriano_da_silva From my experiences with Ceph, I prefer it for the rebalancing and reliability it has offered. I have put a lab ceph setup through hell with a mix of various drives, host capabilities, and even uneven node networking capabilities. The only thing it did not handle well (and...

I want to bring up that Ceph cluster again. Are you expecting there to be OSDs on the nodes you power off to save power? If so Ceph will have to rebalance every time you turn servers off/on, and that will have considerable wear on your storage medium, slow down your storage pools, and cause...

This sounds like a fundamental misunderstanding of Ceph, which is not a Proxmox product.

Well regardless, I suggest starting your diagnostics with vda1 since it is having I/O errors and buffer issues. It is likely what is causing both your storage issues and the massive amount of IO Wait I can see in the CPU graph. What is vda1?

You've got at least one bad disk there @frox . Time to do some testing and/or replacement.

What is this even running on?

Upgraded a 4 node production cluster running Ceph with no downtime or issues. I did do the needful and ran the check script, ensured Prox 6.4 was fully updated, and reviewed my systems for any of the known issues/extra steps. For example, 3/4 nodes in this cluster run old boards and therefore...

FYI, this is resolved in Proxmox 7, at least as of pve-manager: 7.0-9 .

And where is the storage hosted?

Are you sure the heat isn't coming from somewhere else in the system and just being vented by the power supply? Otherwise, have any bulging capacitors on that motherboard?

Depending on the age of the system and the rarity of that hardware at this point, it very well may be that the drivers are not included in the Proxmox installer anymore. You may have to look into drivers for that system and load them into the installer. You could also try putting the controller...

Also in case you do not get the indication well enough from the docs, I strongly suggest you do hardware testing on the drive(s) which held the malfunctioning pg(s).

You may need to confirm what hardware you have, and if the storage controller is still functional on an older unit like that. Doing some quick searching I am unable to find an "H433" controller, but based on my testlabs I can say that other storage controllers of the same generation as that...

@akus I'm glad the SSD cache is helping with that drive. Definitely keep a backup of what's stored on that array though, since I have seen those SMR drives cause issues with cached setups before, notably on my home machine when I tried to make some use of the extra SMR disks I had laying around.