You have to ask Veeam what strange things they do with their backup. I would assume they create a simple diff qemu file in which all writes are redirected, so that the actual disk itself does not change so that they can read consistently and...
Hi, there is already a patch for this on the mailing list [0], I updated the referenced issue accordingly.
Note that the issue itself is not resolved by this just yet, please see the explanation in the bug report.
[0]...
Hi Jonas
What types of drives are connected to your controller?
We use original HP drives, such as the 3.20TB 32.0G NVMe SSD.
Can these failures be reproduced in any way, or do they occur randomly?
The problem occurs almost every time on the...
I seem to have a potentially related regression on a different platform, with the same kernel boundary.
Hardware:
QDNV01 appliance/mainboard
Intel Atom C3558 (Denverton), 4 cores
AMI BIOS 5.13, dated 2024-02-21
UEFI boot
Proxmox VE 9.2...
You have to ask Veeam what strange things they do with their backup. I would assume they create a simple diff qemu file in which all writes are redirected, so that the actual disk itself does not change so that they can read consistently and...
Nothing there, just the boot message from the bnxt_en module
Aug 09 22:10:53 pve002-ewr kernel: bnxt_en 0000:4b:00.0 (unnamed net_device) (uninitialized): Device requests max timeout of 100 seconds, may trigger hung task watchdog (kernel default...
Adding to @SteveITS bandwith is indeed important, if you can get to 25 or even 100gigs between sites..
You have to keep in mind that say, in RF4, each written block gets written to 4 places for safecopying, that's a lot
What is the speed and latency between the two locations? Hopefully 10 Gbit or higher? I could see this being very slow in several cases since at least 1/3 of all I/O is copied to the other data center.
4 nodes is more than 3 but you may want...
I agree with the others that this is a setup with many caveats and has high disk consumption (due to the requirement of having rf4)
Also this sounds like Corosync will be over shared media , so make sure your QoS is tight accordingly
Hey @abhijithiaaxin, welcome to the forum.
Whether it is a single-DC deployment or a split-DC deployment, you generally want an odd number of nodes in your cluster. But, as others have pointed out, a split-DC installation requires three...
Thanks @Patrik_V for providing the outputs!
There were similar issues with the megaraid_sas driver on kernel 6.17 but they were reported to be fixed for most users with kernel 7
What kind of drives are connected to your controller?
Do you have...
Root cause identified, with confirmation from the storage provider.
The provider (Curanet) uses Dell ECS / ObjectScale as the S3 backend. Their
systems architect confirmed the symptom matches AWS's default-checksum change:
newer AWS SDKs send...
It sounds like you might have lost monitor quorum. When ceph -s hangs even though your monitors seem to be running, it usually means the client is stuck waiting for a monmap it can't get.
You can bypass the cluster entirely and query the monitor...
yeah aside from loading a newer kernel in the guest for the proper driver, i don't see anything wrongly configured on your side
btw. the speed reporting issue seems to be a cosmetic issue only...
Looks like your reboot removed one more votes at the same time the cluster was already shedding nodes.
Next thing I'd check is the host-side cause of those stalls:
journalctl -k --since "2026-08-09 21:30" --until "2026-08-09 22:15" | grep -iE...
Can you try switching from io_uring to native and test it?
Maybe, but only maybe, it is related to io_uring, or it hides the problem when native is used.
This is related to...
I'm not sure what I'm looking for - nothing seems to be out whack, most of the stuff is related to the outage itself, and maintenance of hosts (updates / reboots etc)
It did get very weird during the outage, in that we saw about half the nodes...