great to hear that ... thanks for that informationDone this yesterday, issue is gone
the support told us yesterday the same
came in german, but here:
Wir haben nach viele Versuchen (z.B. downgrading von pve-qemu-kvm) via rbd journaling auf dem Failover Cluster geschwenkt.
Auf welche Version haben Sie ein Downgrade durchgeführt? Haben Sie nach dem Downgrade die Live-Migration der VMs angehalten und gestartet? Eins von diese Schritte sind notwendig, damit die VMs die neue Qemu-Version laden können.
Tritt das Problem bei VM-Backups auf oder nur allgemein? Obwohl die Laufwerke nicht so groß sind, mein erster Eindruck ist, dass es sich nach dem Problem anhört, über das in unserem Forum berichtet wurde [1] und das kürzlich durch Rückgängigmachen eines Commits behoben wurde [2]. Die korrigierte Version (pve-qemu-kvm=6.2.0-8) befindet sich in unserem pvetest [3] Repository.
Wenn Sie dieses Paket ausprobieren möchten, um zu sehen, ob es hilft
hi ... my update
we tried today to cleanup our second cluster, after we moved everything to our rbd mirror
we stopped some vms to make a final backup. some of these vms stuck with timeout.
after that, we installed the devel pve-qemu-kvm=6.2.0-8, but this does not help.
at the end, after many tear drops, i had an idea.
- stop stuck vm
- rbd copy pool/image pool/image_neu
- change scsi device name in vm config
- tada ... vm up and running ... sleeping will be possible this night
- now we stopped the vm and moved the image to the correct name
so, for the complete disaster night nightmare ... copy rbd image will help, but also only as a workaround
what do we found there ... unbelievable
next we try to reboot alle the vms ... but not this night
after that, we have to repair the rbd mirror settings and bring our cluster 1 back to party
we tried today to cleanup our second cluster, after we moved everything to our rbd mirror
we stopped some vms to make a final backup. some of these vms stuck with timeout.
after that, we installed the devel pve-qemu-kvm=6.2.0-8, but this does not help.
at the end, after many tear drops, i had an idea.
- stop stuck vm
- rbd copy pool/image pool/image_neu
- change scsi device name in vm config
- tada ... vm up and running ... sleeping will be possible this night
- now we stopped the vm and moved the image to the correct name
so, for the complete disaster night nightmare ... copy rbd image will help, but also only as a workaround
what do we found there ... unbelievable
next we try to reboot alle the vms ... but not this night
after that, we have to repair the rbd mirror settings and bring our cluster 1 back to party
one update again ... makes the parts more strange
maybe, since we enabled journal based replication, we never stopped a vm
so, we cleanup the pool config and removed all the journaling features ...
what should i say ... our vms are booting
i think, we run in a another problem.
but, we also had problems with snapshot replication an peformance after updates ... will tell you later.
thanks to everybody
maybe, since we enabled journal based replication, we never stopped a vm
so, we cleanup the pool config and removed all the journaling features ...
what should i say ... our vms are booting
i think, we run in a another problem.
but, we also had problems with snapshot replication an peformance after updates ... will tell you later.
thanks to everybody
Fabian_E said:
The package pve-qemu-kvm=6.2.0-8 which reverts the problematic change for QEMU's RBD driver, is now available on the pvetest repository.
You can test it by:
Interestingly the hosts that powered off weren't actually included in the backup but all make use of RBD storage for data storage (video archive)
Any further ideas from the community on what i can try to solve this issue once and for all?
I saw mentioned in an earlier post someone referring to disabling krdb flag in the disk config, is that correct?
Thanks in advance.
The package pve-qemu-kvm=6.2.0-8 which reverts the problematic change for QEMU's RBD driver, is now available on the pvetest repository.
You can test it by:
- Adding/Enabling the pvetest repository (can also be done in the Repostiories panel in the UI).
- apt update && apt install pve-qemu-kvm
- Removing/disabling the pvetest repository again.
- VMs need to be stopped/started or migrated to pick up the new version.
After deploying this package across my cluster the issue is much improved, but i still had 3 workloads shutdown (compared to 10 plus before updating kvm pkg) over night which powered off during a backup process.Done this yesterday, issue is gone
Interestingly the hosts that powered off weren't actually included in the backup but all make use of RBD storage for data storage (video archive)
Any further ideas from the community on what i can try to solve this issue once and for all?
I saw mentioned in an earlier post someone referring to disabling krdb flag in the disk config, is that correct?
Thanks in advance.
Please try to pin Kernel Linux 5.13.19-6-pve for now. Has helped with several current raising problems with 7.2-4 the last days.
Hey itNGO
Thanks for your input -
My issue is that i already "cleverly" purged all the 5.13.* kernels from my cluster - so there will be some work in getting the 5.13* kernel downloaded and redeployed to entire cluster.
Was hoping for a quicker fix.
Thanks for your input -
My issue is that i already "cleverly" purged all the 5.13.* kernels from my cluster - so there will be some work in getting the 5.13* kernel downloaded and redeployed to entire cluster.
Was hoping for a quicker fix.
That sounds a bit odd.. Anything in the kernel log /
journalctl, could even be an OOM kill (which then would be rather unrelated to any such bugs and a sign of memory over commitment, if it really was a OOM kill)Enabling KRBD, not disabling. What can also help is enabling IO-Thread (and the
VirtIO SCSI single controller, if you're using SCSI for the VM disks), also using the default No Cache cache mode for disks, as others, like e.g. writeback, while being sometimes seemingly faster, can cause much more erratic IO and sensitivity for host memory pressure for the guest.
Hi all,
I bit the bullet and rolled back the
This has now resolved this issue.
Last edited:
We completed a PVE and Ceph upgrade from 6.4 to 7.2 Friday night. A 9.5TB NVR VM backup started shortly thereafter. We noticed the operation of the VM ground to a halt and the backup performance was fairly slow (71Mbps read, 52Mbps write).
I applied the update to pve-qemu-kvm as described above and restarted just that NVR VM. I also enabled the IO-Thread option for the VM drives.
After a restart of that VM, I started a new backup. VM operation is great even with the backup running. Also of note, the backup performance is substantially better (better than when on 6.4 as well). Backup is now running at ~225 Mbps read, 212Mbps write!
Is it a concern that I only restarted this one VM to use the newer version of pve-qemu-kvm? Restarting all other VMs is a big project which requires a lot of coordination.
I applied the update to pve-qemu-kvm as described above and restarted just that NVR VM. I also enabled the IO-Thread option for the VM drives.
After a restart of that VM, I started a new backup. VM operation is great even with the backup running. Also of note, the backup performance is substantially better (better than when on 6.4 as well). Backup is now running at ~225 Mbps read, 212Mbps write!
Is it a concern that I only restarted this one VM to use the newer version of pve-qemu-kvm? Restarting all other VMs is a big project which requires a lot of coordination.
Thanks for your feedback.
Is this a cluster? As then you could live-migrate the VMs to already updated PVE hosts, that way they also get started with the new PVE-QEMU version without any interruption in the guest itself.