Hello,
to gains some extra performance for a (non hyperconverged nautilus) ceph-storage on spinners we configured several proxmox VMs "years" ago to use a stripe across 4 or 6 vm- (rbd ceph) disks. The VMs disks are used as pv's (LVM) and each logical volume is created as stripe across the existing pv's. This system worked fine until recently for this cluster call it "A". The system version where we are using on this pve cluster is: pve-manager/7.3-6/723bb6ec (running kernel: 5.15.85-1-pve)
Recently we had to shut down all VMs and the pve hosts very quickly (air conditioning problem) . Later on we could restart all the pve hosts and VMs and we noticed that exactly those VMs had a problem that were using the stripe-solution described. Other VMs using a single disk were just running fine. All are on the same ceph storage. The problem of the VMs using a stripe (used only for data filesystems, not for the ubuntu installation) seemed to work right after starting, but the VM quickly gathered a high load. Testing with dd reading some blocks from the underlying VM-disks showed that some disk that are part of the stripe delivered data without problems others hung after reading some blocks. A symptom we never saw before. We even did another complete reboot of the whole pve system but the disk hanging-symptom remained.
So I manually mapped those hanging vm disks (rbd) on my desktop and ran dd to read data. This worked without any problems. The same disks were afterwards still unresponsive in constantly delivering data when read inside the VM. We did not find a solution for this problem.
What we finally did was to add the ceph storage used on "A" to another pve cluster "B" (pve-manager/7.3-4/d69b70d4 (running kernel: 5.15.83-1-pve) and manually migrated the VMs config to this cluster by moving /etc/pve/qemu-server/<num>.conf to the other cluster. This worked the VM could be started on the new pve "B" cluster, but the problems remained. The last way out was to do a storage migration of the VM disks from the original storage on "A" to native storage of the new cluster which worked without any "read"-problems and after the migration finished the VM worked again without any problem.
Since we did not find the reason for this problem we would of course be grateful for ideas what might have happened?
Thanks
Rainer
to gains some extra performance for a (non hyperconverged nautilus) ceph-storage on spinners we configured several proxmox VMs "years" ago to use a stripe across 4 or 6 vm- (rbd ceph) disks. The VMs disks are used as pv's (LVM) and each logical volume is created as stripe across the existing pv's. This system worked fine until recently for this cluster call it "A". The system version where we are using on this pve cluster is: pve-manager/7.3-6/723bb6ec (running kernel: 5.15.85-1-pve)
Recently we had to shut down all VMs and the pve hosts very quickly (air conditioning problem) . Later on we could restart all the pve hosts and VMs and we noticed that exactly those VMs had a problem that were using the stripe-solution described. Other VMs using a single disk were just running fine. All are on the same ceph storage. The problem of the VMs using a stripe (used only for data filesystems, not for the ubuntu installation) seemed to work right after starting, but the VM quickly gathered a high load. Testing with dd reading some blocks from the underlying VM-disks showed that some disk that are part of the stripe delivered data without problems others hung after reading some blocks. A symptom we never saw before. We even did another complete reboot of the whole pve system but the disk hanging-symptom remained.
So I manually mapped those hanging vm disks (rbd) on my desktop and ran dd to read data. This worked without any problems. The same disks were afterwards still unresponsive in constantly delivering data when read inside the VM. We did not find a solution for this problem.
What we finally did was to add the ceph storage used on "A" to another pve cluster "B" (pve-manager/7.3-4/d69b70d4 (running kernel: 5.15.83-1-pve) and manually migrated the VMs config to this cluster by moving /etc/pve/qemu-server/<num>.conf to the other cluster. This worked the VM could be started on the new pve "B" cluster, but the problems remained. The last way out was to do a storage migration of the VM disks from the original storage on "A" to native storage of the new cluster which worked without any "read"-problems and after the migration finished the VM worked again without any problem.
Since we did not find the reason for this problem we would of course be grateful for ideas what might have happened?
Thanks
Rainer
Last edited: