Hello all,
I ran into a problem with guest migration when preparing an upgrade from 7.4 to 8.1.
Migration of guests failed, showing "TASK ERROR: migration aborted"
These were the two relevant lines in the log:
My setup is a smallish 2-3 nodes cluster for home use, running heterogenous hardware, and using btrfs-storage in raid1 mode an all nodes.
I had added a new host to the cluster, which was directly installed using pve-8.0 and updated to 8..
The plan was to move the guests to that newer machine before upgrading the older hosts to pve-8.
Workaround:
It turned out that using snapshots on the guests actually caused the problem:
A useable workaround is, to remove all snapshots and then the migration ran successfully.
This looses all snapshots to return to in case of problems. For my home setup this was acceptable, because i have backups (well, just the important guests...) but is really not something you want in a production environment. A possibility to keep all snapshots would be highly appreciated.
Potential cause:
I have found out, that this is actually related to the Bug in Debian that is apparently related to a feature introduced in btrfs-progs.
If you send a btrfs subvolume which is marked ro and receive it as rw, the btrfs-progs throw an error to protect you from accidentally using that received subvolume as read-write. And Snapshots are apparently marked ro, which really makes sense.
Since pve-8.x is based on the Debian bookworm, it apparently inherited that issue.
I did not try guest migration between the older pve-7.x hosts, it may show the same with recent btrfs-progs, but maybe the kernel is too old. YMMV.
This is potentially something to add to the list of Known upgrade issues in pve-8.x, when using btrfs storage.
Question:
Is there a way to include the force (-f) option for sending btrfs subvolumes, possibly when using commandline?
Keeping all snapshots even across upgrades is quite likely an inportant feature.
As a side note, i also ran into the problem that vm guest migration failed, due to a cdrom with an ISO on the local filesystem was configured as hardware. Even removing this from the hardware did not let me migrate this guest, because that cdrom is also within the snapshots. Removing all snapshots solved this.
Feature-Request (well, sort of):
Could there be a gui checkbox for such operations to force such things?
This would allow to mirgate the vm guest first, and then deal with fixing the hardware config later on the target host, even if the vm won't start direcly after migration?
Thank you,
Florian
I ran into a problem with guest migration when preparing an upgrade from 7.4 to 8.1.
Migration of guests failed, showing "TASK ERROR: migration aborted"
These were the two relevant lines in the log:
Code:
2024-01-03 23:04:47 btrfs error: ERROR: cannot flip ro->rw with received_uuid set, use force if you really want that
send/receive failed, cleaning up snapshot(s)..
2024-01-03 23:04:47 ERROR: storage migration for 'local-btrfs:110/vm-110-disk-0.raw' to storage 'local-btrfs' failed - command 'set -o pipefail && pvesm export local-btrfs:110/vm-110-disk-0.raw btrfs - -with-snapshots 1 -snapshot __migration__ | /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=pve3' root@192.168.1.33 -- pvesm import local-btrfs:110/vm-110-disk-0.raw btrfs - -with-snapshots 1 -snapshot __migration__ -delete-snapshot __migration__ -allow-rename 1' failed: exit code 2My setup is a smallish 2-3 nodes cluster for home use, running heterogenous hardware, and using btrfs-storage in raid1 mode an all nodes.
I had added a new host to the cluster, which was directly installed using pve-8.0 and updated to 8..
The plan was to move the guests to that newer machine before upgrading the older hosts to pve-8.
Workaround:
It turned out that using snapshots on the guests actually caused the problem:
A useable workaround is, to remove all snapshots and then the migration ran successfully.
This looses all snapshots to return to in case of problems. For my home setup this was acceptable, because i have backups (well, just the important guests...) but is really not something you want in a production environment. A possibility to keep all snapshots would be highly appreciated.
Potential cause:
I have found out, that this is actually related to the Bug in Debian that is apparently related to a feature introduced in btrfs-progs.
If you send a btrfs subvolume which is marked ro and receive it as rw, the btrfs-progs throw an error to protect you from accidentally using that received subvolume as read-write. And Snapshots are apparently marked ro, which really makes sense.
Since pve-8.x is based on the Debian bookworm, it apparently inherited that issue.
I did not try guest migration between the older pve-7.x hosts, it may show the same with recent btrfs-progs, but maybe the kernel is too old. YMMV.
This is potentially something to add to the list of Known upgrade issues in pve-8.x, when using btrfs storage.
Question:
Is there a way to include the force (-f) option for sending btrfs subvolumes, possibly when using commandline?
Keeping all snapshots even across upgrades is quite likely an inportant feature.
As a side note, i also ran into the problem that vm guest migration failed, due to a cdrom with an ISO on the local filesystem was configured as hardware. Even removing this from the hardware did not let me migrate this guest, because that cdrom is also within the snapshots. Removing all snapshots solved this.
Feature-Request (well, sort of):
Could there be a gui checkbox for such operations to force such things?
This would allow to mirgate the vm guest first, and then deal with fixing the hardware config later on the target host, even if the vm won't start direcly after migration?
Thank you,
Florian