Hello, this is not really a question, but I write down my expirience solving this awful problem because I have spended some tens of hours on it and it may help someone in the future.
I have used no-subscription version for my home purpose on S5000 daul processor server board. But very old and month ago, it fails permanently. Iam slowly leaving proxmox and full virtualization at all because technology move forward and my home needs are fully satisfied by QNAP NAS (4core Arm CPU, 8GB RAM, container avare) and some stand-alone ARM or low power Atom/Celeron borads with LXC if more CPU power is ever needed. No more 400W old rig running 24/7.
So I needed to get all data from old server. And at startup, grub ends with "grub rescure" prompt and "unknow filesystem". In addition, I have found mainboard almost dead (no USB functional for boot rescue OS, IDE CD not booting) and possibly caused some later disk error problems.
I temporary changed main board with some old desktop with enought SATA ports and old 2core CPU to make it moving boot Debian live, install zfsutils-linux...
Long story short:
Grub can't recognise ZFS filesystem in degraded state. It is not possible to grub-install or grub-probe to any drive with root file system in this state. We are talking about "source" filesystem, no metter what device are you trying to write GRUB. USB device just for bootloader? Just no.
The other question was, why my ZFS is in degraded state when I replace drive and resilver finished (with some permanent Errors). I still saw old drive in zpool status named as "OLD". Again, after some hours searching I have found, that devices can be displayed as GUIDS instead of name of the device (sdX) with command "zpool status -g". And also safely detached this way. After that, my ZFS pool was online, grub can be installed to any drive and my server is very slow, but back online.
I hope this hints can save someone dreamless nights about their data. Happy new year and thank guys from Proxmox for the great product. Keep going.
J.R.
I have used no-subscription version for my home purpose on S5000 daul processor server board. But very old and month ago, it fails permanently. Iam slowly leaving proxmox and full virtualization at all because technology move forward and my home needs are fully satisfied by QNAP NAS (4core Arm CPU, 8GB RAM, container avare) and some stand-alone ARM or low power Atom/Celeron borads with LXC if more CPU power is ever needed. No more 400W old rig running 24/7.
So I needed to get all data from old server. And at startup, grub ends with "grub rescure" prompt and "unknow filesystem". In addition, I have found mainboard almost dead (no USB functional for boot rescue OS, IDE CD not booting) and possibly caused some later disk error problems.
I temporary changed main board with some old desktop with enought SATA ports and old 2core CPU to make it moving boot Debian live, install zfsutils-linux...
Long story short:
Grub can't recognise ZFS filesystem in degraded state. It is not possible to grub-install or grub-probe to any drive with root file system in this state. We are talking about "source" filesystem, no metter what device are you trying to write GRUB. USB device just for bootloader? Just no.
The other question was, why my ZFS is in degraded state when I replace drive and resilver finished (with some permanent Errors). I still saw old drive in zpool status named as "OLD". Again, after some hours searching I have found, that devices can be displayed as GUIDS instead of name of the device (sdX) with command "zpool status -g". And also safely detached this way. After that, my ZFS pool was online, grub can be installed to any drive and my server is very slow, but back online.
I hope this hints can save someone dreamless nights about their data. Happy new year and thank guys from Proxmox for the great product. Keep going.
J.R.