ZFS no pools available yet ONLINE import status, I/O error on Import attempt

[colinstu]

I have a near 2 year old Proxmox install (installed w/latest version at the time and have been keeping it up to date since).
It was built/installed from the beginning with two NVMe SSDs in a ZFS Mirror / RAID1. (With Root on ZFS too). This was all setup right through proxmox and I haven't had any issues.
Nearly a week ago I decided to enable/setup MSISupported (in registry) for a Windows11 VM for an AMD GPU and audio that's been successfully using passthrough since near the beginning. I only gave this new setting a shot because I noticed an occasional audio dropout and the Internet seemed to suggest this would be a fix.
Things seem to be going fine, up until last night however.

I discovered that all my other VMs/CTs were no longer accessible from the internet or network, could no longer hit proxmox web UI.
So I check out the local console on the host, and discover a stack trace has been output to the screen and it's hardlocked. Would not respond to keyboard or even a press of the power button, so I held power until it powered off.

Upon booting back up, my zfs pool (named rpool) can no longer be imported by proxmox. It loads the zfs module, runs command: /sbin/zpool import -c /etc/zfs/zpool.cache -N 'rpool' and returns back right away with Cannot Import 'rpool' I/O error , Destroy and re-create the pool from a backup source. Says the same for cachefile too.
GREAT.
At this point I don't know if this is due to the prev/above change, or due to the (random?) lock (which may've been due to AMD GPU reset bug? I was not aware of this or its fix), OR due to the power down after that lock and being brought back up.

I'm dumped off at the initramfs prompt.
zpool list and info (with and without -v) both return 'no pools available'
'zpool import' returns: the name of my pool (rpool), and even says it's ONLINE, it lists the mirror, and both nvme devices (appearing as nvme-eui-....), again, all of them appear ONLINE.
I checked 'ls -l' under /dev/disk/by-id/ and verified that symlinks still exist for all the corresponding entries, everything seems to line up. (Note that in addition to my nvme-eui entries, there's also nvme-Samsung entries, both ones that end and don't end with _1. All of them have corresponding _part1, _part2, _part3 entries too. (three partitions are stock I believe, and the 3rd one being the main/biggest I believe).

Under a separate live CD (using ubuntu server 24.04 LTS and flipping from the install to a new TTY screen), I get all the exact same output for zpool list, status, and import as well. 'lsblk' still returns my nvme devices and their partitions. (though they are not mounted here. and I can't run lsblk from initramfs).
I've run 'smartctl' and verified that all the SMART output there appears fine/correct and it does. I've run 'nvme list' and that seems to return fine/correct stuff. 'nvme smart-log' again, nothing alarming being returned here either for its SMART output too. 'zbd -C -e -p /dev/nvme0n1 rpool' returns a configuration and all that appears to be in order as well (both children/disks listed, paths matching prev output, nothing else alarming etc).

I tried booting into FreeBSD 14.1 earlier but was running into unrelated startup issues (BSD stuff). I can share more but doesn't seem at all related to any of this.
Gparted (live CD) showed both devices just fine too.

I've updated the BIOS on mobo, updated all the settings to match/be correct from prior to update, no change.

I've removed the GPU card entirely and tested all this as well, no change.

I've removed all but one stick of RAM, no change. I've run memtest, no errors being found.

I've been chronicalling all/most of this over here so far / comments / help from other folks, for more pics/output please review these threads.
Initial call for help:
https://furry.engineer/@colinstu/113315894821471446
Next (this) morning after reviewing responses / current status:
https://furry.engineer/@colinstu/113317908970470143
more replies/comments follow in those threads.

I am at a loss at this point. What can I do, is there still any way I can repair or fix my existing zfs pool 'rpool'?

All of my VMs and LXCs are backed up to an 8TB WD SATA drive also present in the machine. My proxmox config/zfs itself however were not backed up.

1) If I can recover w/o reinstalling, this would be the most ideal. What can I try?
2) If I can't recover the pool itself, how can I backup or save all/most of any relevant proxmox config/data that's not included in the guests themselves (host data)? I very much appreciate the steps to do this, or linking to guides/steps elsewhere if they exist.
3) Once #2 is complete above, then I suppose I would feel safe reinstalling Proxmox at that point. Any specific steps on this being different vs a fresh install w/no intent on restorations? Just want to make 100% sure that I don't somehow blow away or mess up my backup drive(s). Eventually I should be able to restore all my VM/LXC backups?

Also is there anything I can check on the old pool to find out what caused this lockup in the first place? Anything I can check that caused this zfs pool issue in the first place?

I appreciate any/all help. Thank you.

 

[esi_y]

zbd -C -e -p /dev/nvme0n1 rpool returns a configuration and all that appears to be in order as well (both children/disks listed, paths matching prev output, nothing else alarming etc).

If I had not missed it, did you let it show you [1] : zdb -d -e -p /dev/nvme0n1 rpool

[1] https://openzfs.github.io/openzfs-docs/man/master/8/zdb.8.html

 

[colinstu]

If I had not missed it, did you let it show you [1] : zdb -d -e -p /dev/nvme0n1 rpool

[1] https://openzfs.github.io/openzfs-docs/man/master/8/zdb.8.html

I believe this is the output for that one I ran a few hours ago. (command was cut off... if this isn't it I'll gather that too)

Also including it for 'zdb -C -e -p'

(Also apologize for this pics, some sort of pikvm is definitely coming in my future).

 

[esi_y]

I believe this is the output for that one I ran a few hours ago. (command was cut off... if this isn't it I'll gather that too)

Also including it for 'zdb -C -e -p'

(Also apologize for this pics, some sort of pikvm is definitely coming in my future).

I see you have tried quite a lot, but I have not seen you attempted -o readonly=on for any of the imports, correct?

Obviously, all this now only makes sense on that e.g. live Ubuntu, so forget the zpool.cache, etc.

 

[esi_y]

(Also apologize for this pics, some sort of pikvm is definitely coming in my future).

Also, you don't need this, from any LIVE system, I suppose you have network (or can dump it on USB), so just do your:

command | tee output.txt

And you get to both see what it did and have it saved to a file you can paste in here.

 

[esi_y]

For future readers, please check posts below on mounting root under alternative mountpoint with -R. This is important when ZFS was used for root, obviously.

I just popped into your linked discussions, you already got some good pieces of advice, I am not sure what you ran and you did not from there.

I would completely disregard the attempt to get it working with initramfs if that pool is not normally importable on a separate system even.

As I have no idea in what way ZFS shipped with PVE might be different from Ubuntu stock, so what I would do is make myself tiny install of PVE somewhere aside, just the basic ext4. In my past experience, the Installer rescue was useless, but getting small extra install of pve helps, then boot into it. (BTW This is why I would prefer to e.g. have root NOT on ZFS even if you do use ZFS for everything else, you just don't have to do this step.)

So this is now better than that live Ubuntu in that your tooling is exactly as it should be.

Try import the rpool there. (This is why I suggested ext4 install for that rescue system install, you won't mix up things.)

You can go on increasingly aggressive (save some forensics approach) on importing that pool, but in the end you may be just happy to be able to mount it at all, even in some incosistent state, to copy out what you cannot get from backups (more on that below).

So when I look at your priorities:

1) If I can recover w/o reinstalling, this would be the most ideal. What can I try?

You are already past that (this is not initramfs issue), also reinstalling host (when you can just implant configs and VMs from backups) is fast. There's nothing valuable on the root that is not in fresh ISO install except configs.

2) If I can't recover the pool itself, how can I backup or save all/most of any relevant proxmox config/data that's not included in the guests themselves (host data)? I very much appreciate the steps to do this, or linking to guides/steps elsewhere if they exist.

I would try (all on the rescue system now) to go on with the importing rabbithole [1], so first and foremost:

zpool import -o readonly=on rpool

It will probably ask you to add -f by itself, if it thinks it's not been exported.

If you got lucky, you will have access to copy the configs out already with this.

If you are getting error 5, that's EIO [2], what would be nice checking is dmesg, but i think you already did and it was rubbish and it will be rubbish as long as you don't have kernel with debug symbols which Proxmox do not ship (and are not planning to). Without a debug kernel, it's really useless to try (for me at the least) to pinpoint what that EIO really is caused by.

So if no luck, at this point, since this was a mirror, I would be considering to go on with importing only one of the drives connected, in case something goes terribly wrong, you should have identical copy on the second. More cautious approach would be to dd [2] image of the drive away and work on that instead, but I think you do not care for that (WRT data loss), especially you have backups, and it's not practical.

I have not seen anything broken in that zdb -d -e rpool of yours, but not sure if it was complete (it would normally show inconsistencies you would be able to read out easily).

So assuming the readonly import alone was not helping, step it up (check the man pages from [1] what those switches do):

zpool import -F rpool

If no luck, you can press on (you might damage things, but this is why you are importing it degraded):

zpool import -FX rpool

Failing all that, my last resort would be [4]:

echo 0 > /sys/module/zfs/parameters/spa_load_verify_metadata
echo 0 > /sys/module/zfs/parameters/spa_load_verify_data
zpool import -FX rpool

Hopefully at any of the points above, you got lucky.

WHAT TO GET OUT:

Well, apart from files to make fresh install easier for you (e.g. /etc/network/interfaces), you want to get out the config (something you should have been backing up):

https://forum.proxmox.com/threads/backup-cluster-config-pmxcfs-etc-pve.154569/
https://forum.proxmox.com/threads/r...ffline-extract-configurations-etc-pve.155374/

3) Once #2 is complete above, then I suppose I would feel safe reinstalling Proxmox at that point. Any specific steps on this being different vs a fresh install w/no intent on restorations? Just want to make 100% sure that I don't somehow blow away or mess up my backup drive(s).

Install without the backup drives connected.

Eventually I should be able to restore all my VM/LXC backups?

Implant your config.db as per links above and happy days looking for having the drives set the same way.

Also is there anything I can check on the old pool to find out what caused this lockup in the first place? Anything I can check that caused this zfs pool issue in the first place?

Not to my knowledge, but as mentioned, you can keep a dd copy of that (if you have the capacity) for further forensics. This is a valuable anectodal experience.

I appreciate any/all help. Thank you.

So I kind of dumped it here on you, but gotta go, so hopefully some if it helps. Proceed carefully, read the linked docs before each step.



[1] https://openzfs.github.io/openzfs-docs/man/master/8/zpool-import.8.html
[2] https://www.kernel.org/doc/Documentation/i2c/fault-codes
[3] https://manpages.debian.org/bookworm/coreutils/dd.1.en.html
[4] https://openzfs.github.io/openzfs-docs/Performance and Tuning/Module Parameters.html#spa-load-verify-data

 

[colinstu]

Finally got around to trying this out... Installed proxmox on a spare SATA SSD, findings:

zpool import -o readonly=on rpool
zpool import -F rpool
zpool import -FX rpool
=all resulted in nothing happening (and with -f too)

That last one though, with disabling the metadata/data, it actually worked....
Well now it says it's DEGRADED.

NAME                                  SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
rpool                                1.81T  1.27T   558G        -         -    51%    69%  1.00x  DEGRADED  -
  mirror-0                           1.81T  1.27T   558G        -         -    51%  69.9%      -  DEGRADED
    nvme-eui.0025384b21404284-part3  1.82T      -      -        -         -      -      -      -  DEGRADED
    nvme-eui.0025384b21404272-part3  1.82T      -      -        -         -      -      -      -  DEGRADED
root@pm:~# zpool status -v
  pool: rpool
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 00:24:26 with 0 errors on Sun Oct 13 00:48:27 2024
config:

        NAME                                 STATE     READ WRITE CKSUM
        rpool                                DEGRADED     0     0     0
          mirror-0                           DEGRADED     0     0     0
            nvme-eui.0025384b21404284-part3  DEGRADED     0     0    29  too many errors
            nvme-eui.0025384b21404272-part3  DEGRADED     0     0    29  too many errors

errors: Permanent errors have been detected in the following files:

        rpool/data/vm-100-disk-2:<0x1>
        rpool/data/vm-105-disk-0:<0x1>
        rpool/data/vm-101-disk-2:<0x1>

oot@pm:~# df -hT
Filesystem                   Type      Size  Used Avail Use% Mounted on
udev                         devtmpfs   48G     0   48G   0% /dev
tmpfs                        tmpfs     9.5G  1.6M  9.5G   1% /run
rpool/ROOT/pve-1             zfs        70G  4.2G   66G   6% /
tmpfs                        tmpfs      48G   46M   48G   1% /dev/shm
tmpfs                        tmpfs     5.0M     0  5.0M   0% /run/lock
efivarfs                     efivarfs  192K   92K   96K  49% /sys/firmware/efi/efivars
/dev/sda2                    vfat     1022M   12M 1011M   2% /boot/efi
rpool                        zfs       501G  128K  501G   1% /rpool
rpool/ROOT                   zfs       501G  128K  501G   1% /rpool/ROOT
rpool/data                   zfs       501G  128K  501G   1% /rpool/data
rpool/data/subvol-103-disk-0 zfs        30G  933M   30G   4% /rpool/data/subvol-103-disk-0
rpool/data/subvol-200-disk-0 zfs       100G  1.6G   99G   2% /rpool/data/subvol-200-disk-0
rpool/data/subvol-203-disk-0 zfs       300G  3.8G  297G   2% /rpool/data/subvol-203-disk-0
rpool/data/subvol-206-disk-0 zfs       200G  1.7G  199G   1% /rpool/data/subvol-206-disk-0
rpool/data/subvol-302-disk-1 zfs       100G   13G   88G  13% /rpool/data/subvol-302-disk-1
rpool/data/subvol-204-disk-0 zfs        50G  2.4G   48G   5% /rpool/data/subvol-204-disk-0
rpool/data/subvol-301-disk-0 zfs       100G  229M  100G   1% /rpool/data/subvol-301-disk-0
rpool/data/subvol-108-disk-0 zfs        80G  3.1G   77G   4% /rpool/data/subvol-108-disk-0
rpool/data/subvol-303-disk-0 zfs       100G  822M  100G   1% /rpool/data/subvol-303-disk-0
rpool/data/subvol-205-disk-0 zfs        50G  960M   50G   2% /rpool/data/subvol-205-disk-0
rpool/data/subvol-202-disk-0 zfs       100G  2.2G   98G   3% /rpool/data/subvol-202-disk-0
rpool/data/subvol-305-disk-0 zfs        50G  5.9G   45G  12% /rpool/data/subvol-305-disk-0
rpool/data/subvol-304-disk-0 zfs        50G  789M   50G   2% /rpool/data/subvol-304-disk-0
rpool/data/subvol-306-disk-0 zfs        80G  497M   80G   1% /rpool/data/subvol-306-disk-0
/dev/fuse                    fuse      128M   16K  128M   1% /etc/pve

Wait.. did it just mount the zfs over my / again?
And I can't export because it's in use now. How can I get my SATA SSD mounted back as root again? Or export the zpool again to allow import back into my old Root on ZFS / boot from nvme SSD setup directly again in order to grab some proxmox configs from my prior environment?

How can I clear the degraded state on this pool? Can I delete the files it lists as having errors to clear these errors?
How do I get the imported pool to appear in the proxmox UI again? (well it shows as degraded under Disks > ZFS, but how to get the files visible again under Datacenter > Storage? If you select ZFS from the list, can't select "VZDump backup" which would be needed to restore the VMs. Potentially select Directory instead?

It's questionable the state of any of this ZFS data anyway. Suggestions as well / steps on how to reimport my SATA 8TB Mechanical drive again into Proxmox so that I can use the Proxmox web UI to restore backed up CTs/VMs I had?
I had it unplugged during all of this work, can get that reconnected.

 

[esi_y]

Oh, sorry, the pitfalls of ZFS (on root), you can tell I am not a fan.

The unfortunate mountpoint has a simple solution, -R option [1] for import.

You may need to unmount with -f [2]

On the rest of your questions, I want to make one thing clear - your pool is messed up, you will be copying out (possibly) bad data.

So do not expect to get it "fixed", copy out what you need and start over - was at least my advice.

As you can tell all the ZFS aficionados in this forum stayed silent (I linked your thread to some of them), as was staff.

EDIT:

NAME                                  SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
rpool                                1.81T  1.27T   558G        -         -    51%    69%  1.00x  DEGRADED  -
  mirror-0                           1.81T  1.27T   558G        -         -    51%  69.9%      -  DEGRADED
    nvme-eui.0025384b21404284-part3  1.82T      -      -        -         -      -      -      -  DEGRADED
    nvme-eui.0025384b21404272-part3  1.82T      -      -        -         -      -      -      -  DEGRADED

Also I kind of hoped you would be doing this with only one of them (see my advice in the original post).

Suggestions as well / steps on how to reimport my SATA 8TB Mechanical drive again into Proxmox so that I can use the Proxmox web UI to restore backed up CTs/VMs I had?

I am not sure what this one is about, there was no mention of this (or I missed it)?



[1] https://openzfs.github.io/openzfs-docs/man/master/8/zpool-import.8.html#R
[2] https://openzfs.github.io/openzfs-docs/man/master/8/zfs-unmount.8.html#f~2

 

[esi_y]

All of my VMs and LXCs are backed up to an 8TB WD SATA drive also present in the machine. My proxmox config/zfs itself however were not backed up.

So I found this. If you retrieve your original config, you could have this taken care of on new install without additional work.

WHAT TO GET OUT:

Well, apart from files to make fresh install easier for you (e.g. /etc/network/interfaces), you want to get out the config (something you should have been backing up):

https://forum.proxmox.com/threads/backup-cluster-config-pmxcfs-etc-pve.154569/
https://forum.proxmox.com/threads/r...ffline-extract-configurations-etc-pve.155374/

 

[colinstu]

Hello esi_y,

I have updates and good news... I'll share more once I make sure this stuff can come back up, we're almost there, thank you for the help so far.

I'm trying to restore one of the VMs right now, I don't want to blow away the existing nvme drives (which had ZFS on them) right now to recreate the old zfs storage, I want to just restore one of the VMs to the new ext4 boot drive just to make sure the VM can even start back up again.

How do I point the restore to new storage?

If I try to restore as-is, it can't get to any of the old zfs stuff (no shit, it's not available yet).
If I make a new HD for it on the new storage, as soon as I kick off the restore again it blows away that config I just made and back at square one above.
When restoring there's also a dropdown to change the storage: great, so I select desired storage yet during the restore, it seems to still ignore what was selected and still stuck at 100% after it tries to decompress the backup... what am I missing here?

Header
Proxmox
Virtual Environment 8.2.7
Virtual Machine 100 (VM 100) on node 'pm'
No Tags
Filter VMID
Logs
()
restore vma archive: zstd -q -d -c /mnt/pve/wd8tb/dump/vzdump-qemu-100-2024_10_15-02_00_01.vma.zst | vma extract -v -r /var/tmp/vzdumptmp42118.fifo - /var/tmp/vzdumptmp42118
CFG: size: 838 name: qemu-server.conf
DEV: dev_id=1 size: 4194304 devname: drive-tpmstate0-backup
DEV: dev_id=2 size: 107374182400 devname: drive-virtio0
CTIME: Tue Oct 15 02:00:01 2024
Formatting '/mnt/pve/wd8tb/images/100/vm-100-disk-0.raw', fmt=raw size=4194304 preallocation=off
new volume ID is 'wd8tb:100/vm-100-disk-0.raw'
Formatting '/mnt/pve/wd8tb/images/100/vm-100-disk-1.raw', fmt=raw size=107374182400 preallocation=off
new volume ID is 'wd8tb:100/vm-100-disk-1.raw'
map 'drive-tpmstate0-backup' to '/mnt/pve/wd8tb/images/100/vm-100-disk-0.raw' (write zeros = 0)
map 'drive-virtio0' to '/mnt/pve/wd8tb/images/100/vm-100-disk-1.raw' (write zeros = 0)
progress 1% (read 1073807360 bytes, duration 0 sec)
progress 2% (read 2147614720 bytes, duration 0 sec)
progress 3% (read 3221356544 bytes, duration 0 sec)
progress 4% (read 4295163904 bytes, duration 1 sec)
progress 5% (read 5368971264 bytes, duration 1 sec)
progress 6% (read 6442713088 bytes, duration 1 sec)
progress 7% (read 7516520448 bytes, duration 1 sec)
progress 8% (read 8590327808 bytes, duration 1 sec)
progress 9% (read 9664069632 bytes, duration 1 sec)
progress 10% (read 10737876992 bytes, duration 1 sec)
progress 11% (read 11811684352 bytes, duration 1 sec)
progress 12% (read 12885426176 bytes, duration 1 sec)
progress 13% (read 13959233536 bytes, duration 1 sec)
progress 14% (read 15032975360 bytes, duration 2 sec)
progress 15% (read 16106782720 bytes, duration 2 sec)
progress 16% (read 17180590080 bytes, duration 2 sec)
progress 17% (read 18254331904 bytes, duration 2 sec)
progress 18% (read 19328139264 bytes, duration 2 sec)
progress 19% (read 20401946624 bytes, duration 2 sec)
progress 20% (read 21475688448 bytes, duration 2 sec)
progress 21% (read 22549495808 bytes, duration 2 sec)
progress 22% (read 23623303168 bytes, duration 2 sec)
progress 23% (read 24697044992 bytes, duration 3 sec)
progress 24% (read 25770852352 bytes, duration 3 sec)
progress 25% (read 26844594176 bytes, duration 3 sec)
progress 26% (read 27918401536 bytes, duration 3 sec)
progress 27% (read 28992208896 bytes, duration 3 sec)
progress 28% (read 30065950720 bytes, duration 3 sec)
progress 29% (read 31139758080 bytes, duration 3 sec)
progress 30% (read 32213565440 bytes, duration 3 sec)
progress 31% (read 33287307264 bytes, duration 3 sec)
progress 32% (read 34361114624 bytes, duration 3 sec)
progress 33% (read 35434921984 bytes, duration 3 sec)
progress 34% (read 36508663808 bytes, duration 4 sec)
progress 35% (read 37582471168 bytes, duration 4 sec)
progress 36% (read 38656278528 bytes, duration 4 sec)
progress 37% (read 39730020352 bytes, duration 4 sec)
progress 38% (read 40803827712 bytes, duration 4 sec)
progress 39% (read 41877569536 bytes, duration 4 sec)
progress 40% (read 42951376896 bytes, duration 4 sec)
progress 41% (read 44025184256 bytes, duration 4 sec)
progress 42% (read 45098926080 bytes, duration 4 sec)
progress 43% (read 46172733440 bytes, duration 5 sec)
progress 44% (read 47246540800 bytes, duration 5 sec)
progress 45% (read 48320282624 bytes, duration 5 sec)
progress 46% (read 49394089984 bytes, duration 5 sec)
progress 47% (read 50467897344 bytes, duration 5 sec)
progress 48% (read 51541639168 bytes, duration 5 sec)
progress 49% (read 52615446528 bytes, duration 5 sec)
progress 50% (read 53689188352 bytes, duration 5 sec)
progress 51% (read 54762995712 bytes, duration 5 sec)
progress 52% (read 55836803072 bytes, duration 5 sec)
progress 53% (read 56910544896 bytes, duration 5 sec)
progress 54% (read 57984352256 bytes, duration 6 sec)
progress 55% (read 59058159616 bytes, duration 6 sec)
progress 56% (read 60131901440 bytes, duration 6 sec)
progress 57% (read 61205708800 bytes, duration 6 sec)
progress 58% (read 62279516160 bytes, duration 6 sec)
progress 59% (read 63353257984 bytes, duration 6 sec)
progress 60% (read 64427065344 bytes, duration 6 sec)
progress 61% (read 65500872704 bytes, duration 6 sec)
progress 62% (read 66574614528 bytes, duration 6 sec)
progress 63% (read 67648421888 bytes, duration 6 sec)
progress 64% (read 68722163712 bytes, duration 7 sec)
progress 65% (read 69795971072 bytes, duration 7 sec)
progress 66% (read 70869778432 bytes, duration 7 sec)
progress 67% (read 71943520256 bytes, duration 7 sec)
progress 68% (read 73017327616 bytes, duration 7 sec)
progress 69% (read 74091134976 bytes, duration 7 sec)
progress 70% (read 75164876800 bytes, duration 7 sec)
progress 71% (read 76238684160 bytes, duration 7 sec)
progress 72% (read 77312491520 bytes, duration 7 sec)
progress 73% (read 78386233344 bytes, duration 7 sec)
progress 74% (read 79460040704 bytes, duration 7 sec)
progress 75% (read 80533782528 bytes, duration 8 sec)
progress 76% (read 81607589888 bytes, duration 8 sec)
progress 77% (read 82681397248 bytes, duration 8 sec)
progress 78% (read 83755139072 bytes, duration 8 sec)
progress 79% (read 84828946432 bytes, duration 8 sec)
progress 80% (read 85902753792 bytes, duration 8 sec)
progress 81% (read 86976495616 bytes, duration 8 sec)
progress 82% (read 88050302976 bytes, duration 8 sec)
progress 83% (read 89124110336 bytes, duration 8 sec)
progress 84% (read 90197852160 bytes, duration 8 sec)
progress 85% (read 91271659520 bytes, duration 8 sec)
progress 86% (read 92345466880 bytes, duration 9 sec)
progress 87% (read 93419208704 bytes, duration 9 sec)
progress 88% (read 94493016064 bytes, duration 9 sec)
progress 89% (read 95566757888 bytes, duration 9 sec)
progress 90% (read 96640565248 bytes, duration 9 sec)
progress 91% (read 97714372608 bytes, duration 9 sec)
progress 92% (read 98788114432 bytes, duration 9 sec)
progress 93% (read 99861921792 bytes, duration 9 sec)
progress 94% (read 100935729152 bytes, duration 9 sec)
progress 95% (read 102009470976 bytes, duration 9 sec)
progress 96% (read 103083278336 bytes, duration 9 sec)
progress 97% (read 104157085696 bytes, duration 9 sec)
progress 98% (read 105230827520 bytes, duration 10 sec)
progress 99% (read 106304634880 bytes, duration 10 sec)
progress 100% (read 107378376704 bytes, duration 10 sec)

Edit:

apparently it just needs extra time, it finally did complete:

total bytes read 107378376704, sparse bytes 102948888576 (95.9%)
space reduction due to 4K zero blocks 125%
rescan volumes...
zfs error: cannot open 'rpool': no such pool

cannot import 'rpool': pool was previously in use from another system.
Last accessed by (none) (hostid=2792220e) at Sun Oct 20 18:11:10 2024
zfs error: cannot open 'rpool': no such pool

could not activate storage 'local-zfs', zfs error: The pool can be imported, use 'zpool import -f' to import the pool.

TASK OK

 

[esi_y]

Hello esi_y,

I have updates and good news... I'll share more once I make sure this stuff can come back up, we're almost there, thank you for the help so far.

Great!

I'm trying to restore one of the VMs right now, I don't want to blow away the existing nvme drives (which had ZFS on them) right now to recreate the old zfs storage, I want to just restore one of the VMs to the new ext4 boot drive just to make sure the VM can even start back up again.

How do I point the restore to new storage?

If I try to restore as-is, it can't get to any of the old zfs stuff (no shit, it's not available yet).

That old pool was bound to be gone, I just wanted you to be able to peek in and take out bits and pieces, like the config.

If I make a new HD for it on the new storage, as soon as I kick off the restore again it blows away that config I just made and back at square one above.
When restoring there's also a dropdown to change the storage: great, so I select desired storage yet during the restore, it seems to still ignore what was selected and still stuck at 100% after it tries to decompress the backup... what am I missing here?

Header
Proxmox
Virtual Environment 8.2.7
Virtual Machine 100 (VM 100) on node 'pm'
No Tags
Filter VMID
Logs
()
restore vma archive: zstd -q -d -c /mnt/pve/wd8tb/dump/vzdump-qemu-100-2024_10_15-02_00_01.vma.zst | vma extract -v -r /var/tmp/vzdumptmp42118.fifo - /var/tmp/vzdumptmp42118
CFG: size: 838 name: qemu-server.conf
DEV: dev_id=1 size: 4194304 devname: drive-tpmstate0-backup
DEV: dev_id=2 size: 107374182400 devname: drive-virtio0
CTIME: Tue Oct 15 02:00:01 2024
Formatting '/mnt/pve/wd8tb/images/100/vm-100-disk-0.raw', fmt=raw size=4194304 preallocation=off
new volume ID is 'wd8tb:100/vm-100-disk-0.raw'
Formatting '/mnt/pve/wd8tb/images/100/vm-100-disk-1.raw', fmt=raw size=107374182400 preallocation=off
new volume ID is 'wd8tb:100/vm-100-disk-1.raw'
map 'drive-tpmstate0-backup' to '/mnt/pve/wd8tb/images/100/vm-100-disk-0.raw' (write zeros = 0)
map 'drive-virtio0' to '/mnt/pve/wd8tb/images/100/vm-100-disk-1.raw' (write zeros = 0)
progress 1% (read 1073807360 bytes, duration 0 sec)
...

I was just going to say it appears to be going well creating the VM disks as RAW.

Edit:

apparently it just needs extra time, it finally did complete:

total bytes read 107378376704, sparse bytes 102948888576 (95.9%)
space reduction due to 4K zero blocks 125%

That's good to hear!

rescan volumes...

zfs error: cannot open 'rpool': no such pool

cannot import 'rpool': pool was previously in use from another system.
Last accessed by (none) (hostid=2792220e) at Sun Oct 20 18:11:10 2024
zfs error: cannot open 'rpool': no such pool

could not activate storage 'local-zfs', zfs error: The pool can be imported, use 'zpool import -f' to import the pool.

TASK OK

I am not that familiar with PVE's own backup tooling, but I suspect it was just routinely rescanning for volumes that it had in the config.

Unless I missed something, all good now?

 

[esi_y]

I could not find any deeper troubleshooting on the forum (in its entire history) than the above, chances are users will come across this post eventually (if all else failed).

I might not be able to provide further updates, so the least I can do is point everyone to the filed BZ report:
https://bugzilla.proxmox.com/show_bug.cgi?id=5813

If resolved, there will be a better / comprehensive source to look for help from there.

 

[colinstu]

I plan on writing up / chronicling what I all went thru to resolve, but the meat of everything is pretty much here already posted by you.
But do want to list it out just to simplify it for potential future folks. Thank you for filing that as well!!
Truly a huge miss when it comes to official documentation on this, it should not be this difficult.

 

[esi_y]

I plan on writing up / chronicling what I all went thru to resolve, but the meat of everything is pretty much here already posted by you.
But do want to list it out just to simplify it for potential future folks. Thank you for filing that as well!!
Truly a huge miss when it comes to official documentation on this, it should not be this difficult.

Please do. I might not be around to see further BZ updates myself:
https://forum.proxmox.com/threads/educational-content.152530/page-3#post-713891

PS People underestimate contributing to BZ, then it looks like I myself have 20 issues that are solely mine.

Cheers!