Imported ZFS is empty - I cannot access backup files saved in Directory created in ZFS on USB drive

[Sypho]

Hello Everyone.

The problem I've got is accessing data (backups of VMs) from the former instance of Proxmox I was using on the same hardware

I've got fresh installation of Proxmox VE 8.1.4 running as a host for couple of VMs with things like TrueNAS, Unbound and services like Plex, Syncthing etc. I ran into a problem of not being able to (or simply not knowing how to) access VM backups stored on a USB stick.

Some backgroud and story line:

I recently made some fix to my Proxmox machine (replaced a faulty SATA controller) and decided to make some overall improvements to the set-up / make it a bit more aligned to some foundational principles. After relpacing the controller I installed new NVME disk where new version of Proxmox would go to so that I release one of SATA ports for a proper ZFS mirror config in one of the pools (due to SATA ports deficiency - SSD with a former Proxmox (kernel ~5.15.74) to be repurposed). I made back-ups of 4 VMs in the original Proxmox. I decided to move these back-up files onto external USB3 attached drive

So what I did was:
1. I plugged the drive in and created ZFS pool on it (which I think wasnt necessary).
2. I then created a Directory in that pool and this was the target location the back-ups were stored to by Proxmox (stop type).
3. After having it done I took another USB stick with Proxmox 8.1.4 image and booted the host with that, not disconnecting any of the drives including the older SSD where Proxmox 7.3 was present.
4. Proxmox installed well but labelled the former OS volume with OLD suffix and some number. Wasnt happy about that as suspected that to be a risk but this SDD is anyway planned to be cleaned up later after it's all done.
5. Proxmox starts well from NVME but UEFI got mixed a bit and I cannot enforce it to boot from the original SATA SSD now - the best it does is to boot in some hybrid fashion mode into Proxmox 8.1.4 GUI with 5.15 kernel - some odd thing that shouldnt be be happening.

My orignal Proxmox installation is at huge risk now so it's tending to be one way now.

In the fresh installation of 8.1.4 I'm able to import the 'Backup' ZFS pool (the pool on USB3 stick where there should be a Directory with backup files) and it's showing there's some data (used space) but I cannot access any of that data and have no clue how to do it.

Can anyone help me out please?

root@proxmox:~# zfs get all Backup
NAME    PROPERTY              VALUE                  SOURCE
Backup  type                  filesystem             -
Backup  creation              Fri Mar 22 22:08 2024  -
Backup  used                  12.4G                  -
Backup  available             218G                   -
Backup  referenced            12.4G                  -
Backup  compressratio         1.00x                  -
Backup  mounted               yes                    -
Backup  quota                 none                   default
Backup  reservation           none                   default
Backup  recordsize            128K                   default
Backup  mountpoint            /Backup                default
Backup  sharenfs              off                    default
Backup  checksum              on                     default
Backup  compression           off                    local
Backup  atime                 on                     default
Backup  devices               on                     default
Backup  exec                  on                     default
Backup  setuid                on                     default
Backup  readonly              off                    default
Backup  zoned                 off                    default
Backup  snapdir               hidden                 default
Backup  aclmode               discard                default
Backup  aclinherit            restricted             default
Backup  createtxg             1                      -
Backup  canmount              on                     default
Backup  xattr                 on                     default
Backup  copies                1                      default
Backup  version               5                      -
Backup  utf8only              off                    -
Backup  normalization         none                   -
Backup  casesensitivity       sensitive              -
Backup  vscan                 off                    default
Backup  nbmand                off                    default
Backup  sharesmb              off                    default
Backup  refquota              none                   default
Backup  refreservation        none                   default
Backup  guid                  17171301617694326878   -
Backup  primarycache          all                    default
Backup  secondarycache        all                    default
Backup  usedbysnapshots       0B                     -
Backup  usedbydataset         12.4G                  -
Backup  usedbychildren        1.15M                  -
Backup  usedbyrefreservation  0B                     -
Backup  logbias               latency                default
Backup  objsetid              54                     -
Backup  dedup                 off                    default
Backup  mlslabel              none                   default
Backup  sync                  standard               default
Backup  dnodesize             legacy                 default
Backup  refcompressratio      1.00x                  -
Backup  written               12.4G                  -
Backup  logicalused           12.4G                  -
Backup  logicalreferenced     12.4G                  -
Backup  volmode               default                default
Backup  filesystem_limit      none                   default
Backup  snapshot_limit        none                   default
Backup  filesystem_count      none                   default
Backup  snapshot_count        none                   default
Backup  snapdev               hidden                 default
Backup  acltype               off                    default
Backup  context               none                   default
Backup  fscontext             none                   default
Backup  defcontext            none                   default
Backup  rootcontext           none                   default
Backup  relatime              on                     default
Backup  redundant_metadata    all                    default
Backup  overlay               on                     default
Backup  encryption            off                    default
Backup  keylocation           none                   default
Backup  keyformat             none                   default
Backup  pbkdf2iters           0                      default
Backup  special_small_blocks  0                      default

root@proxmox:~# zpool status
  pool: Backup
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
config:

        NAME                                            STATE     READ WRITE CKSUM
        Backup                                          ONLINE       0     0     0
          usb-Samsung_Flash_Drive_0321821050002687-0:0  ONLINE       0     0     0

errors: No known data errors

root@proxmox:~# mount /dev/sdg1 /mnt/pve/restore
mount: /mnt/pve/restore: unknown filesystem type 'zfs_member'.
       dmesg(1) may have more information after failed mount system call.

root@proxmox:~# fdisk -l
Disk /dev/sdg: 239.02 GiB, 256641603584 bytes, 501253132 sectors
Disk model: Flash Drive     
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 42F3685E-41BE-A34B-89FC-7A1F6E97EEA6

Device         Start       End   Sectors  Size Type
/dev/sdg1       2048 501235711 501233664  239G Solaris /usr & Apple ZFS
/dev/sdg9  501235712 501252095     16384    8M Solaris reserved 1

I know it's more about basics - please appreciate I'm not an expert in Proxmox nor unix/linux shell stuff and this is not an enterprise set-up

Thank you!

 

[Dunuin]

1. I plugged the drive in and created ZFS pool on it (which I think wasnt necessary).

ZFS on a USB stick would be a very bad idea.

To see whats on that pool you should run zfs list -v as well as checking if datasets are mounted and where zfs get mounted,mountpoint and then ls -la /path/to/mounted/dataset to see the files.

I guess you didn't enabled the "is_mountpoint" option for that directroy storage via CLI using "pvesm"? Because when using a wrong path or failing a mount it might be possible that you actually wrote all your backups to the root filesystem of your old PVE installation instead of to the USB stick. You wouldn't be the first person I'm reading here that wiped the old PVE installation with the backups on its root partition thinking there are backups on an external disk which weren't because of a badly configured directory storage.

 

[Sypho]

ZFS on a USB stick would be a very bad idea.

To see whats on that pool you should run zfs list -v as well as checking if datasets are mounted and where zfs get mounted,mountpoint and then ls -la /path/to/mounted/dataset to see the files.

I guess you didn't enabled the "is_mountpoint" option for that directroy storage via CLI using "pvesm"? Because when using a wrong path or failing a mount it might be possible that you actually wrote all your backups to the root filesystem of your old PVE installation instead of to the USB stick. You wouldn't be the first person I'm reading here that wiped the old PVE installation with the backups on its root partition thinking there are backups on an external disk which weren't because of a badly configured directory storage.

Thank you Dunuin for your response!

Yeah, agree on USB stick not being a wise idea... Nevertheless I checked your suggestions.

1. I checked zfs get mounted with that in result

root@proxmox:~# zfs get mounted
NAME    PROPERTY  VALUE    SOURCE
Backup  mounted   yes      -

2. and mountpoint

root@proxmox:~# mountpoint -d /Backup
0:38

3. I then checked zfs list -vbut it didn't work in my case giving a list of options for zfs list available. I thought of trying zpool list -v though and it worked giving this in result

root@proxmox:/Backup/backup/dump# zpool list -v
NAME                                             SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
Backup                                           238G  12.4G   226G        -         -     0%     5%  1.00x    ONLINE  -
  usb-Samsung_Flash_Drive_0321821050002687-0:0   239G  12.4G   226G        -         -     0%  5.21%      -    ONLINE

4. I moved to /Backup based on the above

root@proxmox:~# cd /Backup
root@proxmox:/Backup#

5. and ran ls -la

root@proxmox:/Backup# ls -la
total 13
drwxr-xr-x  3 root root    3 Mar 22 22:14 .
drwxr-xr-x 19 root root 4096 Mar 23 22:42 ..
drwxr-xr-x  7 root root    7 Mar 22 22:14 backup

root@proxmox:/Backup# cd backup
root@proxmox:/Backup/backup# ls
dump  images  private  snippets  template

root@proxmox:/Backup/backup# ls -la
total 36
drwxr-xr-x 7 root root  7 Mar 22 22:14 .
drwxr-xr-x 3 root root  3 Mar 22 22:14 ..
drwxr-xr-x 3 root root 16 Mar 22 22:49 dump
drwxr-xr-x 2 root root  2 Mar 22 22:14 images
drwxr-xr-x 2 root root  2 Mar 22 22:14 private
drwxr-xr-x 2 root root  2 Mar 22 22:14 snippets
drwxr-xr-x 4 root root  4 Mar 22 22:14 template

root@proxmox:/Backup/backup# cd dump
root@proxmox:/Backup/backup/dump# ls -la
total 13021948
drwxr-xr-x 3 root root         16 Mar 22 22:49 .
drwxr-xr-x 7 root root          7 Mar 22 22:14 ..
-rw-r--r-- 1 root root        654 Mar 22 22:16 vzdump-lxc-103-2024_03_22-22_15_18.log
-rw-r--r-- 1 root root 1170268160 Mar 22 22:16 vzdump-lxc-103-2024_03_22-22_15_18.tar
-rw-r--r-- 1 root root          6 Mar 22 22:16 vzdump-lxc-103-2024_03_22-22_15_18.tar.notes
-rw-r--r-- 1 root root 5386357760 Mar 22 22:21 vzdump-qemu-100-2024_03_22-22_20_23.dat
drwxr-xr-x 2 root root          3 Mar 22 22:20 vzdump-qemu-100-2024_03_22-22_20_23.tmp
-rw-r--r-- 1 root root       7497 Mar 22 22:49 vzdump-qemu-100-2024_03_22-22_45_11.log
-rw-r--r-- 1 root root 6382380544 Mar 22 22:49 vzdump-qemu-100-2024_03_22-22_45_11.vma
-rw-r--r-- 1 root root          7 Mar 22 22:49 vzdump-qemu-100-2024_03_22-22_45_11.vma.notes
-rw-r--r-- 1 root root       1609 Mar 22 22:17 vzdump-qemu-101-2024_03_22-22_16_40.log
-rw-r--r-- 1 root root  375743488 Mar 22 22:17 vzdump-qemu-101-2024_03_22-22_16_40.vma
-rw-r--r-- 1 root root          7 Mar 22 22:17 vzdump-qemu-101-2024_03_22-22_16_40.vma.notes
-rw-r--r-- 1 root root       2984 Mar 22 22:18 vzdump-qemu-102-2024_03_22-22_17_38.log
-rw-r--r-- 1 root root   10427392 Mar 22 22:18 vzdump-qemu-102-2024_03_22-22_17_38.vma
-rw-r--r-- 1 root root          9 Mar 22 22:18 vzdump-qemu-102-2024_03_22-22_17_38.vma.notes

root@proxmox:/Backup/backup/images# ls -la
total 9
drwxr-xr-x 2 root root 2 Mar 22 22:14 .
drwxr-xr-x 7 root root 7 Mar 22 22:14 ..

root@proxmox:/Backup/backup/private# ls -la
total 9
drwxr-xr-x 2 root root 2 Mar 22 22:14 .
drwxr-xr-x 7 root root 7 Mar 22 22:14 ..

root@proxmox:/Backup/backup/snippets# ls -la
total 9
drwxr-xr-x 2 root root 2 Mar 22 22:14 .
drwxr-xr-x 7 root root 7 Mar 22 22:14 ..

root@proxmox:/Backup/backup/template# ls -la
total 10
drwxr-xr-x 4 root root 4 Mar 22 22:14 .
drwxr-xr-x 7 root root 7 Mar 22 22:14 ..
drwxr-xr-x 2 root root 2 Mar 22 22:14 cache
drwxr-xr-x 2 root root 2 Mar 22 22:14 iso

-----

Looks like I've had it in front of my eyes but was unaware how to get to this.
Started restoring VMs with qmrestore

How would I do it in GUI while I'm unable to see these files in GUI?

Big thanks again!

 

[Dunuin]

How would I do it in GUI while I'm unable to see these files in GUI?

Adding a Directory storage at "Datacenter -> Storage -> Add -> Directroy" pointing to "/Backup" with "Vzdump Backups" as content type and then looking at the "Backups" tab of that Directory storage.

 

[Sypho]

Adding a Directory storage at "Datacenter -> Storage -> Add -> Directroy" pointing to "/Backup" with "Vzdump Backups" as content type and then looking at the "Backups" tab of that Directory storage.

Thanks! I'd tried that before asking the question but it hadn't worked. I must've had something wrong.
Succeeded with restoring the VMs and the container the way described above though - it went smooth from that point onward.
It was a good lesson to me. Thanks again!