Manually Restore Backup After Joining Cluster

Tatariushi · Jul 12, 2023

Hello.
I just added a cluster and have two total nodes. One node was a machine with 2 VMs on it before joining the cluster.

To prepare to join the cluster, I created a backup (locally) of the two VMs.
After joining the cluster, any reference to the two backups is now gone from the web GUI, but I see the files from the backups at /mnt/pve/Sabrent1TB/dump
I'm attempting to restore the VMs manually via the CLI, but running into trouble.
I learned about qmrestore, and using the following command, the backup created a VM with the original config:
- qmrestore vzdump-qemu-100-2023_07_11-22_27_35.vma.zst 200 -storage local-lvm
However when I go to start the VM, I get the following error
- TASK ERROR: storage 'ZFS-MIRROR-13TB' does not exist
ZFS-MIRROR-13TB is where I had a ~13TB drive that was the D: drive in my windows install - mostly media, about half full
Looking at Datacenter > node > Disks > ZFS, I still see the following entry
- Name: ZFS-MIRROR-13TB, Size: 13.98TB, Free: 13.97 TB, Allocated: 9.61GB, Fragmentation: 1%, Health: Online
That's far more free space than I would expect. Did joining the cluster wipe out the ~13TB disk image I had stored there? Is it possible this might be recoverable?

Edit: I've confirmed the disk image ZFS-MIRROR-13TB is actually gone from the file system.

Chris · Jul 13, 2023

Tatariushi said:
Hello.
I just added a cluster and have two total nodes. One node was a machine with 2 VMs on it before joining the cluster.

To prepare to join the cluster, I created a backup (locally) of the two VMs.

After joining the cluster, any reference to the two backups is now gone from the web GUI, but I see the files from the backups at /mnt/pve/Sabrent1TB/dump

I'm attempting to restore the VMs manually via the CLI, but running into trouble.

I learned about qmrestore, and using the following command, the backup created a VM with the original config:

qmrestore vzdump-qemu-100-2023_07_11-22_27_35.vma.zst 200 -storage local-lvm

However when I go to start the VM, I get the following error

TASK ERROR: storage 'ZFS-MIRROR-13TB' does not exist

ZFS-MIRROR-13TB is where I had a ~13TB drive that was the D: drive in my windows install - mostly media, about half full

Looking at Datacenter > node > Disks > ZFS, I still see the following entry

Name: ZFS-MIRROR-13TB, Size: 13.98TB, Free: 13.97 TB, Allocated: 9.61GB, Fragmentation: 1%, Health: Online

That's far more free space than I would expect. Did joining the cluster wipe out the ~13TB disk image I had stored there? Is it possible this might be recoverable?

Edit: I've confirmed the disk image ZFS-MIRROR-13TB is actually gone from the file system.

Hi,
as stated in the docs, you will have to recreate the storage topology after the join [0].

Please provide the outputs to the following commands:

Bash:

lsblk -o +FSTYPE
zpool status
zfs list
lvdisplay
cat /etc/pve/storage.cfg

from the node which joined the cluster.

On a side note: It is not recommended to run a 2 node cluster, consider adding an external voter as tie breaker, see [1] for more details.

[0] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_join_node_to_cluster
[1] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_corosync_external_vote_support

Tatariushi · Jul 13, 2023

Thanks you, Chris. Here are the different outputs:

Code:

root@proxmox-ve1:/mnt# lsblk -o +FSTYPE
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT          FSTYPE
sda                            8:0    0  12.7T  0 disk                    
├─sda1                         8:1    0  12.7T  0 part                     zfs_member
└─sda9                         8:9    0     8M  0 part                    
sdb                            8:16   0  12.7T  0 disk                    
├─sdb1                         8:17   0  12.7T  0 part                     zfs_member
└─sdb9                         8:25   0     8M  0 part                    
sdc                            8:32   0 894.3G  0 disk                    
└─sdc1                         8:33   0 894.3G  0 part                     exfat
nvme1n1                      259:0    0 931.5G  0 disk                    
└─nvme1n1p1                  259:1    0 931.5G  0 part /mnt/pve/Sabrent1TB xfs
nvme0n1                      259:2    0 238.5G  0 disk                    
├─nvme0n1p1                  259:3    0  1007K  0 part                    
├─nvme0n1p2                  259:4    0   512M  0 part /boot/efi           vfat
└─nvme0n1p3                  259:5    0   238G  0 part                     LVM2_member
  ├─pve-swap                 253:0    0     8G  0 lvm  [SWAP]              swap
  ├─pve-root                 253:1    0  69.5G  0 lvm  /                   ext4
  ├─pve-data_tmeta           253:2    0   1.4G  0 lvm                    
  │ └─pve-data-tpool         253:4    0 141.6G  0 lvm                    
  │   ├─pve-data             253:5    0 141.6G  1 lvm                    
  │   ├─pve-vm--210--disk--0 253:7    0   128G  0 lvm                    
  │   └─pve-vm--200--disk--2 253:9    0   256G  0 lvm                    
  └─pve-data_tdata           253:3    0 141.6G  0 lvm                    
    └─pve-data-tpool         253:4    0 141.6G  0 lvm                    
      ├─pve-data             253:5    0 141.6G  1 lvm                    
      ├─pve-vm--210--disk--0 253:7    0   128G  0 lvm                    
      └─pve-vm--200--disk--2 253:9    0   256G  0 lvm

Code:

root@proxmox-ve1:/mnt# zpool status
  pool: ZFS-MIRROR-13TB
 state: ONLINE
  scan: scrub repaired 0B in 09:17:19 with 0 errors on Sun Jul  9 09:41:20 2023
config:

        NAME                                  STATE     READ WRITE CKSUM
        ZFS-MIRROR-13TB                       ONLINE       0     0     0
          mirror-0                            ONLINE       0     0     0
            ata-WDC_WUH721414ALE604_9JJR1MAT  ONLINE       0     0     0
            ata-WDC_WUH721414ALE604_9JKSWYKT  ONLINE       0     0     0

errors: No known data errors

Code:

root@proxmox-ve1:/mnt# zfs list
NAME                   USED  AVAIL     REFER  MOUNTPOINT
ZFS-MIRROR-13TB       8.95G  12.6T       96K  /ZFS-MIRROR-13TB
ZFS-MIRROR-13TB/ISOs  8.83G  12.6T     8.83G  /ZFS-MIRROR-13TB/ISOs
root@proxmox-ve1:/mnt# lvdisplay
  --- Logical volume ---
  LV Path                /dev/pve/swap
  LV Name                swap
  VG Name                pve
  LV UUID                vhD9C0-ypdg-IYLk-siWQ-5b4L-jitQ-mUwO5C
  LV Write Access        read/write
  LV Creation host, time proxmox, 2022-11-26 09:59:27 -0800
  LV Status              available
  # open                 2
  LV Size                8.00 GiB
  Current LE             2048
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0
 
  --- Logical volume ---
  LV Path                /dev/pve/root
  LV Name                root
  VG Name                pve
  LV UUID                4JoTrH-rrH3-9JkE-sOTc-CEMO-QUvr-jGyUgN
  LV Write Access        read/write
  LV Creation host, time proxmox, 2022-11-26 09:59:27 -0800
  LV Status              available
  # open                 1
  LV Size                <69.50 GiB
  Current LE             17791
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1
 
  --- Logical volume ---
  LV Name                data
  VG Name                pve
  LV UUID                i9ka1s-8We0-54Gn-p0ST-0XbU-0hWW-uBdQT2
  LV Write Access        read/write (activated read only)
  LV Creation host, time proxmox, 2022-11-26 09:59:28 -0800
  LV Pool metadata       data_tmeta
  LV Pool data           data_tdata
  LV Status              available
  # open                 0
  LV Size                <141.59 GiB
  Allocated pool data    100.00%
  Allocated metadata     3.61%
  Current LE             36247
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:5
 
  --- Logical volume ---
  LV Path                /dev/pve/vm-210-disk-0
  LV Name                vm-210-disk-0
  VG Name                pve
  LV UUID                xf9pqQ-drcy-CxO5-0xzX-fZcv-aae6-0Kyrwd
  LV Write Access        read/write
  LV Creation host, time proxmox-ve1, 2023-07-11 23:46:53 -0700
  LV Pool name           data
  LV Status              available
  # open                 0
  LV Size                128.00 GiB
  Mapped size            38.10%
  Current LE             32768
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:7
 
  --- Logical volume ---
  LV Path                /dev/pve/vm-200-disk-2
  LV Name                vm-200-disk-2
  VG Name                pve
  LV UUID                9YrjeZ-5AgP-xBrZ-VRHo-Vt7I-sg4H-GznUjD
  LV Write Access        read/write
  LV Creation host, time proxmox-ve1, 2023-07-11 23:55:32 -0700
  LV Pool name           data
  LV Status              available
  # open                 0
  LV Size                256.00 GiB
  Mapped size            36.26%
  Current LE             65536
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:9

Code:

root@proxmox-ve1:/mnt# cat /etc/pve/ctorage.cfg
cat: /etc/pve/ctorage.cfg: No such file or directory

Regarding creating a cluster at all - as a fairly new home user, my primary goal was simply to be able to manage both nodes from a single webGUI. I'm not trying to do replication or anything like that (yet.)

Would I be better off removing the cluster entirely? Would that mean losing the existing VMs again?
Would it be fairly simple to backup the individual VMs and restore them after doing this? I'll be setting up Proxmox Backup Server this weekend on a new system. Given I got myself into this mess with local backups, does PBS make this easier? (I think what I'm really asking is, if I backup the current VMs under the cluster, then break the cluster, can PBS adapt restores to the change in topology if I restore them in a non-cluster environment?)

Chris · Jul 14, 2023

Tatariushi said:
Would I be better off removing the cluster entirely? Would that mean losing the existing VMs again?

Would it be fairly simple to backup the individual VMs and restore them after doing this? I'll be setting up Proxmox Backup Server this weekend on a new system. Given I got myself into this mess with local backups, does PBS make this easier? (I think what I'm really asking is, if I backup the current VMs under the cluster, then break the cluster, can PBS adapt restores to the change in topology if I restore them in a non-cluster environment?)

So I am afraid you deleted your data volume containing the media when deleting the VM prior to joining the cluster. This data is gone, as it was not included in the backup. The cluster join itself should not have deleted the volume.

The restored VM fails to start, because the VM config still references the now unavailable storage and disk. You can readd the storage by simply readding it via `Datacenter -> Storage -> Add` and recreate it using the same ID, or recreate it in `/etc/pve/storage.cfg`. Same is true for your `Sabrent1TB` storage.

Regarding your questions:

1. a 2 node cluster will always cause issues, since if one of the two node fails, the other one looses quorum and switches into read only. Therefore it is suggested to use a tie breaker. You will not loose your VMs when removing a node from a cluster, as long as they are not on that cluster.

2. You can backup to a PBS instance on a different host, reinstall PVE, add the PBS as storage again and restore the VMs. Upon restore you must give a target storage if the configured storage for a disk is not present.

Hope this answers your questions.

Tatariushi · Jul 14, 2023

Thank you Chris, very helpful.

My current plan is to get PBS up and running this weekend, backup the two currently-functional VMs to PBS (OPNSense and HA - my two most critical systems for my home,) break the cluster, then restore those two VMs to the now-non-cluster PVE.

Next, for the other two VMs which I have been unable to restore so far (a Windows server and an Ubuntu server running docker.) These exist as local backups that I have been unable to restore after joining the cluster and the change in topology. I can see the backup files here

Code:

/mnt/pve/Sabrent1TB/dump# ls
dump                                               vzdump-qemu-100-2023_07_11-22_27_35.vma.zst
vzdump-qemu-100-2023_07_11-19_36_45.log            vzdump-qemu-100-2023_07_11-22_27_35.vma.zst.notes
vzdump-qemu-100-2023_07_11-19_36_45.vma.zst        vzdump-qemu-102-2023_07_11-20_58_27.log
vzdump-qemu-100-2023_07_11-19_36_45.vma.zst.notes  vzdump-qemu-102-2023_07_11-20_58_27.vma.zst
vzdump-qemu-100-2023_07_11-22_27_35.log            vzdump-qemu-102-2023_07_11-20_58_27.vma.zst.notes

I will reformat and reinstall PVE on this machne, so I'll copy these to a removable drive first.

After reinstalling PVE on this machine, is there a way to get them to appear in the GUI?
Will I create extra trouble for myself if I change the node name when I reinstall PVE?

Chris · Jul 17, 2023

Tatariushi said:
Thank you Chris, very helpful.

My current plan is to get PBS up and running this weekend, backup the two currently-functional VMs to PBS (OPNSense and HA - my two most critical systems for my home,) break the cluster, then restore those two VMs to the now-non-cluster PVE.

Next, for the other two VMs which I have been unable to restore so far (a Windows server and an Ubuntu server running docker.) These exist as local backups that I have been unable to restore after joining the cluster and the change in topology. I can see the backup files here

Code:

/mnt/pve/Sabrent1TB/dump# ls dump vzdump-qemu-100-2023_07_11-22_27_35.vma.zst vzdump-qemu-100-2023_07_11-19_36_45.log vzdump-qemu-100-2023_07_11-22_27_35.vma.zst.notes vzdump-qemu-100-2023_07_11-19_36_45.vma.zst vzdump-qemu-102-2023_07_11-20_58_27.log vzdump-qemu-100-2023_07_11-19_36_45.vma.zst.notes vzdump-qemu-102-2023_07_11-20_58_27.vma.zst vzdump-qemu-100-2023_07_11-22_27_35.log vzdump-qemu-102-2023_07_11-20_58_27.vma.zst.notes

I will reformat and reinstall PVE on this machne, so I'll copy these to a removable drive first.

After reinstalling PVE on this machine, is there a way to get them to appear in the GUI?

Will I create extra trouble for myself if I change the node name when I reinstall PVE?

1. -> Yes, just place them in the dump folder of the storage where you want to make them appear and make sure the storage is configured for containing backups.
2. -> No, renaming the node will not cause any troubles, at least not for the backups if that is your question.

Search

Search

Manually Restore Backup After Joining Cluster

Tatariushi

New Member

Chris

Proxmox Staff Member

Tatariushi

New Member

Chris

Proxmox Staff Member

Tatariushi

New Member

Chris

Proxmox Staff Member