[SOLVED] Add SSD/ZFS/Pool with VMs to new Proxmox server

[mrE]

Good day.

I am practicing with Proxmox; my interest is to get the following to be prepared:

Simulate a disaster on Proxmox (without PBS), install a new Proxmox and use the SSD/ZFS/Pool with VMs on the new server.
This, because I want to use a development Proxmox that I could take from the office to another remote location, where for some reason I would not have access to a PBS and there will be major changes to the VM.

I initially did a Promox installation (ISO), and added an SSD with ZFS (Single Disk):
ID:local-zfs
ZFS Pool: rpool/data
With a VM/windows.

Then, simulating a Proxmox damage (with reinstallation), I try to add the disk with the previous pool in this new installation to continue using the VM/windows.

Note: I did this new installation with Debian Bulls-eye; following the guide; so far so good. Except that I don't manage to add the SSD/ZFS/pool.




The disk with the previous server VM is not showing up as unused.



Thank you in advance.

 

[shanreich]

I assume you want to add the existing ZFS pool as a storage to your new installation? You are in the Disk overview section, but in order to add it as a storage, it should be sufficient to navigate to Datacenter > Storage > Add > ZFS > *select your zfs pool* .

 

[mrE]

I assume you want to add the existing ZFS pool as a storage to your new installation? You are in the Disk overview section, but in order to add it as a storage, it should be sufficient to navigate to Datacenter > Storage > Add > ZFS > *select your zfs pool* .

I tried that last night (reading other posts), but no pool appears.
I remember that I even tried typing the data in the fields (in Datacenter > Storage > Add > ZFS > *select your zfs pool*) but it does not show the pool.
I'll post a screenshot when I get home.

Some time ago when I tried to do another reinstall where the contents of the pool didn't matter, I remember I had to do something to make the disks show up as unused; but I can't find my notes.
I wonder if that will need to be done (and how to do it?), but without losing the VM inside the pool.

Thanks

 

[Dunuin]

Did you import the pool first? zpool import to list importable pools and zpool import PoolName to import it. You might need to use the "-f" flag to force importing it in case it wasn't exported previously with zpool export PoolName .

 

[mrE]

Did you import the pool first? zpool import to list importable pools and zpool import PoolName to import it. You might need to use the "-f" flag to force importing it in case it wasn't exported previously with zpool export PoolName .

I haven't done that (yet).
I'm still away from home, I'll try again later as soon as I can.

Thanks for your help!

 

[mrE]

Did you import the pool first? zpool import to list importable pools and zpool import PoolName to import it. You might need to use the "-f" flag to force importing it in case it wasn't exported previously with zpool export PoolName .

It works!!! Thank you @Dunuin !!!

--
Now I need to research how to add a "new" VM (maybe with the same Id) and attach the disks (vm-100-disk-#) from the pool (next step of my simulation).

 

[Dunuin]

You might need to use this, so new VMs will recognize the old disks as "unused" disks which you then can add:

qm rescan [OPTIONS]
Rescan all storages and update disk sizes and unused disk images.
--dryrun <boolean> (default =0)
Do not actually write changes out to VM config(s).
--vmid <integer> (1 - N)
The (unique) ID of the VM.

 

[mrE]

You might need to use this, so new VMs will recognize the old disks as "unused" disks which you then can add:

After a few days busy with other projects, I managed to do all the suggested steps.

Dunion. Where you say that I should use "qm rescan", when I try it without creating the VM it shows me nothing: Only after creating a VM with the same ID it showed me the SSD disks: Perfect!

Sadly, with so many pauses, I forgot to take note of my VM configuration and now I can't start it; but of course, the VM does try to boot from the SSD.



Now with all your advice in hand, I will try to replicate everything again (as soon as I get some free time):

• New Proxmox + VM (taking note of the configuration of each VM).
• Simulate the failure/disaster in Proxmox
• Reinstall and add the SSD/ZFS/Storage + Disks

Thank you! @shanreich and @Dunuin

 

[Dunuin]

(taking note of the configuration of each VM).

VM/LXC configs are stored in "/etc/pve/qemu-server" and "/etc/pve/lxc". You don't really have to take notes on creating VMs. Just make sure you always have a recent backup of the entire "/etc" folder.

 

[mrE]

VM/LXC configs are stored in "/etc/pve/qemu-server" and "/etc/pve/lxc". You don't really have to take notes on creating VMs. Just make sure you always have a recent backup of the entire "/etc" folder.

Good advice; I didn't know that.
And since I really simulated a total disaster (no backups) I didn't have any /etc/folder backups.

I want to simulate a total disaster without backups; of course taking note of the previous Proxmox configuration. It will be easier to copy the configurations as you mention.

But right now I take note of that advice.

 

[Dunuin]

For a desaster I like to:
1.) every several months or before upgrading PVE to a new major version I will shutdown the PVE server, boot a...
1.a) clonezilla ISO form USB stick to backup the whole PVE system disk on block level to a archive file or
1.b) Debian from USB stick with the proxmox-backup-client (PBC) and use that PBC to do a block-level backup of the full PVE system disk and store it on my proxmox-backp-server (PBS)
2.) setup a automated task using crontab that will do a daily or weekly recursive backup of the folders "/etc", "/root", "/var" and "/home". This can be done using...
2.a) a tool of your choice like cp + tar + gzip if you just want to copy the folder to a backup location or
2.b) use the PBC to backup those folders and store them on my PBS
3.) setup 2 backup job using the proxmox webUI. One for daily backups and one for weekly backups and let them backup your VMs and LXCs. You might want to keep several backups to have some versioning. For example 6 daily and 12 weekly backups. This can either be done using...
3.a) Vzdump to store the backups at some location like a USB HDD or NAS or
3.b) to a PBS (which is recommended as it is way faster and only needs a fraction of the space)
4.) setup cv4pve-autosnapshot to take hourly snapshots and prune them after 24 hours. So in case you did something wrong and realize it fast, you can just rollback a snapshot so you don't have to restore a daily backup. This is way faster and you only loose a hour of data and not a whole day.

And yes, keeping a good documentation is also a good idea. Especially when you install and setup additional packages to your PVE.

 

[mrE]

For a desaster I like to:
1.) every several months or before upgrading PVE to a new major version I will shutdown the PVE server, boot a...
1.a) clonezilla ISO form USB stick to backup the whole PVE system disk on block level to a archive file or
1.b) Debian from USB stick with the proxmox-backup-client (PBC) and use that PBC to do a block-level backup of the full PVE system disk and store it on my proxmox-backp-server (PBS)
2.) setup a automated task using crontab that will do a daily or weekly recursive backup of the folders "/etc", "/root", "/var" and "/home". This can be done using...
2.a) a tool of your choice like cp + tar + gzip if you just want to copy the folder to a backup location or
2.b) use the PBC to backup those folders and store them on my PBS
3.) setup 2 backup job using the proxmox webUI. One for daily backups and one for weekly backups and let them backup your VMs and LXCs. You might want to keep several backups to have some versioning. For example 6 daily and 12 weekly backups. This can either be done using...
3.a) Vzdump to store the backups at some location like a USB HDD or NAS or
3.b) to a PBS (which is recommended as it is way faster and only needs a fraction of the space)
4.) setup cv4pve-autosnapshot to take hourly snapshots and prune them after 24 hours. So in case you did something wrong and realize it fast, you can just rollback a snapshot so you don't have to restore a daily backup. This is way faster and you only loose a hour of data and not a whole day.

And yes, keeping a good documentation is also a good idea. Especially when you install and setup additional packages to your PVE.

Thanks for your list to prevent disasters @Dunuin

I need to read it very carefully, as there are things I don't know (even though I started reading I still don't understand how to use or install some).

• (1b) PBC, I didn't know it or that I could use it in Promox.
• (2b) Debian USB with PBC
• (3b) seems to me very necessary and useful, I already use PBS for my VMs
• (4) is totally unknown to me I have to read your link.

Since I have a PBS: (3b) is similar to (4)?

Thanks, I look forward to asking you after reading and researching your points.

 

[Dunuin]

Thanks for your list to prevent disasters @Dunuin

I need to read it very carefully, as there are things I don't know (even though I started reading I still don't understand how to use or install some).

• (1b) PBC, I didn't know it or that I could use it in Promox.

• (2b) Debian USB with PBC

I think you mixed 1b and 2b up.
Have look at the documentation: https://pbs.proxmox.com/docs/backup-client.html
When doing block level backups that block device shouldn't be mounted, so you shouldn't do block level backup of your PVEs system disk while PVE is booted. Thats why I said you should boot a Debian from a USB stick and install the PBC to it.
But what you can do while PVE is booted is doing file level backups to for example backup the PVE config files.
There is even an example of how to backup the PVE servers root folder: https://pbs.proxmox.com/docs/backup-client.html#creating-backups

Since I have a PBS: (3b) is similar to (4)?

Snapshots will never replace a real backup, as snapshots are stored on the same disk. If your disk fails, you lose the VMs and the snapshots of those VMs. But snapshots are nice if you want to quickly revert some changes. For example, doing a snapshot before updating a VM. If you then realize you screwed up the update or the VMs aren't able to run any longer you can easily roll back that snapshot and try it again or dn't update them at all.
Snapshots aren't great for long-term archival, as they will grow the more the older they get. But they are great as they are fast. You can snapshot or roll back a 10TB VM within a second. Doing a backup and restore might take many hours.
So it's a good idea to combine snapshots and backups. Backups for daily/weekly/monthly/annually long-term archival and snapshots for the short-term archival wen you want to be able to revert your guests in something like 1 hour of 15 minute intervals.

 

[mrE]

I think you mixed 1b and 2b up.
Have look at the documentation: https://pbs.proxmox.com/docs/backup-client.html
When doing block level backups that block device shouldn't be mounted, so you shouldn't do block level backup of your PVEs system disk while PVE is booted. Thats why I said you should boot a Debian from a USB stick and install the PBC to it.
But what you can do while PVE is booted is doing file level backups to for example backup the PVE config files.
There is even an example of how to backup the PVE servers root folder: https://pbs.proxmox.com/docs/backup-client.html#creating-backups

Snapshots will never replace a real backup, as snapshots are stored on the same disk. If your disk fails, you lose the VMs and the snapshots of those VMs. But snapshots are nice if you want to quickly revert some changes. For example, doing a snapshot before updating a VM. If you then realize you screwed up the update or the VMs aren't able to run any longer you can easily roll back that snapshot and try it again or dn't update them at all.
Snapshots aren't great for long-term archival, as they will grow the more the older they get. But they are great as they are fast. You can snapshot or roll back a 10TB VM within a second. Doing a backup and restore might take many hours.
So it's a good idea to combine snapshots and backups. Backups for daily/weekly/monthly/annually long-term archival and snapshots for the short-term archival wen you want to be able to revert your guests in something like 1 hour of 15 minute intervals.

Yes, when re-editing the text I mistakenly mixed the two. (facepalm)

Now that I re-read your comment, I understand about cv4pve-autosnapshot, it sounds very attractive.

One doubt here:
In case of having to migrate the VM (with snapshots) from server-A to server-B. what happens to the snapshots, do they stay on server-A?
I don't remember well, but I think I had problems at some point because of snapshots, so I stopped using them.

And related to snapshots (Sorry for the change of topic):
I wonder why I can't do snapshot on this VM with windows 11, all the others (Linux, CentOS, Windows 2019, 7) can do snapshot (I don't use snapshot actually).
The biggest difference I see between this VM and the others is that in W11 I added some USB ports from the host (since I program for some android devices) and to support UEFI use another BIOS.

Is it because I have shared hardware from the virtualized Host in the VM?


Thank you! @Dunuin

 

[Dunuin]

Yes, when re-editing the text I mistakenly mixed the two. (facepalm)

Now that I re-read your comment, I understand about cv4pve-autosnapshot, it sounds very attractive.

One doubt here:
In case of having to migrate the VM (with snapshots) from server-A to server-B. what happens to the snapshots, do they stay on server-A?
I don't remember well, but I think I had problems at some point because of snapshots, so I stopped using them.

I think that depends on the underlaying storage, as PVE won't do it's own snapshots, it will make use of the snapshots features of the LVM/ZFS/Ceph/Qcow2 storage. So snapshots will behave differently depending on the storage used.
But even if you can't migrate snapshot this wouldn't be a problem as you should have daily backups anyway.
Doing backups all 15 minutes would be way too slow and ressource intense. So the option would be no snapshots at all and only backups every day or all several hours. Or the same backups but snapshots every couple of minutes/hours in addition to that.
So using snapshots will just make it better.

And related to snapshots (Sorry for the change of topic):
I wonder why I can't do snapshot on this VM with windows 11, all the others (Linux, CentOS, Windows 2019, 7) can do snapshot (I don't use snapshot actually).
The biggest difference I see between this VM and the others is that in W11 I added some USB ports from the host (since I program for some android devices) and to support UEFI use another BIOS.

Is it because I have shared hardware from the virtualized Host in the VM?

View attachment 42487
Thank you! @Dunuin

You are only allowed to do snapshots if all storages of that guest will support snapshotting. So this won't work if you for example do a HDD passthrough to that VM.

 

[mrE]

You are only allowed to do snapshots if all storages of that guest will support snapshotting.

I am using two Synology with NFS folders (one in production, the other in standby) to store the virtual disks of the VMs (datacenter).

And in each Proxmox I have some local disks with ZFS to be able to do snapshoots, but I don't do it due to some ignorance on my part (due to past experiences).

I do daily backups (in PBS) in some cases a couple of times daily, almost one always after hours, except for one VM which is backed up twice at 12:30pm and later.

My nightmare: with the backups to PBS, if I had any serious problem I will lose a lot of data in one day since I would have to restore the last backup from a previous night (In this VM I backed up its files with sync-toy, but my ERP will not know the generated files I backed up).

I would like to do more backups and more often, but I have noticed that under certain circumstances that I have not fully identified, the backup can take hours (dirty backup?/yes) and it slows down the VM until the backup finishes; I think for example:
- After a Proxmox update or reboot of (what?) NAS, PBS: the next backup on PBS can take many minutes, almost hours (full dirty backup).

 

[Dunuin]

Any stopping of VMs will reset the dirty-bitmapping which PVE uses to keep track of which data of the vitual disk has changed since the last backup or not. So if you stop the VM or reboot the server PVE will have to read and hash the complete virtual disk again.