[SOLVED] Add SSD/ZFS/Pool with VMs to new Proxmox server

mrE

Renowned Member
Apr 21, 2014
26
2
68
Good day.

I am practicing with Proxmox; my interest is to get the following to be prepared:

Simulate a disaster on Proxmox (without PBS), install a new Proxmox and use the SSD/ZFS/Pool with VMs on the new server.
This, because I want to use a development Proxmox that I could take from the office to another remote location, where for some reason I would not have access to a PBS and there will be major changes to the VM.

I initially did a Promox installation (ISO), and added an SSD with ZFS (Single Disk):
ID:local-zfs
ZFS Pool: rpool/data
With a VM/windows.

Then, simulating a Proxmox damage (with reinstallation), I try to add the disk with the previous pool in this new installation to continue using the VM/windows.

Note: I did this new installation with Debian Bulls-eye; following the guide; so far so good. Except that I don't manage to add the SSD/ZFS/pool.

1665725953608.png


The disk with the previous server VM is not showing up as unused.
1665726059916.png


Thank you in advance.
 
I assume you want to add the existing ZFS pool as a storage to your new installation? You are in the Disk overview section, but in order to add it as a storage, it should be sufficient to navigate to Datacenter > Storage > Add > ZFS > *select your zfs pool* .
 
I assume you want to add the existing ZFS pool as a storage to your new installation? You are in the Disk overview section, but in order to add it as a storage, it should be sufficient to navigate to Datacenter > Storage > Add > ZFS > *select your zfs pool* .
I tried that last night (reading other posts), but no pool appears.
I remember that I even tried typing the data in the fields (in Datacenter > Storage > Add > ZFS > *select your zfs pool*) but it does not show the pool.
I'll post a screenshot when I get home.

Some time ago when I tried to do another reinstall where the contents of the pool didn't matter, I remember I had to do something to make the disks show up as unused; but I can't find my notes.
I wonder if that will need to be done (and how to do it?), but without losing the VM inside the pool.

Thanks
 
Last edited:
Did you import the pool first? zpool import to list importable pools and zpool import PoolName to import it. You might need to use the "-f" flag to force importing it in case it wasn't exported previously with zpool export PoolName .
 
Did you import the pool first? zpool import to list importable pools and zpool import PoolName to import it. You might need to use the "-f" flag to force importing it in case it wasn't exported previously with zpool export PoolName .
I haven't done that (yet).
I'm still away from home, I'll try again later as soon as I can.

Thanks for your help!
 
Did you import the pool first? zpool import to list importable pools and zpool import PoolName to import it. You might need to use the "-f" flag to force importing it in case it wasn't exported previously with zpool export PoolName .
It works!!! Thank you @Dunuin !!!:D
1665808123081.png
--
Now I need to research how to add a "new" VM (maybe with the same Id) and attach the disks (vm-100-disk-#) from the pool (next step of my simulation).:)
 
You might need to use this, so new VMs will recognize the old disks as "unused" disks which you then can add:
qm rescan [OPTIONS]
Rescan all storages and update disk sizes and unused disk images.
--dryrun <boolean> (default =0)
Do not actually write changes out to VM config(s).
--vmid <integer> (1 - N)
The (unique) ID of the VM.
 
  • Like
Reactions: mrE
You might need to use this, so new VMs will recognize the old disks as "unused" disks which you then can add:
After a few days busy with other projects, I managed to do all the suggested steps.

Dunion. Where you say that I should use "qm rescan", when I try it without creating the VM it shows me nothing: Only after creating a VM with the same ID it showed me the SSD disks: Perfect!

Sadly, with so many pauses, I forgot to take note of my VM configuration and now I can't start it; but of course, the VM does try to boot from the SSD.

2022-10-19_19-33.png 2022-10-19_19-48failure.png

Now with all your advice in hand, I will try to replicate everything again (as soon as I get some free time):
  • New Proxmox + VM (taking note of the configuration of each VM).
  • Simulate the failure/disaster in Proxmox
  • Reinstall and add the SSD/ZFS/Storage + Disks
Thank you! @shanreich and @Dunuin
 
(taking note of the configuration of each VM).
VM/LXC configs are stored in "/etc/pve/qemu-server" and "/etc/pve/lxc". You don't really have to take notes on creating VMs. Just make sure you always have a recent backup of the entire "/etc" folder.
 
  • Like
Reactions: mrE
VM/LXC configs are stored in "/etc/pve/qemu-server" and "/etc/pve/lxc". You don't really have to take notes on creating VMs. Just make sure you always have a recent backup of the entire "/etc" folder.
Good advice; I didn't know that.
And since I really simulated a total disaster (no backups) I didn't have any /etc/folder backups.

I want to simulate a total disaster without backups; of course taking note of the previous Proxmox configuration. It will be easier to copy the configurations as you mention.

But right now I take note of that advice.
 
For a desaster I like to:
1.) every several months or before upgrading PVE to a new major version I will shutdown the PVE server, boot a...
1.a) clonezilla ISO form USB stick to backup the whole PVE system disk on block level to a archive file or
1.b) Debian from USB stick with the proxmox-backup-client (PBC) and use that PBC to do a block-level backup of the full PVE system disk and store it on my proxmox-backp-server (PBS)
2.) setup a automated task using crontab that will do a daily or weekly recursive backup of the folders "/etc", "/root", "/var" and "/home". This can be done using...
2.a) a tool of your choice like cp + tar + gzip if you just want to copy the folder to a backup location or
2.b) use the PBC to backup those folders and store them on my PBS
3.) setup 2 backup job using the proxmox webUI. One for daily backups and one for weekly backups and let them backup your VMs and LXCs. You might want to keep several backups to have some versioning. For example 6 daily and 12 weekly backups. This can either be done using...
3.a) Vzdump to store the backups at some location like a USB HDD or NAS or
3.b) to a PBS (which is recommended as it is way faster and only needs a fraction of the space)
4.) setup cv4pve-autosnapshot to take hourly snapshots and prune them after 24 hours. So in case you did something wrong and realize it fast, you can just rollback a snapshot so you don't have to restore a daily backup. This is way faster and you only loose a hour of data and not a whole day.

And yes, keeping a good documentation is also a good idea. Especially when you install and setup additional packages to your PVE.
 
Last edited:
For a desaster I like to:
1.) every several months or before upgrading PVE to a new major version I will shutdown the PVE server, boot a...
1.a) clonezilla ISO form USB stick to backup the whole PVE system disk on block level to a archive file or
1.b) Debian from USB stick with the proxmox-backup-client (PBC) and use that PBC to do a block-level backup of the full PVE system disk and store it on my proxmox-backp-server (PBS)
2.) setup a automated task using crontab that will do a daily or weekly recursive backup of the folders "/etc", "/root", "/var" and "/home". This can be done using...
2.a) a tool of your choice like cp + tar + gzip if you just want to copy the folder to a backup location or
2.b) use the PBC to backup those folders and store them on my PBS
3.) setup 2 backup job using the proxmox webUI. One for daily backups and one for weekly backups and let them backup your VMs and LXCs. You might want to keep several backups to have some versioning. For example 6 daily and 12 weekly backups. This can either be done using...
3.a) Vzdump to store the backups at some location like a USB HDD or NAS or
3.b) to a PBS (which is recommended as it is way faster and only needs a fraction of the space)
4.) setup cv4pve-autosnapshot to take hourly snapshots and prune them after 24 hours. So in case you did something wrong and realize it fast, you can just rollback a snapshot so you don't have to restore a daily backup. This is way faster and you only loose a hour of data and not a whole day.

And yes, keeping a good documentation is also a good idea. Especially when you install and setup additional packages to your PVE.

Thanks for your list to prevent disasters @Dunuin

I need to read it very carefully, as there are things I don't know (even though I started reading I still don't understand how to use or install some).
  • (1b) PBC, I didn't know it or that I could use it in Promox.
  • (2b) Debian USB with PBC
  • (3b) seems to me very necessary and useful, I already use PBS for my VMs
  • (4) is totally unknown to me I have to read your link.
Since I have a PBS: (3b) is similar to (4)?

Thanks, I look forward to asking you after reading and researching your points.
 
Thanks for your list to prevent disasters @Dunuin

I need to read it very carefully, as there are things I don't know (even though I started reading I still don't understand how to use or install some).
  • (1b) PBC, I didn't know it or that I could use it in Promox.
  • (2b) Debian USB with PBC
I think you mixed 1b and 2b up.
Have look at the documentation: https://pbs.proxmox.com/docs/backup-client.html
When doing block level backups that block device shouldn't be mounted, so you shouldn't do block level backup of your PVEs system disk while PVE is booted. Thats why I said you should boot a Debian from a USB stick and install the PBC to it.
But what you can do while PVE is booted is doing file level backups to for example backup the PVE config files.
There is even an example of how to backup the PVE servers root folder: https://pbs.proxmox.com/docs/backup-client.html#creating-backups
Since I have a PBS: (3b) is similar to (4)?
Snapshots will never replace a real backup, as snapshots are stored on the same disk. If your disk fails, you lose the VMs and the snapshots of those VMs. But snapshots are nice if you want to quickly revert some changes. For example, doing a snapshot before updating a VM. If you then realize you screwed up the update or the VMs aren't able to run any longer you can easily roll back that snapshot and try it again or dn't update them at all.
Snapshots aren't great for long-term archival, as they will grow the more the older they get. But they are great as they are fast. You can snapshot or roll back a 10TB VM within a second. Doing a backup and restore might take many hours.
So it's a good idea to combine snapshots and backups. Backups for daily/weekly/monthly/annually long-term archival and snapshots for the short-term archival wen you want to be able to revert your guests in something like 1 hour of 15 minute intervals.
 
  • Like
Reactions: mrE
I think you mixed 1b and 2b up.
Have look at the documentation: https://pbs.proxmox.com/docs/backup-client.html
When doing block level backups that block device shouldn't be mounted, so you shouldn't do block level backup of your PVEs system disk while PVE is booted. Thats why I said you should boot a Debian from a USB stick and install the PBC to it.
But what you can do while PVE is booted is doing file level backups to for example backup the PVE config files.
There is even an example of how to backup the PVE servers root folder: https://pbs.proxmox.com/docs/backup-client.html#creating-backups

Snapshots will never replace a real backup, as snapshots are stored on the same disk. If your disk fails, you lose the VMs and the snapshots of those VMs. But snapshots are nice if you want to quickly revert some changes. For example, doing a snapshot before updating a VM. If you then realize you screwed up the update or the VMs aren't able to run any longer you can easily roll back that snapshot and try it again or dn't update them at all.
Snapshots aren't great for long-term archival, as they will grow the more the older they get. But they are great as they are fast. You can snapshot or roll back a 10TB VM within a second. Doing a backup and restore might take many hours.
So it's a good idea to combine snapshots and backups. Backups for daily/weekly/monthly/annually long-term archival and snapshots for the short-term archival wen you want to be able to revert your guests in something like 1 hour of 15 minute intervals.
Yes, when re-editing the text I mistakenly mixed the two. (facepalm)

Now that I re-read your comment, I understand about cv4pve-autosnapshot, it sounds very attractive.

One doubt here:
In case of having to migrate the VM (with snapshots) from server-A to server-B. what happens to the snapshots, do they stay on server-A?
I don't remember well, but I think I had problems at some point because of snapshots, so I stopped using them.

And related to snapshots (Sorry for the change of topic):
I wonder why I can't do snapshot on this VM with windows 11, all the others (Linux, CentOS, Windows 2019, 7) can do snapshot (I don't use snapshot actually).
The biggest difference I see between this VM and the others is that in W11 I added some USB ports from the host (since I program for some android devices) and to support UEFI use another BIOS.

Is it because I have shared hardware from the virtualized Host in the VM?

1666456987454.png
Thank you! @Dunuin
 
Last edited:
Yes, when re-editing the text I mistakenly mixed the two. (facepalm)

Now that I re-read your comment, I understand about cv4pve-autosnapshot, it sounds very attractive.

One doubt here:
In case of having to migrate the VM (with snapshots) from server-A to server-B. what happens to the snapshots, do they stay on server-A?
I don't remember well, but I think I had problems at some point because of snapshots, so I stopped using them.
I think that depends on the underlaying storage, as PVE won't do it's own snapshots, it will make use of the snapshots features of the LVM/ZFS/Ceph/Qcow2 storage. So snapshots will behave differently depending on the storage used.
But even if you can't migrate snapshot this wouldn't be a problem as you should have daily backups anyway.
Doing backups all 15 minutes would be way too slow and ressource intense. So the option would be no snapshots at all and only backups every day or all several hours. Or the same backups but snapshots every couple of minutes/hours in addition to that.
So using snapshots will just make it better.

And related to snapshots (Sorry for the change of topic):
I wonder why I can't do snapshot on this VM with windows 11, all the others (Linux, CentOS, Windows 2019, 7) can do snapshot (I don't use snapshot actually).
The biggest difference I see between this VM and the others is that in W11 I added some USB ports from the host (since I program for some android devices) and to support UEFI use another BIOS.

Is it because I have shared hardware from the virtualized Host in the VM?

View attachment 42487
Thank you! @Dunuin
You are only allowed to do snapshots if all storages of that guest will support snapshotting. So this won't work if you for example do a HDD passthrough to that VM.
 
  • Like
Reactions: mrE
You are only allowed to do snapshots if all storages of that guest will support snapshotting.

I am using two Synology with NFS folders (one in production, the other in standby) to store the virtual disks of the VMs (datacenter).

And in each Proxmox I have some local disks with ZFS to be able to do snapshoots, but I don't do it due to some ignorance on my part (due to past experiences).

I do daily backups (in PBS) in some cases a couple of times daily, almost one always after hours, except for one VM which is backed up twice at 12:30pm and later.

My nightmare: with the backups to PBS, if I had any serious problem I will lose a lot of data in one day since I would have to restore the last backup from a previous night (In this VM I backed up its files with sync-toy, but my ERP will not know the generated files I backed up).

I would like to do more backups and more often, but I have noticed that under certain circumstances that I have not fully identified, the backup can take hours (dirty backup?/yes) and it slows down the VM until the backup finishes; I think for example:
- After a Proxmox update or reboot of (what?) NAS, PBS: the next backup on PBS can take many minutes, almost hours (full dirty backup).
 
Last edited:
Any stopping of VMs will reset the dirty-bitmapping which PVE uses to keep track of which data of the vitual disk has changed since the last backup or not. So if you stop the VM or reboot the server PVE will have to read and hash the complete virtual disk again.
 
  • Like
Reactions: mrE
Hello! I also try this experiment but have no result.
Code:
zpool import PoolName
dont't work for me, but with the -f key i mount drive but have no data on it.
But it is not empty because there are no a liitle bit free space on it.
Is there any solution to get data? and add backups to ZFS list?
1720814921028.png
not only VM Disks or CT Volumes



1720814603005.png
1720814624418.png
1720814651837.png
1720814770898.png
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!