Rebuild server preparations

[IroesStrongarm]

When I built my primary Proxmox server I didn't even realize that I could install the boot OS as a ZFS pool. Currently there's nothing wrong with my server but we all know how failures occur. I want to proactively solve this problem. After much research it certainly seems the smartest move is to reinstall Proxmox from scratch. Nothing about the main hardware in my server will change except for new SSDs that will be installed for the OS.

1. So far I have made a fresh backup of all my VMs to a NAS.
2. All my VMs live on a separate ZFS pool
3. I plan to use clonezilla to make an image of the existing OS just in case, and then disconnect the main drive as I will use the opportunity to size up from 250GB to 500GB OS.
4. Note that my server has a few VMs that get different hardware passed through. Two get GPUs. One has an SSD attached, and another has an HBA.

After reinstalling Proxmox, I just update my configs for iommu flags, vfio, and driver blocks and I should be good to go? From there, would it be better to import the old VM pool or should I create the pool as new? I assume if I import it's not like my VMs will magically show up?

Finally, assuming a rebuild and then restore of VMs from the backup, any reason all the existing hardware passthroughs won't just work as expected? They should all be added to the configs from the backup automatically, yes?

Sorry for some dumb questions that even I believe I know the answers to. I just want to be sure before I destroy a perfectly working server. Thanks.

 

[Dunuin]

I assume if I import it's not like my VMs will magically show up?

You would need to add a new storage with the same name first, pointing to the same Pool. So best you copy it from a backup of your /etc/pve/storage.cfg. And you won't see any guests without copying over the VM/LXC config files from /etc/pve/lxc and /etc/pve/qemu-server.
Also important is the /etc/pve/firewall folder if you don't want to lose all your firewall rules, security groups, IP sets, aliases and so on.
Your network config is stored in /etc/network/interfaces, /etc/hosts and /etc/resolv.conf.

 

[IroesStrongarm]

You would need to add a new storage with the same name first, pointing to the same Pool. So best you copy it from a backup of your /etc/pve/storage.cfg. And you won't see any guests without copying over the VM/LXC config files from /etc/pve/lxc and /etc/pve/qemu-server.
Also important is the /etc/pve/firewall folder if you don't want to lose all your firewall rules, security groups, IP sets, aliases and so on.
Your network config is stored in /etc/network/interfaces, /etc/hosts and /etc/resolv.conf.

So if I import the existing pool and give it the same name, then if were to backup the configs from those locations, and just restore then in the new build, all would just "magically" show up and be ready to go?

 

[Dunuin]

So if I import the existing pool and give it the same name, then if were to backup the configs from those locations, and just restore then in the new build, all would just "magically" show up and be ready to go?

Yes, but I still would make sure to stop all guests and then back them up. And you should backup the whole /etc folder...I do that once per week via cron.

 

[IroesStrongarm]

Yes, but I still would make sure to stop all guests and then back them up. And you should backup the whole /etc folder...I do that once per week via cron.

Thanks for that. I think I'll give that a shot. And yes, I made fresh backups of all VMs this morning in preparation. Usually I do fully stop for the backups, but this time did it while they were live. Any reason I should redo them when fully stopped?

Is it worth it to backup the whole /etc folder and then just straight restore it for a new build or would there likely be some strange behavior? As mentioned, only thing changing is that the OS drive will be on a mirrored pool.

 

[Dunuin]

Thanks for that. I think I'll give that a shot. And yes, I made fresh backups of all VMs this morning in preparation. Usually I do fully stop for the backups, but this time did it while they were live. Any reason I should redo them when fully stopped?

Restoring a VM from a snapshot backup is like rebooting a machine after a crash. All running programs crashed with everything lost in RAM. Therefore its better to do stop mode backups, as all software will be backuped in a properly defined not running state. Really depends on how well your services are programmed, so you lose as less data as possible with a easy recovery from such a crashed state. In case all VMs use the qemu guest agent PVE will use fsfreeze to make sure caches are flushed before doing the backup to ensure data integrity. When not using the QEMU guest agent you might lose data in the write cache.

Is it worth it to backup the whole /etc folder and then just straight restore it for a new build or would there likely be some strange behavior? As mentioned, only thing changing is that the OS drive will be on a mirrored pool.

You can't restore the whole /etc folder. Then your new host won't work anymore. You need to only copy the files (or sometimes even only specific lines of a file) that you really want/need to preserve.

 

[IroesStrongarm]

Hey, thanks again for all the tips and help. All my VMs use the QEMU guest-agent. In that case should I not worry about redoing those backups, or would you still recommend it?

And thanks for the info about /etc as well. I know a decent amount, but I'd say I'm at the middle point where I know enough to cause lots of damage to my system, but not enough to avoid doing those, haha.

 

[Dunuin]

Hey, thanks again for all the tips and help. All my VMs use the QEMU guest-agent. In that case should I not worry about redoing those backups, or would you still recommend it?

I personally would still redo it. I like my programs not to crash.
For daily backups snapshot mode is fine. But if I know I might need to restore a guest (migrations, upgrades, ... ) I always do stop mode backups. So that I know that the guests are restored in exaclty the state they were when I stopped them for the backup.

 

[IroesStrongarm]

I personally would still redo it. I like my programs not to crash

This is incredibly fair. Haha. Thanks again for all your help. Still nervous about the rebuild as they never go as smooth as you'd like but hopefully I'm well prepared for it.

 

[IroesStrongarm]

Hey, one last question. After restoring the configs in /etc/pve/qemu-server is it just a simple reboot and all VMs should show up?

 

[Dunuin]

This is incredibly fair. Haha. Thanks again for all your help. Still nervous about the rebuild as they never go as smooth as you'd like but hopefully I'm well prepared for it.

As long as you backuped everything and documented everything you did in the past, it can't get that worse. Then downtime is the biggest problem in case something went wrong.
I for example run a DokuWiki in a LXC where I write down everything I do with the hosts or guests. DokuWiki stores all documentations in plain txt files, so it can easily be backed up and you can even read the documentations from a backup without needing to start that LXC.
Great if you ever need to redo anything but you can'T remember what commands you used to do it.
Posting your stuff here in the forum for other people is also nice. So often, I asked google how to fix my problem and the first result was pointing to this forum with a post by me, telling myself how to fix it.

 

[IroesStrongarm]

So often, I asked google how to fix my problem and the first result was pointing to this forum with a post by me, telling myself how to fix it.

Haha. I always make it a point when I ask questions on forums to go back and edit in the solution for both myself and any future passersby.

Not a bad idea to keep a record of what you've done. I actually make a text document earlier for myself documenting the main changes I would need to make to get my iommu and vfio flags setup and ready to go.

 

[IroesStrongarm]

Sorry to ask one more question, but I've got a guy on another forum claiming that even with a mirrored ZFS pool, if a drive fails, the OS won't boot "Cause the bootloader, initramfs etc. is still installed on the master outside of ZFS. You surely end up reinstalling the whole os."

This doesn't seem right to me, but again I default to many people knowing much more than me. Thoughts?

 

[Dunuin]

Sorry to ask one more question, but I've got a guy on another forum claiming that even with a mirrored ZFS pool, if a drive fails, the OS won't boot "Cause the bootloader, initramfs etc. is still installed on the master outside of ZFS. You surely end up reinstalling the whole os."

This doesn't seem right to me, but again I default to many people knowing much more than me. Thoughts?

The grub/ESP is outside of ZFS and not mirrored. Thats you you write the grub bootloader to both disks or use the proxmox-boot-tool to sync your ESP between both disks. So yes, you can lose either of the disks and your PVE will still boot, as long as you tell your BIOS/UEFI to try to boot from both disks:
https://pve.proxmox.com/wiki/Host_Bootloader#sysboot_proxmox_boot_tool

In setups with redundancy all disks are partitioned with an ESP, by the installer. This ensures the system boots even if the first boot device fails or if the BIOS can only boot from a particular disk.

 

[IroesStrongarm]

The grub/ESP is outside of ZFS and not mirrored. Thats you you write the grub bootloader to both disks or use the proxmox-boot-tool to sync your ESP between both disks. So yes, you can lose either of the disks and your PVE will still boot, as long as you tell your BIOS/UEFI to try to boot from both disks.

The first part you reference, is that all automated by having the installation create a pool for the OS drives?

After that is it just as simple as telling the BIOS/UEFI to have the two drives at the top of the boot order?

 

[Dunuin]

The first part you reference, is that all automated by having the installation create a pool for the OS drives?

After that is it just as simple as telling the BIOS/UEFI to have the two drives at the top of the boot order?

Jup. But when replacing a failed disk you will have to follow the steps in "Changing a fail bootable device" in order to partition the new disk and sync the ESP again: https://pve.proxmox.com/wiki/ZFS_on_Linux#_zfs_administration
Otherwise you would end up with just a single disk that could boot.

 

[IroesStrongarm]

Thank you for all of this information. I'm currently reading the documentation you linked as we speak. This is all very helpful and just what I was looking for. Thank you!!

 

[IroesStrongarm]

Yes, but I still would make sure to stop all guests and then back them up. And you should backup the whole /etc folder...I do that once per week via cron.

Hey, sorry to bother you again, but any chance you'd be willing to share your cronjob code so that I can reference it to make similar backups myself?

 

[Dunuin]

I'm sending my config backups to my PBS using the proxmox-backup-client. Are you running a PBS? Otherwise that won't help you.
If you are not running a PBS you could for example use tar + gz to create a named backup archive of your "/etc" folder and store that on a mounted NAS or backup disk.

 

[IroesStrongarm]

I'm sending my config backups to my PBS using the proxmox-backup-client. Are you running a PBS? Otherwise that won't help you.
If you are not running a PBS you could for example use tar + gz to create a named backup archive of your "/etc" folder and store that on a mounted NAS or backup disk.

No, not currently. I've debated making a PBS but haven't bothered to as of yet. Mainly was looking for the code needed to accomplish the task and send to a NAS, but obviously that was me assuming that was how you were doing it too. Thanks for taking the time to respond again.