Setup Suggestions

[HamsDa]

Hello dear Proxmox Community and merry Christmas to everyone!

I am fairly new to Proxmox, only experimented with it inside another Hypervisor.
I got myself an Intel NUC10i7 (6 cores + hyperthreading, 64 GB RAM and 2x 2TB SSD [1x SATA, 1x M.2]), which I want to be a Proxmox VE Hypervisor.

I hope I can count on some more experienced Proxmox Users for some tipps and tricks regarding a good Proxmox Setup.

What I want to achieve with Proxmox

• Firewall-VM (pfSense)
• Various Ubuntu VMs / Docker Containers for various tasks
• Backup Routine (I'm thinking Proxmox Backup Server VM inside Proxmox with an external Hard Drive mounted in the VM)
• Possibly nested HyperVisor (mainly ESXi, a little Hyper-V maybe, both for trying out stuff [no production use])

But I do have a few questions about how to start this the right way, as I do not really want to do all of this again in a month just because I learned a better way of doing it

File Systems
Is ZFS a wise decision for PVE root filesystem or are there any known ramifications that I am not aware of?
I like the idea of having RAID-Controller, Logical Volume Manager and Filesystem combined.
Also, ZFS can send snapshots to other ZFS storage units, so if I take my USB HDD, mount it inside Proxmox Backup Server and make it a ZFS Pool, Backups should be fast and non-disruptive, right?

Proxmox Backup Server
I recently noticed PBS going from Beta to Stable and I just can't wait to give it a shot.
Again, here comes the question about ZFS root filesystem, especially because of sending snapshots etc.

Is a file level restore with PBS possible or is it VM restore only?
I read in some thread via google a few months ago that this feature may be on the way.

Storage Units
As I have 2x 2TB SSDs, I'd like to combine these 2 into a single 4 TB Logical Volume.
Is this a good Idea or would it be better to separate Storage Units?
I will be using these SSDs for VMs, ISOs and about 800 GB worth of Data.

How would I go about storing these 800 GB (photos, music, documents, ...)?
What if multiple VMs needed access to these at the same time?
Should I create a vDisk and mount it inside multiple VMs?

Thanks for taking the time to read and help

Greetings,
HamsDa

 

[apoc]

Is this a good Idea or would it be better to separate Storage Units?
I will be using these SSDs for VMs, ISOs and about
800 GB worth of Data.

If you combine elements they become a failure domain. If one faisl, all the data will be gone. I wouldn't go down that path.
Either do a mirror (cutting your capacity in half) or use the disks individually. If one device goes at least half the storage is unaffected.
In the end the question is: what is your data worth to you?

Also, ZFS can send snapshots to other ZFS storage units, so if I take my USB HDD, mount it inside Proxmox Backup Server and make it a ZFS Pool,

What's the point in doing it this way?
You can do a ZFS-send / receive to a local pool. No need to have a VM and mount things.
If you really plan to implement safe yourself a VM and do it on the PVE itself.

What if multiple VMs needed access to these at the same time?

You need to create a share. This typically is NFS or SMB/CIFS.

Should I create a vDisk and mount it inside multiple VMs?

That's not going to work unless you have a cluster filesystem which is complex to implement.
Go for a "share"-approach as described above.
Install a Linux server as a Samba server, use Freenas - there are plenty options out there.

 

[Dunuin]

• Backup Routine (I'm thinking Proxmox Backup Server VM inside Proxmox with an external Hard Drive mounted in the VM)

I don't see the point. The idea is that PBS should be run on another computer so if the Proxmox Hypervisor dies you keep an working PBS to restore your VMs to a new Proxmox. If you just run PBS as a VM on the Proxmox host itself and Proxmox crashes, your PBS is also down and you can't restore your VMs to a new Proxmox.
If you just want to store some Backups use the buidlin backup function and add a usb HDD as a vzdump storage so your backups are directly stored on that usb HDD and you don't need a running PBS to restore them.

 

[HamsDa]

Hi tburger and Dunuin, thank you for helping me!

After reading your posts, I guess I really jumped on the wrong train of thought there.
Especially did I not think about failure domains, thanks for that hint!

Regarding your advice, what about this:

• M.2 SSD => ZFS Root pool => VMs, ISOs, etc.
• SATA SSD => ZFS Data pool

This way I could create a vDisk of the appropriate size for a Samba VM, which would be hosting and SMB-Sharing my data.
Also, I'll definitely get a new USB3 HDD to plug into the PVE host for storing ZFS snapshots, while all my file-level backup (my data and whatever else may come) will be saved to my raspberry pi.
If I didn't get this wrong, with this setup I'll be able to fast-restore complete VMs and vDisks, while also being able to restore individual files (though at a slower speed).

How does the integrated backup solution work?
Does it always do incremental backups or does it know that It runs on ZFS and will be using snapshots anyway?

 

[Dunuin]

Regarding your advice, what about this:

• M.2 SSD => ZFS Root pool => VMs, ISOs, etc.
• SATA SSD => ZFS Data pool

This way I could create a vDisk of the appropriate size for a Samba VM, which would be hosting and SMB-Sharing my data.

No, you don't need a vHDD or a SMB-VM. Virtualization isn't good because you loose performance and capacity due to overhead. Install a samba/NFS server (CLI and manual setup needed) directly to your proxmox host so your host itself works like a NAS. That way all VMs and other computers in your LAN can use the SMB/NFS shares and access the data on that SSD. Share the data directly stored on a dataset on that ZFS pool. If you want to use LXCs, you can bind-mount a folder of that dataset into a LXCs too.
You could use that M.2 SSD for the system and vHDDs of your VMs and you could create a dataset on that SATA SSD and share that using SMB/NFS.

Also, I'll definitely get a new USB3 HDD to plug into the PVE host for storing ZFS snapshots, while all my file-level backup (my data and whatever else may come) will be saved to my raspberry pi.

You can't store snapshots on another drive (except for the RAM dumps if you enable that). They are always on the same drive you snapshot. I've got VMs where the snapshots are doubling or trippleing the size of my actual data on it. So keep in mind to save enough space for snapshots.
Also a ZFS pool should never be filled up more then 80%. So if you got 2TB you shouldn't use more then 1.6TB. And if you want to use snapshots its a good idea to save half of the space for snapshots. So you only got like 0.8TB for your vHDDs.

If I didn't get this wrong, with this setup I'll be able to fast-restore complete VMs and vDisks, while also being able to restore individual files (though at a slower speed).

How does the integrated backup solution work?
Does it always do incremental backups or does it know that It runs on ZFS and will be using snapshots anyway?

Backups aren't incremental. The complete VM will be compressed and saved as a single file. So if you want to backup 0.8TB of VMs, you get hundereds of GB of Backup archives each time. But these can be stored on an external storage like a Raspi with USB HDD acting as a NAS if you don't want to attach a USB HDD directly to your Proxmox host.

 

[apoc]

Regarding your advice, what about this:

• M.2 SSD => ZFS Root pool => VMs, ISOs, etc.
• SATA SSD => ZFS Data pool

I personally wouldn't do it at all.
I wouldn't gamble with my data and/or VMs in that way.
I'd go for a single zpool with a mirrored vdev
But that's my opinion.

No, you don't need a vHDD or a SMB-VM. Virtualization isn't good because you loose performance and capacity due to overhead. Install a samba/NFS server (CLI and manual setup needed) directly to your proxmox host so your host itself works like a NAS.

Again my opinion: don't polute/abuse the virtualization lyer for additional services.

You can't store snapshots on another drive. They are always on the same drive you snapshot.

Kind of, you can. Use zfs-send/receive to move the dataset/snapshot to another pool. Then delete the snap on the source pool.
These snapshots also can be stored incremental at least from what I read.
But you need ZFS on both ends...

 

[Dunuin]

I personally wouldn't do it at all.
I wouldn't gamble with my data and/or VMs in that way.
I'd go for a single zpool with a mirrored vdev
But that's my opinion.

Me too.

Kind of, you can. Use zfs-send/receive to move the dataset/snapshot to another pool. Then delete the snap on the source pool.
These snapshots also can be stored incremental at least from what I read.
But you need ZFS on both ends...

I'm replicating datasets and snapshots between two FreeNAS server. Sometimes if you delete a snapshot on your source you can't do incremental replication anymore because source and target get out of sync. Had that several times where I was forced to replicate the complete 32TB pool, because my snapshot task deleted some old snapshots.

 

[HamsDa]

I personally wouldn't do it at all.
I wouldn't gamble with my data and/or VMs in that way.
I'd go for a single zpool with a mirrored vdev
But that's my opinion.

I have absolutely no problem with losing all my VMs, the Hypervisor and all of the configuration.
I'm trying to use ansible whenever possible, so it wouldn't be that hard to recover from losing everything.
What I absolutely cannot lose is my data (and it may be the best way to RSYNC it to my Raspberry Pi).

But just out of curiosity I wanted to try out Proxmox Backup capabilities.
I wanted to see if there is any way of replicating something like NAKIVO Backup & Replication (we use this to back up our ESXi Hypervisor in my company).
NAKIVO takes a snapshot and creates differential backups for every VM, including the ability to mount the backup vmdisk and restore a single file if need be and features deduplication too.

But Proxmox only takes full backups, so if I may ask:
How does one properly Backup VMs and whatever else may be running on the Proxmox VE Host including the vmdisks on a regular basis in an automated way that does not eat disk space for breakfast?

 

[Dunuin]

It think I've read that they are working on incremental BPS backups. But then again, running PBS as a guest on the same host you want to backup your VMs isn't a good idea.

 

[HamsDa]

It think I've read that they are working on incremental BPS backups. But then again, running PBS as a guest on the same host you want to backup your VMs isn't a good idea.

To be honest, that is exactly what we are doing in our company.
We're a small company and our customers are small companies too.
I only got into this company this year, but I've never seen any of our customers having 2 servers.
All of them just have 1 server where the Backup-VM is running and are saving the backups to USB HDDs.
But that's okay, because the Backup-Software cannot backup ESXi itself (all of our customers are VMware-Users), so in case of a catastrophe I need to reinstall ESXi + the NAKIVO Backup VM anyway.

Would it be a problem to run PBS like this?
Let's say somehow I need to completely reinstall PVE and all of the configs and VMs have been lost.
Is it possible to just install PVE, then PBS (as VM), mount the Backup-USB HDD inside PBS and restore all the VMs from there to the newly installed PVE?

If the above is possible, what would be the benefit of having PBS on a different server?

 

[Dunuin]

Is it possible to just install PVE, then PBS (as VM), mount the Backup-USB HDD inside PBS and restore all the VMs from there to the newly installed PVE?

That could be possible. The question is, do you plan to keep that USB-HDD connected all the time or do you just connect it for each backup?

If the above is possible, what would be the benefit of having PBS on a different server?

Its never a good idea to store your backups on the same system. Atleast if you don't got an additional offsite backup. There are enough cases where your backup HDDs will be lost too, if your main drives die. Fire, lightningstrike, water damage, failing PSU killing everything connected, kernel error/malware/defect Hardware like misfunctioning Non-ECC-RAM corrupting everything connected, theft, ...

 

[HamsDa]

That could be possible. The question is, do you plan to keep that USB-HDD connected all the time or do you just connect it for each backup?

I would keep that USB HDD connected at all times, to automate the VM backups.

Its never a good idea to store your backups on the same system. Atleast if you don't got an additional offsite backup.

It wouldn't be nice if my USB HDD die by lightning strike or fire, but this drive would only backup VMs for fast-restore if I mess something up while meddling with the system.

For the data backup, I would use my Raspberry Pi 3 via SMB + rsync.
I know, it is no offsite backup.
I guess I will have to look up some way of storing my data online and encrypted in a convenient and automated way.
Thanks for bringing that to my mind!