Help Needed to Design Ideal ZFS Setup on PVE

[pvpaulo]

Good morning everyone,


I could use your help in designing the best setup for my current Proxmox environment.


Here's the current scenario:

• I'm running Proxmox VE with 4 x 1TB disks.
• I've created a ZFS pool using RAID10, resulting in a pool named VMS with ~2TB of usable space.
• I'm virtualizing a Proxmox Backup Server (PBS) inside this PVE.
• I’d like to use the VMS pool to allocate virtual disks for PBS and also for other virtual machines, such as one that provides Samba file sharing.

However, I'm facing issues:

• The PBS virtual disk stored in the VMS pool is showing errors.
• Another VM using the same pool (Samba server) is reporting I/O errors.
• The zpool status command reports the VMS pool is degraded.

Given this situation, what would be the best approach to using ZFS in this setup? Should I reconsider how I'm allocating virtual disks from the pool, or is it more likely a hardware or configuration issue with ZFS?


Any suggestions or insights would be greatly appreciated.


Thank you in advance!

 

[LnxBil]

PBS is itself also running ZFS, so ZFS-on-ZFS is often a performance killer and many people in the forum will advice against it.

Do you see the i/o errors on the pve host in zpool status?

 

[gfngfn256]

Also having PBS as a VM within Proxmox is not really a reliable backup - this, coupled with the fact that all this resides within one ZFS pool is basically a useless setup.

 

[leesteken]

• I'm running Proxmox VE with 4 x 1TB disks.
• I've created a ZFS pool using RAID10, resulting in a pool named VMS with ~2TB of usable space.

• The zpool status command reports the VMS pool is degraded.

Maybe show the output of zpool status for the VMS pool (in CODE-tags)? Are you using SMR or QLC drives? Maybe we can find out the actual problem with your physical zpool and suggest improvements?

 

[pvpaulo]

Since the environment is still being created,
I have already removed the disks and formatted them.
Now I am studying a better way to do this operation.
Since I have 4 1TB disks.
When running RAID 10, I will have 2TB of storage.

However, I want to add PBS and other services on internal PVE VMs.

I thought about creating a ZFS pool using 2 1TB disks for the local VMs.

And add the 2 1TB disks in Passthrough on the PBS VM and make the ZFS pool only inside the PBS VM.

Would this be a good approach?

 

[Curt Hall]

We had trouble with PBS until we decided to install it on another host altogether (CPU,RAM,HDD) It was nothing but trouble until it was isolated (hardware wise) from the VM's and drives said VM's are on, its backing up. You can install it in Proxmox, but just let that host be for PBS purposes only and it'll work fine.
EDIT: we have a SAN that all backups go to, so there is no local HDD contention during backups.

 

[alexskysilk]

I'm virtualizing a Proxmox Backup Server (PBS) inside this PVE.

Whats the point of doing this?

• Another VM using the same pool (Samba server) is reporting I/O errors.
• The zpool status command reports the VMS pool is degraded.

Fix your filesystem problems first. Your zfs layout isnt the issue, but the subsystem is somehow defective.

Once we past the "dont store your payload and backups on the same storage," If you really want to use PBS nested on top of pve, there's nothing stopping you using native storage from either the host (nfs, virtiofs, of bind mount container) or external storage so you dont end up with write amplification.

 

[gfngfn256]

Since the environment is still being created,

Exactly the reason why you have received the initial posts advising you of a structure change in your setup.

Think about what you expect to do if your PVE node fails & needs a rebuild/reinstall. Where will your VM backups be?

 

[leesteken]

Since I have 4 1TB disks.
When running RAID 10, I will have 2TB of storage.

Whether this is a good approach depends a on the type of drives (which you apparently don't want to share) and not the ZFS pool layout.

However, I want to add PBS and other services on internal PVE VMs.

Also having PBS as a VM within Proxmox is not really a reliable backup - this, coupled with the fact that all this resides within one ZFS pool is basically a useless setup.

Please make sure to keep additional copies of the data and at least one in another place. PBS has remote sync which is perfect for that.

I thought about creating a ZFS pool using 2 1TB disks for the local VMs.

And add the 2 1TB disks in Passthrough on the PBS VM and make the ZFS pool only inside the PBS VM.

That also possible (but please don't use SMR or QLC drives). Alternatively, install PBS as a container (instead of a VM) and you can use the PVE storage directly (and avoid ZFS on ZFS). Or install PBS without ZFS (and avoid ZFS on ZFS).

Whats the point of doing this?

I also use local PBS (as container) on each PVE for quick backups (and restore as long as the hardware does not fail) and sync between them regularly (which in in different places with a slower connection).

Fix your filesystem problems first. Your zfs layout isnt the issue, but the subsystem is somehow defective.

Since the environment is still being created,
I have already removed the disks and formatted them.

That won't fix the underlying issues and I also think you should investigate this first.