Best choice for a datastore for PBS as a VM in proxmox

[Fra]

We are setting up an on-premises remote pbs for our proper (bare metal) pbs server in a DataCenter: we opted for a pbs in a VM in proxmox.

Since pbs is a VM, for the datastore I guess we can just choose any standard Unix file system (ext4, xfs or zfs) in one RAW or even QEMU VM disk (or even more then one disk).

We arrived to the conclusione that ZFS on QEMU does not make sense, did we get it right?

We experimented one RAW disk with zfs and we've got this (connection is fast since is just a local test)

# proxmox-backup-client benchmark
Uploaded 142 chunks in 5 seconds.
Time per request: 36857 microseconds.
TLS speed: 113.80 MB/s
SHA256 speed: 371.45 MB/s
Compression speed: 562.08 MB/s
Decompress speed: 1076.48 MB/s
AES256/GCM speed: 1082.12 MB/s
Verify speed: 276.43 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 113.80 MB/s (9%)   │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 371.45 MB/s (18%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 562.08 MB/s (75%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 1076.48 MB/s (90%) │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 276.43 MB/s (36%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 1082.12 MB/s (30%) │
└───────────────────────────────────┴────────────────────┘

but to be honest we were really just playing, and now we need a reliable local pbs remote.

(yes, we are talking about a pbs, in a proxmox, to be a remote of a pbs that backups other proxmox)


Maybe it adds nothing, but the proxmox host, that will host the pbs VM, has hardware RAID: does this means we should not use ZFS in a VM disk?

 

[oversite]

We are setting up an on-premises remote pbs for our proper (bare metal) pbs server in a DataCenter: we opted for a pbs in a VM in proxmox.

Since pbs is a VM, for the datastore I guess we can just choose any standard Unix file system (ext4, xfs or zfs) in one RAW or even QEMU VM disk (or even more then one disk).

We arrived to the conclusione that ZFS on QEMU does not make sense, did we get it right?

We experimented one RAW disk with zfs and we've got this (connection is fast since is just a local test)

# proxmox-backup-client benchmark
Uploaded 142 chunks in 5 seconds.
Time per request: 36857 microseconds.
TLS speed: 113.80 MB/s
SHA256 speed: 371.45 MB/s
Compression speed: 562.08 MB/s
Decompress speed: 1076.48 MB/s
AES256/GCM speed: 1082.12 MB/s
Verify speed: 276.43 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 113.80 MB/s (9%)   │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 371.45 MB/s (18%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 562.08 MB/s (75%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 1076.48 MB/s (90%) │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 276.43 MB/s (36%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 1082.12 MB/s (30%) │
└───────────────────────────────────┴────────────────────┘

but to be honest we were really just playing, and now we need a reliable local pbs remote.

(yes, we are talking about a pbs, in a proxmox, to be a remote of a pbs that backups other proxmox)


Maybe it adds nothing, but the proxmox host, that will host the pbs VM, has hardware RAID: does this means we should not use ZFS in a VM disk?

Normally the VMs virtual disks would already be redundant by ceph or zfs (or hw raid in your case) on the host meaning zfs does not contribute much, and if underlaying fs is zfs, well nested zfs is hard to recommend. However, if you do passthru of physical disks to your PBS VM, zfs is a great choise. If you use virtual disks, any filesystem of your choise like ext4 or xfs is great as you already thought and i don't see the benefit of zfs in that case.

 

[Fra]

thanks.
Yes, no passthru.
So we'll opt for ext4 (or xfs), and forget zfs.

What about RAW vs QEMU?

~~~

Another choice, as a datastore, for us would be mounting NFS (a simple Qnap raid 5 storage: not that fast, but reliable): in that case, if we understood correctly, we can just mount the NFS inside the PBS VM, can't we?

 

[oversite]

thanks.
Yes, no passthru.
So we'll opt for ext4 (or xfs), and forget zfs.

What about RAW vs QEMU?

~~~

Another choice, as a datastore, for us would be mounting NFS (a simple Qnap raid 5 storage: not that fast, but reliable): in that case, if we understood correctly, we can just mount the NFS inside the PBS VM, can't we?

I am mostly using CEPH so i cannot tell, but for anything that needs disk i/o i prefer RAW before QEMU img, but i cannot tell really.
NFS would work great as you say mounted within the VM, it's a great redundancy to have PBS remote VM with NFS, you can then restore to PVE even if PBS hw is totally down.

 

[floh8]

zfs is safe inside vms. ->https://serverfault.com/questions/985252/zfs-inside-a-virtual-machine
the question is: is it necessary? do you need snapshot, compression..? no. Maybe u wanne use the Dedupliction Feature, so u can save much more space in connection with the proxmox deduplication. If u wanne more speed than zfs u can benchmark btrfs.
NFS inside VM is no probleme, of course, but proxmox give always the hint to use local storage and u have no extra deduplication, because qnap without zfs not support this.
if u prefere the qnap solution then u can also connect via iscsi and use zfs/btrfs.

 

[Fra]

uh, sorry, in the above I meant "RAW vs Qcow2", and not "RAW vs QEMU"? (I do not edit my question so to leave the original context)

 

[oversite]

uh, sorry, in the above I meant "RAW vs Qcow2", and not "RAW vs QEMU"? (I do not edit my question so to leave the original context)

no worries, i think everybody understood that since QCOW2 is the native QEMU-image (out of 9 types). I would like to emphasize that ext4 on qcow2 works very good. However, my eperience is that QCOW2 is a bit slow if the underlaying transport is NFS,

About the test above, i believe you need to do a real world test to see how it performs, i don't think that benchmark measures the disk speed. Spin up a PBS VM and do some backups to get some real numbers, if NFS that's not relevant since the backup data will not be on the VM disk anyway.

What have you concluded so far, where are you with the virtual PBS?

 

[oversite]

zfs is safe inside vms. ->https://serverfault.com/questions/985252/zfs-inside-a-virtual-machine
the question is: is it necessary? do you need snapshot, compression..? no. Maybe u wanne use the Dedupliction Feature, so u can save much more space in connection with the proxmox deduplication. If u wanne more speed than zfs u can benchmark btrfs.
NFS inside VM is no probleme, of course, but proxmox give always the hint to use local storage and u have no extra deduplication, because qnap without zfs not support this.
if u prefere the qnap solution then u can also connect via iscsi and use zfs/btrfs.

If the VM is a remote syncing the physical PBS the dedup dpes not matter much since the chunks will be deduplicated already.

 

[floh8]

If the VM is a remote syncing the physical PBS the dedup dpes not matter much since the chunks will be deduplicated already.

dont confuse with dedup of the datastore and dedup of the filesystem. these are 2 different things

 

[Fra]

Another wrong term I used is pbs "remote": indeed, I understood now (that I set it up) that the 'remote' is the pbs I want to rsync into another pbs, and in this thread I was asking info about setting this other pbs, and not the remote, sorry.

So, with your advice, we end up with solution:

* ordinary pbs, in bare-metal (the so called "remote"), without RAID and with ZFS, that backup bare-metal proxmoxs in the internet, and on-premises.
* pbs installed in a VM inside the proxmox on-premises (with hardware RAID on SaS hds) and , that rsync data from the pbs-remote: the data-store is NFS (ext4) mounted inside the VM, and is a QNAP (connected directly to the NIC of the proxmox host, that host this pbs-VM).

For this pbs-VM we see it takes all RAM we give it (20GiB), and 50% of 3 cpu (but only at night when it rsync)

We are happy so far; I don't know how to benchmark this pbs-VM since it's on-premises, so the internet connection for the rsync with the remote has that bottleneck (about 50Mb/s): the remote has 2.5T that we are taking home a little bit every night (we kill the rsync every morning): this weekend we should be able to conclude the rsync and have all data from the remote on-premises in this pbs-VM.

 

[Fra]

floh8 said:​

​

> dont confuse with dedup of the datastore and dedup of the filesystem. these are 2 different things

ah, good point; for "dedup of the datastore" you mean how the pbs software dedup the data, don't you?

so:

* If I have 1T of data saved in the datastore in a bare-metal pbs with ZFS (the "remote"), when I rsync that pbs into another pbs with a different datastore (ext4) should I expect a change in the datastore size?

 

[Fra]

floh8 said:​

​

> dont confuse with dedup of the datastore and dedup of the filesystem. these are 2 different things

ah, good point; for "dedup of the datastore" you mean how the pbs software dedup the data, don't you?

so:

* If I have 1T of data saved in the datastore in a bare-metal pbs with ZFS (the "remote"), when I rsync that pbs into another pbs with a different datastore (ext4) should I expect a change in the datastore size?

I answer to myself:

the two proxmox use about the same space.

The above have been in production for 3 months now: everything is fine: the bandwidth to sync the remote is really low (super pbs!)

thanks to all of you for your help and to Proxmox developer of course!