I suspect that you are mixing up the layers. The encryption and deduplication in Proxmox Backup Server is filesystem-agnostic so it has nothing to do with the encyption you do on the ProxmoxVE host or inside a vm and especially nothing to do with ZFS.
Let's go through some scenarios:
- You have an encrypted ZFS dataset (for a container or with some iso files) or volume (VM disc) on the Proxmox host: The backup won' backup the encrypted state, since it backups the state after you decrypted the ZFS. As explained by _gabriel qemu, ceph, lxc don't know and don'care about the encryption. For them the unlocked dataset or volume behaves like regular storage. So: As long as you use the same key for all your backups PBS deduplication will work and the backups will all be encrypted. As explained by aaron usually you will have one key one PBS configured as backup target on your ProxmoxVE single-node or cluster. If you use the same key for another cluster or single-node, the backup will still be encrypted and deduplication will still work. if however you use a different key (maybe one of your clusters is on a rented dedicated server so you don't trust it enough to use the same key) then it will still be encrypted but you will need more storage space on the datastore.
- You have encryption inside your VMs (with ZFS or Linux LUKS): The VM will still be encrypted and saved to the PBS datastore. Since you most likely won't use the same key in any VM the deduplication will only work for every backup of one VM but not of all. Let's say you have two vms with the ids 100 und 101, both with the same linux distribution and applications. Each one has their own dedicated encryption inside the vm with a different key. You do hourly backups. So deduplication will still save space (because of all the backups of one vm, only the differences needs to be saved) but not as much as you would if the backup of VM101 and VM 100 would be able to profit from already saved contents of the other vm.
- You don't use encryption on the ProxmoxVE host or inside the VMs at all but only on the backup to PBS: Deduplication will still work and backups will still be encrypted.
So in the end you need to decide which layer (ProxmoxVE host, VM/LXCs, backup) needs encryption and where you can live without it.
For example: In my homelab my servers and vms are all NOT encrypted: The worst which might happen is that some burglar would break in at night and take all my stuff with him. This would be quite annoying and expensive (due to the need to replace all my toys aka homelab hardware
but in the end I would still have my backups (more on this later) and nothing of the data is that sensible that I can't bear that some burglar goes through them and look up my questionable music/movie taste, tax records or vacation photos
And if I throw the stuff away I would have to wipe the discs non the less.
Now what's the benefit of not having encryption: No need to enter any passphrases, easier recovery in case of an error or an update gone wrong.
My notebook and backups are a different story though: I take my notebook with me all the time, if I ever loose it I don't want anybody to access the contents. So it's encrypted and I need to enter a passphrase everytime I boot the thing. And for the backups: The raw data from my notebook and NAS is backed up to a hetzner storagebox with restic. I can't trust it too much because (although they have a good track record in terms of security) Hetzner might get hacked. Thus I use restics function for encrypted backups.
Same is true for the backup of my lxcs and vms: I have a remote PBS on a cheap vserver at netcup which pulls my local backups from the PBS in my local network . I assume that it might get hacked at any time without me even noticing it. Thus I encrypt all my vm and lxcs backups so even if some attacker might be able to access the backups on my storagebox or vserver he wouldn't be able to do much with them.
Companys also need to think of such tradeoffs: For notebooks or workstations in their office encryption is a good idea since burglars might break and take all the workstations with them. In a protected, guarded datacenter you don't need to worry much about burglars but more about your employees and anybody with datacenter access can do enough damage with or without encryption. So most of the time you don't need it and can save yourself the added complexitys and issues in case of the need for recovery. The story might be different though if some regulations you are subject too (e.g. if you are a military contractor) force you to encrypt all your stuff.
So in the end it depends on your level of paranoia and which poison you want to pick
