Push sync with push side encryption, must every chunk be re-read and hashed each run?

[Imitate7971]

I am seeing a lot of read on the source, multiple times the overall data size already and a little CPU load ~10%. But nearly no network traffic anymore. First i thought the sync might hang so i restarted it but it ended in the same way. No i guess it is the encryption on the fly, see below.

Setup

• Source PBS 4.2.7 (kernel 7.0.14-20-pve), unencrypted local datastore
• Target PBS 4.2.8, untrusted offsite, connected via WireGuard
• Sync job in push direction with active-encryption-key
• ~2 TB physical datastore, daily snapshots, mostly CTs
• (split archives .mpxar/.ppxar)

Observation

• Initial push transferred ~1.5 TB normally.
• Since then: source reads ~45-50 MB/s from disk with constant CPU load, but network traffic is close to zero.
• Target log for the current snapshot shows the previous snapshot is found and its chunks registered as known:
  • download 'index.json.blob' from previous backup 'ct/107/2026-09-30T10:00:26Z'
  • register chunks in 'root.ppxar.didx' from previous backup ...
  • created new dynamic index 1 ("ct/107/2026-10-01T10:00:21Z/root.ppxar.didx")
• Source journal for that single snapshot: processed 1.01 TiB in 5h 41m, uploaded 481.474 MiB

My theory

The docs say encrypted chunk digests are computed from plaintext + key.
The source datastore only holds plaintext digests, so to find out
whether a chunk is already known on the target, the source must read
and hash every chunk of every snapshot. Only new chunks are then
encrypted and uploaded. So an encrypted push costs a full local read of
each snapshot's logical size, even if almost nothing changed.

Questions
1. Is this correct and expected behaviour for push with
active-encryption-key?
2. Is there any mapping/cache (plaintext digest -> encrypted digest)
to avoid re-reading known chunks, or is one planned?

 

[Chris]

Questions
1. Is this correct and expected behaviour for push with
active-encryption-key?

Yes, each chunk must be read and encrypted first, only then the chunk digests can be compared. This step would otherwise be performed by the client if client side encryption is being used. Chunks are however only read and encrypted once, subsequent identical chunks for the same push sync do not need to redo the same operation. So if the goal is to reduce the load on the PBS server, encryption can be setup on the client directly, with all it's implications.

2. Is there any mapping/cache (plaintext digest -> encrypted digest)
to avoid re-reading known chunks, or is one planned?

Yes, there is such a mapping in place. As mentioned it is however only valid during one push sync and kept fully in-memory. In principle it would be possible to persist such a mapping on disk and consult it for plain -> encrypted digest mapping, that has however several security implications.

 

[Imitate7971]

With client you mean the PVE host? I guess that would be the only way then to avoid rereading everything for each sync job. In that case I could also use pull instead of push when the backups are already encrypted on the main PBS.