SETUP
- 2 nodes, PVE 9.2.3 (kernel 7.0.14-4-pve, EFI). No shared storage.
- Cluster history: originally built on PVE 7, upgraded through to PVE 9
(so some storage naming/layout predates current versions).
- Each node has its own local ZFS RAID10 pool (4x mirror, WD SN640 NVMe).
- Dedicated migration net: isolated, non-routed, 100 GbE (ConnectX-5), MTU 9000.
- datacenter.cfg: migration: type=insecure,network=<isolated /24>
STORAGE LAYOUT (the core issue)
Each pool is named after its node and split into datasets by volblocksize,
each exposed as its own PVE storage:
PRX01: PRX01-ZFSRaid10/BlockSize32K (32K), /BlockSize64K (64K)
PRX02: PRX02-ZFSRaid10/BlockSize32K (32K), /BlockSize64K (64K)
PVE storage IDs (ZFS, node-restricted):
BS16K/BS32K/BS64K-PRX01ZFSRaid10 (PRX01 only)
BS16K/BS32K/BS64K-PRX02ZFSRaid10 (PRX02 only)
BS512-PRX02ZFS10 (PRX02 only) [benchmarking only]
Structure is symmetric and blocksizes match tier-for-tier, BUT the pool names
differ (PRX01-ZFSRaid10 vs PRX02-ZFSRaid10) and every storage ID is node-scoped.
WORKLOAD (why the tiering exists)
Mixed VMs on both nodes: Windows/Debian/Ubuntu; file servers and SQL/DB servers.
File servers split into large-sequential-video vs. thousands-of-small-files.
Disks are placed on the blocksize tier that suits each pattern — I'd rather not
collapse to a single blocksize.
WHAT I WANT
Faster moves for large VMs. Live migration works but sits at ~145 MB/s.
WHAT'S BLOCKED / WHAT WORKS
1) Offline migration: fails — source storage IDs (BS*-PRX02...) don't exist on
PRX01, so target storage is "not available".
2) Replication (pvesr): needs matching storage ID + dataset path on both nodes;
node-scoped IDs and differing pool names mean no matching target.
3) Works today: live migration with per-disk remap
(BS64K-PRX02 -> BS64K-PRX01, BS32K-PRX02 -> BS32K-PRX01). This is the
~145 MB/s drive-mirror path but preserves tiers.
4) Considering: PBS backup/restore, then move the large 64K disk to its tier
manually.
ALREADY RULED OUT (why 145 MB/s isn't network/disk/CPU/crypto)
Instrumented a full live migration:
- Net ~1.1% ifutil; iperf3 single ~6.9 / 8-parallel ~44 Gbit, ~0 retransmits.
- Jumbo verified (ping -M do -s 8972, 0% loss).
- Both pools idle w/ headroom (zpool iostat), dest shows normal txg sawtooth.
- CPU ~84% idle; top -H shows only guest vCPU threads busy, no pinned migration thread.
- Encryption off (type=insecure).
=> ~145 MB/s is the inherent latency-bound QEMU drive-mirror loop. Acceptable,
but I want delta-only moves for big VMs.
QUESTIONS
1) With node-local pools whose names differ per node, what's the recommended way
to enable pvesr replication? Does it require identical storage IDs AND identical
dataset paths on both nodes?
2) Is the fix to define ONE shared storage ID per tier (both nodes) resolving to
each node's local dataset? If so, how do people handle the differing pool names
— rename a pool, or can the "pool" field differ per node under one ID?
3) Is preserving multiple blocksize tiers across replication sensible, or do people
collapse to one pool/blocksize per node?
4) For a mixed-disk VM (some 32K, some 64K), can each disk keep its matching-blocksize
target, or must all disks land in one storage?
5) Is replication pre-seed the only real lever here, or is there a supported knob
I'm missing for the drive-mirror ceiling?
6) PBS restore: can I place each disk on a chosen target storage on restore, or does
it restore to one storage and I move disk-by-disk afterward? Does stopped-VM
"Move disk" beat the live mirror path or use the same mechanism?
7) Any cases where one method is clearly better given the workload mix — e.g. offline
backup/restore for small-files servers (per-op latency hurts), live mirror fine for
big-video servers?
sidenote: This system was never designed to migrate between hosts. That has now become a requierment
- 2 nodes, PVE 9.2.3 (kernel 7.0.14-4-pve, EFI). No shared storage.
- Cluster history: originally built on PVE 7, upgraded through to PVE 9
(so some storage naming/layout predates current versions).
- Each node has its own local ZFS RAID10 pool (4x mirror, WD SN640 NVMe).
- Dedicated migration net: isolated, non-routed, 100 GbE (ConnectX-5), MTU 9000.
- datacenter.cfg: migration: type=insecure,network=<isolated /24>
STORAGE LAYOUT (the core issue)
Each pool is named after its node and split into datasets by volblocksize,
each exposed as its own PVE storage:
PRX01: PRX01-ZFSRaid10/BlockSize32K (32K), /BlockSize64K (64K)
PRX02: PRX02-ZFSRaid10/BlockSize32K (32K), /BlockSize64K (64K)
PVE storage IDs (ZFS, node-restricted):
BS16K/BS32K/BS64K-PRX01ZFSRaid10 (PRX01 only)
BS16K/BS32K/BS64K-PRX02ZFSRaid10 (PRX02 only)
BS512-PRX02ZFS10 (PRX02 only) [benchmarking only]
Structure is symmetric and blocksizes match tier-for-tier, BUT the pool names
differ (PRX01-ZFSRaid10 vs PRX02-ZFSRaid10) and every storage ID is node-scoped.
WORKLOAD (why the tiering exists)
Mixed VMs on both nodes: Windows/Debian/Ubuntu; file servers and SQL/DB servers.
File servers split into large-sequential-video vs. thousands-of-small-files.
Disks are placed on the blocksize tier that suits each pattern — I'd rather not
collapse to a single blocksize.
WHAT I WANT
Faster moves for large VMs. Live migration works but sits at ~145 MB/s.
WHAT'S BLOCKED / WHAT WORKS
1) Offline migration: fails — source storage IDs (BS*-PRX02...) don't exist on
PRX01, so target storage is "not available".
2) Replication (pvesr): needs matching storage ID + dataset path on both nodes;
node-scoped IDs and differing pool names mean no matching target.
3) Works today: live migration with per-disk remap
(BS64K-PRX02 -> BS64K-PRX01, BS32K-PRX02 -> BS32K-PRX01). This is the
~145 MB/s drive-mirror path but preserves tiers.
4) Considering: PBS backup/restore, then move the large 64K disk to its tier
manually.
ALREADY RULED OUT (why 145 MB/s isn't network/disk/CPU/crypto)
Instrumented a full live migration:
- Net ~1.1% ifutil; iperf3 single ~6.9 / 8-parallel ~44 Gbit, ~0 retransmits.
- Jumbo verified (ping -M do -s 8972, 0% loss).
- Both pools idle w/ headroom (zpool iostat), dest shows normal txg sawtooth.
- CPU ~84% idle; top -H shows only guest vCPU threads busy, no pinned migration thread.
- Encryption off (type=insecure).
=> ~145 MB/s is the inherent latency-bound QEMU drive-mirror loop. Acceptable,
but I want delta-only moves for big VMs.
QUESTIONS
1) With node-local pools whose names differ per node, what's the recommended way
to enable pvesr replication? Does it require identical storage IDs AND identical
dataset paths on both nodes?
2) Is the fix to define ONE shared storage ID per tier (both nodes) resolving to
each node's local dataset? If so, how do people handle the differing pool names
— rename a pool, or can the "pool" field differ per node under one ID?
3) Is preserving multiple blocksize tiers across replication sensible, or do people
collapse to one pool/blocksize per node?
4) For a mixed-disk VM (some 32K, some 64K), can each disk keep its matching-blocksize
target, or must all disks land in one storage?
5) Is replication pre-seed the only real lever here, or is there a supported knob
I'm missing for the drive-mirror ceiling?
6) PBS restore: can I place each disk on a chosen target storage on restore, or does
it restore to one storage and I move disk-by-disk afterward? Does stopped-VM
"Move disk" beat the live mirror path or use the same mechanism?
7) Any cases where one method is clearly better given the workload mix — e.g. offline
backup/restore for small-files servers (per-op latency hurts), live mirror fine for
big-video servers?
sidenote: This system was never designed to migrate between hosts. That has now become a requierment