Set up ZPool, Replication, and HA - no snapshots. Grand total of less than 10TB worth of data. 32TB STZ reporting full, io-fault on a VM???

[NuAngel]

2 nodes, 3rd node does not have access to ZPool, it is just a tiebreaker node for quorum. No snapshotting, backups are handled by Veeam to their own hardware (in other words, no backups in the web-interface of ProxMox). Everything migrated from a HyperV computer, where the 7TB was thick-provisioned, so that's the absolute MAXIMUM the data should be. When converted to RAW with compression, etc... should be roughly 5 TB on disk. The PHYSICAL server these images all came from was 14TB total. Migrated to 2 identical 32TB servers, with ZFS3 enabled. EVEN IF that only left me with 17TB of usable space, I should be under 15TB even if every virtual drive was full (they aren't even close).

Anyway, I experienced an IO-fault on one of the VMs on node2 ("the big one" with a single 6TB disk). In order to recover after IO-fault on node2, I migrated one of the smallers machines from node2 to node1. I removed all replication jobs / HA settings through the GUI, but one of them kept saying removal pending, so I manually edited /etc/pve/replication.cfg and rebooted the node to get it to stop.

First question: how do I go about cleaning up the mess Replication has left behind?

Next question: what on earth did I do wrong?

 

[NuAngel]

 

[NuAngel]

Theory - the Veeam backup claimed it finished about the time I had the io error - so... does Veeam create a massive snapshot during the backup process? But even if so, why wasn't it being cleaned up when it was done?

 

[NuAngel]

The issue is 101-Disk 1, saying that it has used 12.8TB.

lsblk inside the guest OS shows that SDB = 5.9TB. Why the hell is it using double that?
fdisk concurs:

 

[UdoB]

with ZFS3 enabled

I'm not sure what this is. Maybe RaidZ3? You may show zpool status to show topology information. (Preferable in text form and in [code]...[/code]-tags.)

RaidZ/Z2/Z3 can be tricky and may seem to waste a large amount of space. Example: https://forum.proxmox.com/threads/zfs-vs-single-disk-configuration-recomendation.138161/#post-616199

Short snippet: "A 8 disk raidz3 pool would require that you increase the block size from 8K (75% capacity loss) to ..."


In the PVE context I use mostly mirrored vdevs, so I have not much experience with that problem.

 

[NuAngel]

I'm not sure what this is. Maybe RaidZ3? You may show zpool status to show topology information. (Preferable in text form and in [code]...[/code]-tags.)

RaidZ/Z2/Z3 can be tricky and may seem to waste a large amount of space. Example: https://forum.proxmox.com/threads/zfs-vs-single-disk-configuration-recomendation.138161/#post-616199

Short snippet: "A 8 disk raidz3 pool would require that you increase the block size from 8K (75% capacity loss) to ..."


In the PVE context I use mostly mirrored vdevs, so I have not much experience with that problem.

Yes, RAIDZ3 on 8x4TB drives, with a single ZFS created on them. I expected a max of about 30% capacity loss, so this has definitely been a learning experience.

 

[NuAngel]

root@stpmve2:~# zpool status
  pool: STZ
 state: ONLINE
config:

        NAME                                               STATE     READ WRITE CKSUM
        STZ                                                ONLINE       0     0     0
          raidz3-0                                         ONLINE       0     0     0
            ata-SPCC_Solid_State_Disk_AA250701S304KG00067  ONLINE       0     0     0
            ata-SPCC_Solid_State_Disk_AA250217S304TB01174  ONLINE       0     0     0
            ata-SPCC_Solid_State_Disk_AA250217S304TB02434  ONLINE       0     0     0
            ata-SPCC_Solid_State_Disk_AA250701S304KG00579  ONLINE       0     0     0
            ata-SPCC_Solid_State_Disk_AA250701S304KG00089  ONLINE       0     0     0
            ata-SPCC_Solid_State_Disk_AA250701S304KG00540  ONLINE       0     0     0
            ata-SPCC_Solid_State_Disk_AA250701S304KG01613  ONLINE       0     0     0
            ata-SPCC_Solid_State_Disk_AA250701S304KG01617  ONLINE       0     0     0

errors: No known data errors

 

[Impact]

According to here you'll have about 16T~ real space: https://wintelguy.com/zfs-calc.pl