Help with Proof of Concept Proxmox Setup

[NugentS]

Hi,
I have been running VMWare as a cluster for many years in my home lab. Its got nominally three nodes, but I usually only run one with the others turned off. I have some spare hardware that I can run a proxmox POC on - to see if I can replace my VMWare cluster with a proxmox cluster. I am however running into a couple of issues.

The hardware at the moment (remember POC only) is a SFF HP Elitedesk with a single NIC, 16GB RAM and 500GB local NVMe storage. I am not expecting quick
Additional Storage is 3 NAS units, 2 TrueNAS (Scale) and 1 elderly Synology.
The TrueNAS boxes each have a NFS Share available
The Synology has 2 NFS shares, one for VM's and the other for backup.
These are all defined at the DC level in proxmox

Issue 1: NFS Thin Storage.
I have created a Ubuntu Server VM as a guest on my single Proxmox Node
The disk was created as 24GB on local-lvm - which I assume is thin storage on the NVMe drive.
Hardware says its 24GB, but my datastore useage says 5.85GB is used - so I am assuming this is stored in a thin format. I have added the QEMU Agent to the guest
On backup - this backs up to a 2.09GB File
I have selected Run guest-trim after a disk move or VM Migration

I have created an identical Ubuntu Server VM on the synology NFS share. Disk format is qcow2 - but it appears to be thick provisioned. The store says 24GB is used.

I have created an identical Ubuntu Server VM on a TrueNAS NFS share. Disk format is qcow2 however in this case the dataset only uses 3.05GB - so would appear to be thin provisioned.

Question: Is thin provisioning not possible on the synology? (not that it matters a great deal as the synology is backup space only)

Issue 2: iSCSI - or more specifically ZFS over iSCSI
My main datastore (for VMWare atm) is an iSCSI Pool, on a bunch of mirrored SSD's on my main TrueNAS box - this is where I would, should I move to proxmox, store the VM's under a steady state. Obviously I cannot use that store at the moment - but I have set up an iSCSI dataset on my backup TrueNAS. It won't be nearly as quick - but that doesn't matter at this stage.
From https://pve.proxmox.com/wiki/Storage it would appear I should be using ZFS over iSCSI as that allows snapshots, thin provisioning etc.
On the TrueNAS box - I have a 1TiB ZVOL called pm-1
For the life of me I cannot set ZFS over iSCSI - is this possible in the GUI or is there some commandline wizardry required?

 

[bbgeek17]

Issue 2: iSCSI - or more specifically ZFS over iSCSI

Proxmox iSCSI over ZFS means: PVE is able to login to target host as root, able to manipulate your ZFS pool (create data sets), able to manipulate the target host's iSCSI configuration, specifically export the data set that PVE created via an iSCSI on the fly.

There are more than a few discussions about this specific storage plugin on the forum and off. There is a special plugin out there for truenas.

Good luck

https://pve.proxmox.com/wiki/Storage:_ZFS_over_ISCSI

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[NugentS]

@bbgeek17 Thank you
ZFS over iSCSI up and working (sort of / partially) on Proxmov VE and TrueNAS Scale.

ZFS over iSCSI - TrueNAS

I created a VM - which worked and then migrated the disk this to the local storage (simulating maintenance of the iscsi store). Proxmox was unable to delete the iscsi disk after the migration.

Any attempt to delete the disk failed with proxmox complaining that the VM existed (despite the VM pointing at the local store). I deleted the VM completely and am still unable to delete the disk from the iSCSI store with the message
"Could not find lu_name for zvol vm-103-disk-0 at /usr/share/perl5/PVE/Storage/ZFSPlugin.pm line 118. (500)"


The bit with the red splodge is: iqn.2024-02.domain.stuff.hairynas:target01

A number of other zvols existed on the TN Scale after this mess - all of which I had to manually delete in TrueNAS.
This seems less than reliable which seems to be bourn out by comments on the github

So - a question on storage.

I plan on having 2 proxmox nodes, one of which will likley be turned off running at most half a dozen VM's. Mostly windows. None of which are especially IO heavy.
I have a single TN Server with all the usual iSCSI, NFS. CIFS available as primary storage
Features I need / want (from Storage) are:
1. Thin provision
2. Storage Migration to and from the main storage whilst maintaining thin provisioning to allow for the TrueNAS maintenance

[Note that my secondary TN box is backup (and testing) only and my elderly Synology is my second local backup]. With proxmox I will lose the synology active backup for the VM's - but I guess I can replace with a proxmox backup server - to be tested]

I love the ZFS over iSCSI creating a seperate Extent for each VM disk. Thats preferable to the default VMWare manner of one lump of storage but (at least atm) the ZFS over iSCSI seems unreliable, unless I am doing summat wrong

It would seem to be that NFS (and I need to test different versions) would be my goto storage solution for proxmox anyhow
Thin provision - check
Migration whilst maintaining thin provsioning - check
Native (to proxmox) - check

Agree / Disagree? (I finally got to the question)

 

[bbgeek17]

Agree / Disagree? (I finally got to the question)

I apologize, thats too much for me to process on Friday morning for a one word answer on a home lab.
My recommendation is - try all different things, learn how they work and their advantages and disadvantages. Pick the one that works best for you.

Best of luck.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox