TrueNAS VM using Proxmox local disk (system) and remote iSCSI (data) from another TrueNAS?

[CDuv]

TL;DR: Can I have a TrueNAS SCALE server N1 sharing data over iSCSI with a Proxmox server P1 hosting a TrueNAS SCALE server N2 consuming said iSCSI drive?

Hello,

I have the following two working servers:

• N1: a TrueNAS SCALE (v23.10.0.1) server with nine 2TB HDD (in RAIDZ2 setup) and 64 GB of RAM
• P1: a Proxmox (v7.4) server with eight 1TB HDD (in RAIDZ2 setup) and 64 GB of RAM : it's part of a 4-nodes cluster but other nodes are lighter/smaller

I want to serve files to my users over SMB, so I need a NAS, so I will:

1. Create a Zvol "N2-data-zvol" on N1
2. Share this Zvol over iSCSI
3. Declare a new iSCSI storage on P1 that accesses this iSCSI share from N1
4. Create a VM for N2 with 2 virtual disks: one local (for system) and another that uses the declared iSCSI storage (for data)
5. Install TrueNAS SCALE on the VM

Advantages:

• All my data are on one server (N1: that I must backup of course)
• N2 is virtualized:
  • easier state/configuration backup
  • I can scale up or down CPU and RAM
  • I can move N2 over my cluster when I need to perform maintenance on physical servers (cluster nodes)

Disadvantages:

• The setup requires 2 physical servers and a VM to be up (N1, P1 and N2) and working network between them

That last part is where I have no clue of it's a no go or not: Can TrueNAS, iSCSI and/or Proxmox handle network disconnection fine?

Note: I am using a dedicated LAN (separate Ethernet NICs) between N1 and P1 for the iSCSI trafic: let's call it a "SAN".

Other thought: I initially thought N2 could access iSCSI share from N1 directly (avoiding any configuration on Proxmox) but it looks like TrueNAS is not meant to have it's data on a remote server (oops?).

Is there a network protocol other than iSCSI that would be a better fit for this project?

NB: This is thread was also started on TrueNAS's community forum.

 

[bbgeek17]

TL;DR: Can I have a TrueNAS SCALE server N1 sharing data over iSCSI with a Proxmox server P1 hosting a TrueNAS SCALE server N2 consuming said iSCSI drive?

The short answer is "yes". If you abstract from specific product names then all you are doing:
making storage available via iSCSI to a hypervisor, so that you can run a guest VM that consumes said storage.
Essentially this is the very basic functionality of virtualization.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[bbgeek17]

Can TrueNAS, iSCSI and/or Proxmox handle network disconnection fine?

Proxmox in this case is an iSCSI client. Proxmox utilizes standard Linux iSCSI tools to connect to iSCSI target (TrueNAS). iSCSI protocol has been around for a while and there are well known timeout values that most storage vendors are very careful to adhere to. Especially in Enterprise world.
As long as the disconnect/timeout is within the timeout - it should be transparent to the OS. If, however, the outage exceeds the reasonable time - Kernel IO timeouts come into play.

Running a "storage appliance" on top of a hypervisor, using another "storage appliance" as backing device creates a complex enough environment that its impossible to predict how all pieces will interact in case of failure. You will need to test the failure scenarios that are important to you.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[UdoB]

Once upon a time there was the idea of KISS (https://en.wikipedia.org/wiki/KISS_principle).

I have built some (small) crazy constructs in the past - and I was proud of them: "see my great setup, it works!". Then, when there was a problem, I was very unhappy because inter-dependencies were often the real problem to make the complete system run again. Now I try to avoid this.

If you are happy to debug this thing (and can live with downtime) then it is probably fine!

Have fun!

 

[CDuv]

Thanks for the answers.
So it looks like it can be done, but may not be reliable (at least with these components).

I am pro-KISS and I really thought having dedicated parts hypervisor, VM and storage was a way to keep thing simple (instead of a single monolith server).

What is the alternative? Other components? Other softwares? I mean: will I need a dedicated server (with CPU, RAM and HDD) each time I need a server to serve files over the network (a NAS)?

At least, the part where Proxmox consumes an iSCSI storage from a bare-metal TrueNAS is perfectly fine, right (from the Proxmox perspective)?

 

[UdoB]

At least, the part where Proxmox consumes an iSCSI storage from a bare-metal TrueNAS is perfectly fine, right (from the Proxmox perspective)?

Sure!

My point is: you need local storage stack anyway - so it should be reliable / redundant / stable.

For me fetching storage via a network is an additional point of failure.

To reduce the now added risks there are constructs with redundant network interfaces + switches + storage servers. Whatever amount of complexity you add there is always more things that can fail than not using outsourced storage in the first place.

Of course there are counter arguments: take several "cheap" nodes and only one reliable, redundant networked storage. Then a node can die and get replaced by a spare node easily. This approach is off my small world.

Again: choose your poison. There is more than one solution that might fit your needs.

(And we did not talk about performance: local storage is nearly always faster than inserting a full network stack additionally...)