iSCI SSD and HDD LUN

[marivo]

Have a nice day,
I would like to ask for your opinion and possibly help with the data store.

I currently have a Proxmox installation on a server with a data store connected to it using iSCI.
The data store has two LUNs. One SSD and the second HDD.
Use LUN's directly is not checked.
An LVM is created above the SSD LUN and is used to store VMs in Proxmox.
I would like to offer the HDD LUN to users for data and allow them to access it with TrueNAS Core.
Is it possible to mount the HDD LUN as RAW in the current state?
Within an HA cluster, I would like to access the data array, especially for VMs on SSD LUNs.
It wouldn't be bad if other VMs could access the data on the HDD LUN besides the TrueNAS Core.

Is this idea of using a data array correct?
How to set HDD LUN correctly?

Thank you very much for your opinions and help.

 

[LnxBil]

I would like to offer the HDD LUN to users for data and allow them to access it with TrueNAS Core.
Is it possible to mount the HDD LUN as RAW in the current state?

Yes.

Is this idea of using a data array correct?

No idea what a data array is.

How to set HDD LUN correctly?

First, be sure that you have two lvm groups, one SSD and one HDD. You create the TrueNAS Core VM with a small disk on the SSD group and add a disk from HDD group and if you don't want to have other VMs on the HDD group, just allocate all available space to one LUN for your TrueNAS Core VM. Do whatever you have to do inside of your TrueNAS Core and export the data as NFS/CIFS to your containers / VMs that want to access the data. Permissions, quota etc. done in TrueNAS core directly and PVE is not involved in this.

 

[marivo]

Thank you very much for your reply.

Yes.

Great, but how to do it? Directly attach to VM like hard drives?

No idea what a data array is.

Sorry, it should have been "data store" (with SSD LUN and HDD LUN).

First, be sure that you have two lvm groups, one SSD and one HDD. You create the TrueNAS Core VM with a small disk on the SSD group and add a disk from HDD group and if you don't want to have other VMs on the HDD group, just allocate all available space to one LUN for your TrueNAS Core VM. Do whatever you have to do inside of your TrueNAS Core and export the data as NFS/CIFS to your containers / VMs that want to access the data. Permissions, quota etc. done in TrueNAS core directly and PVE is not involved in this.

This is exactly my original idea and procedure. I was just toying with the possibility that the HDD LUN didn't have LVM above it, but was assigned to the TrueNAS Core as RAW. Everything else would then be managed by the TrueNAS Core.

 

[LnxBil]

This is exactly my original idea and procedure. I was just toying with the possibility that the HDD LUN didn't have LVM above it, but was assigned to the TrueNAS Core as RAW. Everything else would then be managed by the TrueNAS Core.

My idea would be the PVE-way-of-thinking. You could also attach the RAW LUN directly to the TrueNAS Core VM via iSCSI and skip any involvement with PVE. With this, you will have direct and sole access to the RAW LUN.

 

[marivo]

My idea would be the PVE-way-of-thinking. You could also attach the RAW LUN directly to the TrueNAS Core VM via iSCSI and skip any involvement with PVE. With this, you will have direct and sole access to the RAW LUN.

Yes it is true. I'm trying to evaluate the pros and cons of these two approaches, because once I've copied the user data, it won't be easy to change. Not to mention the possible crash of TrueNAS Core and the necessary reinstallation.

 

[LnxBil]

Occams razor ans KISS implies to use iSCSI inside of the TrueNAS Core VM without attaching the HDD LUN to PVE.

 

[bbgeek17]

Yes it is true. I'm trying to evaluate the pros and cons of these two approaches, because once I've copied the user data, it won't be easy to change. Not to mention the possible crash of TrueNAS Core and the necessary reinstallation.

If this is more than home lab, then it sounds unnecessarily complicated with many points of failure.

To achieve what you originally asked you need to have your iSCSI storage provide HDD lun on a separate Target from SSD lun. And create new storage object in PVE with "Direct LUN" enabled.
You could map the lun directly, as @LnxBil mentioned, but you would have no way of "masking" the SSD lun and have a potential exposure there of Truenas trying to scan/write to SSD lun or vice versa.

With iSCSI and RAID protection provided by some external storage device, what does TrueNAS give for all the complexity? Pretty gui for CIFS and NFS? Or were you planning to iSCSI export from that single lun datastore? With TrueNAS using ZFS - wouldnt you be breaking all ZFS warnings of not using RAID behind it?

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[marivo]

If this is more than home lab, then it sounds unnecessarily complicated with many points of failure.

It's not a homelab, so I'm considering different options. What does it bring and where could a problem arise. I want to take the path where there will be the fewest points where the problem will arise.

To achieve what you originally asked you need to have your iSCSI storage provide HDD lun on a separate Target from SSD lun. And create new storage object in PVE with "Direct LUN" enabled.
You could map the lun directly, as @LnxBil mentioned, but you would have no way of "masking" the SSD lun and have a potential exposure there of Truenas trying to scan/write to SSD lun or vice versa.

Currently SSD LUN and HDD LUN are under the same target. Therefore, "Use LUN's directly" is not checked.
I will look at the possibility of dividing it into different targets.
Thank you very much, that is a good reminder.

With iSCSI and RAID protection provided by some external storage device, what does TrueNAS give for all the complexity? Pretty gui for CIFS and NFS? Or were you planning to iSCSI export from that single lun datastore? With TrueNAS using ZFS - wouldnt you be breaking all ZFS warnings of not using RAID behind it?

The data store used is QSAN with SSD and HDD. Both parts in RAID and accessed as LUNs.
User data will be offered to other servers through CIFS and NFS. User data will be offered to users using SMB with LDAP.
If you have another suggestion, I will be very happy to consider it. I'm not convinced of the perfection of my solution, but I haven't found a better one yet.

 

[bbgeek17]

The data store used is QSAN with SSD and HDD. Both parts in RAID and accessed as LUNs.
User data will be offered to other servers through CIFS and NFS. User data will be offered to users using SMB with LDAP.
If you have another suggestion, I will be very happy to consider it. I'm not convinced of the perfection of my solution, but I haven't found a better one yet.

Dont some models of QSAN provide CIFS and NFS directly? As mentioned by @LnxBil - best to use KISS method...

And if yours is strictly SAN, there are hundreds of packaged solutions that install directly on a Linux VM and provide a pretty GUI, so that HA is provided by PVE.

good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[marivo]

Dont some models of QSAN provide CIFS and NFS directly? As mentioned by @LnxBil - best to use KISS method...

Some do, but in this case only iSCI is in the administration. It is exactly this type QSAN XEVO XS3324
I would be very happy if it could be simpler. I have already considered this.

And if yours is strictly SAN, there are hundreds of packaged solutions that install directly on a Linux VM and provide a pretty GUI, so that HA is provided by PVE.

Which one do you mean? On Linux there is TrueNAS Scale or OpenMedia Vault. TrueNAS Scale seemed robust to me, that's why I chose TrueNAS Core, but it's true that ZFS and FreeBSD bothers me a bit.

 

[bbgeek17]

There is https://webmin.com/. In the end TrueNAS, OpenMedia, Webmin and others just manage SAMBA and native Linux NFS server. You could just manage them directly with minimal overhead.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[alexskysilk]

I currently have a Proxmox installation on a server with a data store connected to it using iSCI.
The data store has two LUNs. One SSD and the second HDD.

is the "data store" machine used for any purpose other than feeding storage to pve? if not, why not just move the disks over to pve and remove a whole lot of complexity?

 

[bbgeek17]

is the "data store" machine used for any purpose other than feeding storage to pve? if not, why not just move the disks over to pve and remove a whole lot of complexity?

its a proprietary QSAN with a whole lot of LEDs and a mute button. Hopefully with dual controllers. If it is - it would provide marginally better availability than moving all disks into a single PVE node. Op said it was a single server, so seems like no redundancy there.

As much as we love PVE, if the only goal is to run CIFS and NFS exports, why not just run baremetal TrueNAS with their implementations of web gui CIFS/NFS management.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[alexskysilk]

its a proprietary QSAN

I missed that part. carry on

 

[bbgeek17]

I missed that part. carry on

Actually, considering that OP is unlikely to be able to export direct disks from Qsan, but only a LUN built on top of some raid, I would go even further to simplify this:
direct connect fiber from Qsan to your physical host (or SAS cables). Install your favorite Linux, use your favorite filesystem (xfs, ext, etc.). Then run webmin or alternative, or manage cifs/nfs the good old way.
Adding PVE in the middle would marginally complicate things, but would give you a virtualization platform to expand (ie run a Windows AD).

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[marivo]

There is https://webmin.com/. In the end TrueNAS, OpenMedia, Webmin and others just manage SAMBA and native Linux NFS server. You could just manage them directly with minimal overhead.

Thanks, I know everyone named. I was kind of hoping that the TrueNAS Core would save me some work.

is the "data store" machine used for any purpose other than feeding storage to pve? if not, why not just move the disks over to pve and remove a whole lot of complexity?

In this case it is not possible. The QSAN is a large box with many LEDs and disks, two controllers and an SPF+ connection. The only way to access LUNs is through iSCSI.
Proxmox is on multiple servers in an HA cluster.

As much as we love PVE, if the only goal is to run CIFS and NFS exports, why not just run baremetal TrueNAS with their implementations of web gui CIFS/NFS management.

It would be nice to have TrueNAS as baremetal, but it's not possible here. The only option is to have TrueNAS as a VM, then mount the LUN directly or through Proxmox.

Actually, considering that OP is unlikely to be able to export direct disks from Qsan, but only a LUN built on top of some raid, I would go even further to simplify this:
direct connect fiber from Qsan to your physical host (or SAS cables). Install your favorite Linux, use your favorite filesystem (xfs, ext, etc.). Then run webmin or alternative, or manage cifs/nfs the good old way.
Adding PVE in the middle would marginally complicate things, but would give you a virtualization platform to expand (ie run a Windows AD).

Nice summary that we can agree on. Thank you all for the different perspectives. Now just choose the best way.
I'm still hesitating about adding PVE precisely because of the possible expansion to, for example, AD.

 

[marivo]

After the first attempts with TrueNAS Core/Scale, it is not completely easy to add a HDD LUN via iSCSI directly in the GUI. So only through the command line or using PVE.

 

[alexskysilk]

It would be nice to have TrueNAS as baremetal, but it's not possible here

Why is that?