Virtualizing 70+TB file server best practice?

[jaff]

Hello!

I thought I would reach out here to see if anyone else has run into a similar issue.

Our organization produces and stores a large amount of Video and Audio files currently residing on a bare metal file sharing NAS which amounts to around 70TB and is slowly growing each year. We currently utilize Proxmox VE for other parts of our infrastructure, which is working great. We are planning on converting most if not all of our environment to virtual. The hardware setup would be:

IBM 3550M3
Dell MD1200 connected via SAS to be used as direct attached storage

We are thinking of creating a RAIDZ2 or RAIDZ3 ZFS pool using the disks in the MD1200 and adding it to Proxmox VE.

So, my question is: what is the best way to go about virtualizing this huge NAS? Would I want to pass through individual disks to be used directly by it? I feel like have a 70TB qcow2 disk may be a bad idea, but I could be wrong. Am I going about this the right way or should I just stick to keeping the NAS bare metal and separate from Proxmox?

Any suggestions or nudges in the right direction would be appreciated.

Thank you in advance!

 

[bbgeek17]

IBM 3550M3
Dell MD1200 connected via SAS to be used as direct attached storage

We are thinking of creating a RAIDZ2 or RAIDZ3 ZFS pool using the disks in the MD1200 and adding it to Proxmox VE.

The proper order of operation is to add the disks to the PVE node/s and then create the RAID/FS. The MD1200 appears to be a pretty dumb disk enclosure that relies on host RAID card or host software for disk redundancy.

So, my question is: what is the best way to go about virtualizing this huge NAS?

why do you want to do this? What benefit are you expecting from virtualizing this service?

Would I want to pass through individual disks to be used directly by it?

most likely you would have to, depends on what kind of NAS service you plan to use.

I feel like have a 70TB qcow2 disk may be a bad idea, but I could be wrong.

you are not wrong.

Am I going about this the right way or should I just stick to keeping the NAS bare metal and separate from Proxmox?

again, depends on what you are planning to gain. There are cons and pros to both approaches.

---

Blockbridge: Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[guerbywork]

You can also use 9p to share a filesystem of the PVE ZFS to the VM, you need to manually add "args:" to the VM config as virtio-9p-pci devices are not supported directly by proxmox UI

https://forum.proxmox.com/threads/virtfs-virtio-9p-plans-to-incorporate.35315/

I use this setup for my kopia backup VM, and probably will use it for future PBS VM.

 

[bbgeek17]

You can also use 9p to share a filesystem of the PVE ZFS to the VM

Thats interesting, have not heard of this before. But a quick look at the spec suggests that this will not be suitable for more than one node, i.e. cluster. Especially considering that ZFS is not a cluster aware file system.
I got the impression that OP is using PVE cluster, although that has not been stated explicitly.

---

Blockbridge: Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[ness1602]

The only way i could see offering this big data in proxmox is using CephFS,and then you can some HA.

 

[guerbywork]

Thats interesting, have not heard of this before. But a quick look at the spec suggests that this will not be suitable for more than one node, i.e. cluster. Especially considering that ZFS is not a cluster aware file system.
I got the impression that OP is using PVE cluster, although that has not been stated explicitly.

---

Blockbridge: Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Yes this is only possible for a VM on the same node as the ZFS: as you mentionned ZFS in your first post I was thinking you didn't need/have multimachine redundancy for your NAS. storage.

 

[bbgeek17]

Yes this is only possible for a VM on the same node as the ZFS: as you mentionned ZFS in your first post I was thinking you didn't need/have multimachine redundancy for your NAS. storage.

I missed the ZFS/cluster part in OP's post as well. If he is moving from single server to single virtualization node - I see even less benefits in doing so. Only adding extra layers and barriers to get to, what seems to be, critical data for the company.

---

Blockbridge: Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[jaff]

The proper order of operation is to add the disks to the PVE node/s and then create the RAID/FS. The MD1200 appears to be a pretty dumb disk enclosure that relies on host RAID card or host software for disk redundancy.


why do you want to do this? What benefit are you expecting from virtualizing this service?

most likely you would have to, depends on what kind of NAS service you plan to use.

you are not wrong.

again, depends on what you are planning to gain. There are cons and pros to both approaches.

---

Blockbridge: Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Thanks for reaching out! My goal for our network's storage is essentially to have a central pool rather than distributed amongst various servers for easier backup and management. Right now, we have a number of bare metal servers that I would like to consolidate by virtualization and the NAS is one of them that I've been considering. I haven't decided on what NAS software I would like to use, but possibly something like Turnkey File Server as a LXC.

 

[jaff]

You can also use 9p to share a filesystem of the PVE ZFS to the VM, you need to manually add "args:" to the VM config as virtio-9p-pci devices are not supported directly by proxmox UI

https://forum.proxmox.com/threads/virtfs-virtio-9p-plans-to-incorporate.35315/

I use this setup for my kopia backup VM, and probably will use it for future PBS VM.

This is very cool and sounds like it could be the direction I'd head in if I do decide to virtualize the NAS. Thanks!

 

[hec]

Why not using a NetApp as storage? If you buy a AFF you get a 3:1 guarantee. Depending on you needs you can use SSDs or SAS disks or maybe NL-SAS. If you need a quote i can help you. 70TB does not sounds a lot. I would need 2 rackunits for this.