NetApp & ProxMox VE

[DatacenterDude]

Hello ProxMox enthusiasts!

My name is Nick Howell, and I am the Global Field CTO at NetApp. I've spent the last 6 years helping get our cloud storage offerings online, but from 2006-2011 I was a datacenter admin, and then joined NetApp in 2011 to help lead their virtualization efforts around VMware, KVM/RHEV, Hyper-V, and Xen. We wrote "best practices" configuration documentation for all of them and presented those solutions around the world.

With the recent news of Broadcom's pricing model changes I'm sure you're all mostly aware of, there has been a lot of buzz inside the walls of NetApp. Our stance is: we are Switzerland. We work with all hypervisors, young and old, and provide some of the best enterprise and SMB unified storage out there. And I am re-visiting my old virtualization haunt to reinvigorate NetApp's position as the best storage for virt.

One of the names that has come up constantly for the last 9 months? ProxMox.

Unfortunately, we do not have a lot of experience with it, as traditionally there has not been a huge demand for it within the Enterprise sector. Every hypervisor has it's nuances, and I know ProxMox is no different. As a homelabber, I have dabbled, and plan on making some YouTube content connecting it to NetApp storage and beginning the process of developing some best practices for configuring it for best performance, stability, and resilience.

I'm here to humbly ask... How can I help? AMA. What would you like to see NetApp deliver for you to up-level your storage capabilities?

As an example, one of our partners in Germany, Credativ, recently configured our lightning-fast NVMe storage with ProxMox and wrote up a guided tour. I want to do something similar for FC, NFS, and iSCSI, and just general configuration best practices for working with ONTAP, our operating system.

 

[DatacenterDude]

To the ProxMox team, specifically, we would love to work with you in an official capacity and build integrations over the long term to support the growth of ProxMox VE running on NetApp storage. My email is nhowell@netapp.com and I would love to chat with you.

 

[Max Carrara]

To the ProxMox team, specifically, we would love to work with you in an official capacity and build integrations over the long term to support the growth of ProxMox VE running on NetApp storage. My email is nhowell@netapp.com and I would love to chat with you.

Hello there! If you want to collaborate / contribute to PVE itself, it's best if you hit us up on our pve-devel mailing list. If you're intending to work on our code, it's best if you have a look at our developer documentation as well.

Note that I'm currently working on a refactor of our storage plugin API - this will probably still take a while, but hopefully, in the not-so-distant future, it should be much simpler to integrate third-party storage providers with PVE. I can't share too many details yet, but if that's something you're interested in, it's best to keep an eye on the mailing list and our announcements.

Otherwise, see our Contact Us page on how to reach us, especially for business inquiries etc.

 

[DatacenterDude]

Note that I'm currently working on a refactor of our storage plugin API - this will probably still take a while, but hopefully, in the not-so-distant future, it should be much simpler to integrate third-party storage providers with PVE. I can't share too many details yet, but if that's something you're interested in, it's best to keep an eye on the mailing list and our announcements.

Awesome, thanks Max! I've sent this over to engineering to evaluate and see where we can contribute! We're definitely interested in participating in that, so we'll keep an eye on the progress and see if there are any areas we can help. As you may know, we've had one of the top vSphere plugins for almost 15 years, and did the SMI-S integration for Microsoft with Hyper-V. We've got a lot of bumps and scars doing this kind of work, and would love to help where we can. I'll get the right folks signed up for the appropriate lists for comms. Thank you for the links!

 

[alexskysilk]

I'm here to humbly ask... How can I help? AMA.

The biggest challenge adding a netapp type storage device onto a pve network is the limitation of block and file level access provided.
1. With Block device access (eg, iscsi, nvof, etc) the linux underpinnings of pve provide a limitation of using it backing lvm, which means losing thin provisioning, snapshots, etc. There are third party solutions (eg Blockbridge) but it adds cost, complexity, and yet another vendor into the mix. Netapp could have quite a bit of value proposition if they provided a solution.
2. File (NFS)- this is a rapidly disappearing featureset from current Netapp offering, probably due to inferior performance. I'm not suggesting adding it back, especially if you resolve item 1- but it could act as a stopgap.
3. Additionally/alternatively to 1, add hardware snapshot api access plugin to pve.

just food for thought.

 

[DatacenterDude]

The biggest challenge adding a netapp type storage device onto a pve network is the limitation of block and file level access provided.
1. With Block device access (eg, iscsi, nvof, etc) the linux underpinnings of pve provide a limitation of using it backing lvm, which means losing thin provisioning, snapshots, etc. There are third party solutions (eg Blockbridge) but it adds cost, complexity, and yet another vendor into the mix. Netapp could have quite a bit of value proposition if they provided a solution.
2. File (NFS)- this is a rapidly disappearing featureset from current Netapp offering, probably due to inferior performance. I'm not suggesting adding it back, especially if you resolve item 1- but it could act as a stopgap.
3. Additionally/alternatively to 1, add hardware snapshot api access plugin to pve.

just food for thought.

Great thoughts and comments, thank you.

1) We would likely need to work with the Proxmox team to circumvent LVM similar to how we did with RDM's in the early, early days. We manage LUNs as files (very basically) and allows a lot of our functionality to propagate up-stack. We've been down this road several times in our 30+ years. Being besties with RedHat has helped us master working with LVM and other linux storage initiatives. I have all the faith we could find a solution that worked for everyone. At the end of the day, ZFS is not the end-all be-all storage solution of the world, and often introduces more complications than it's worth.

2) I would love to hear more about this, specifically how file/nfs is perceived as rapidly disappearing as a featureset from NetApp offerings. Frankly, it's what we're most known for, and even though we offer killer SAN functionality, and have since 2003, I'll admit we've had trouble overcoming the perception we don't do SAN, or do it well. So, your comment confuses me a bit, as I tend to hear the opposite.

3) I've sent over Max's link for the refactoring efforts around the Storage Plugin API to our engineering team for review. This would be more of a longer-term effort pending how that progresses, but I can see a future where we have one of the first plugins available since we've basically already got the framework for one from our experience with vCenter/vSphere over the last 15 years. Watch that space!

 

[alexskysilk]

I would love to hear more about this, specifically how file/nfs is perceived as rapidly disappearing as a featureset from NetApp offerings.

I have to admit this is mostly second hand; I run into this mostly with existing VMWare shops who have existing solutions they have in place (eg, ASA) but if you have products that are NAS-first tailored for virtualization, I'd love to hear about it directly from a Netapp FAE/SE

 

[Falk R.]

Hello Nick,

I've always hated the NetApp sales in Germany, who present NetApp as the optimal block storage for VMware.
I always perceived NetApp as the best filer and the Ontap products probably still are. VMware's strong focus on block storage has probably formed the opinions that the previous speaker shares.

I currently see the NetApp storages as the best option for a simple migration from VMware to Proxmox, as NFS is the only open file system supported by VMware.

If you want to create a real added value as customers can easily switch from VMware to Proxmox with minimal downtime, then you could possibly write a plugin that converts the VMware VM configuration into a Proxmox VM, similar to the ESXi importer, but unlike the importer, the VMware VMDK can be used natively in the Proxmox VM. This means that you would always have a migration time of one shutdown and start of the VM.
This would make it possible to automate the procedure I have been using for years and thus migrate large installations more easily. The procedure is described here: https://pve.proxmox.com/wiki/Migrate_to_Proxmox_VE#Attach_Disk_&_Move_Disk_(minimal_downtime)

Best regards,
Falk Rösing

 

[DatacenterDude]

I've always hated the NetApp sales in Germany, who present NetApp as the optimal block storage for VMware.
I always perceived NetApp as the best filer and the Ontap products probably still are.

I will admit... when evaluating storage for my datacenter in 2007, I had never heard the name NetApp either. I was recommended to try 3Par, EqualLogic, and NetApp by CDW. Ended up going with NetApp for the sole purpose of doing fibre-channel SAN for VMware. Enter vSphere 4.0 with support for NFS and 10GbE. We switched in a heartbeat. What I learned was that it was 10x more SAN than I needed for VMware, and had enough overhead to also run all my unstructured NAS (home drives, dept shares, etc) as well as streamline backups/replication... all consolidated down to this single OS called ONTAP. Not to mention eliminating FC switches from the racks in lieu of consolidated Nexus ethernet kit. It's how I organically fell in love with it. Previously, I had only really worked with FC SAN products like DataGeneral Clariion, Symmetrix, and SUN gear.

The unified nature of NetApp is what makes it enticing. A little phrase I coined a few years ago: ONTAP Everywhere.

VMware's strong focus on block storage has probably formed the opinions that the previous speaker shares.

As long as everyone knows this is a sales tactic to get you to buy-in to VSAN. And we'll see how long that sticks around now having never surpassed 10% market share.

you could possibly write a plugin that converts the VMware VM configuration into a Proxmox VM, similar to the ESXi importer, but unlike the importer, the VMware VMDK can be used natively in the Proxmox VM. This means that you would always have a migration time of one shutdown and start of the VM.

Check out Project Shift. It's in the process of being updated for current times with Hyper-V seeing a big update coming with Server 2025. This is something we've discussed widening for additional hypervisors for refactoring and is actively being evaluated and worked on as we speak. We're just now getting our hands on RTM's of 2025. I could easily see PVE fitting into the scope of this in the near-future.

Appreciate the feedback, Falk!

 

[DatacenterDude]

I have to admit this is mostly second hand; I run into this mostly with existing VMWare shops who have existing solutions they have in place (eg, ASA) but if you have products that are NAS-first tailored for virtualization, I'd love to hear about it directly from a Netapp FAE/SE

Absolutely! There's about 5,000 of us in the official NetApp discord (incl support/exec personnel and many of our VAR partners). All are welcome!

 

[Max Carrara]

Awesome, thanks Max! I've sent this over to engineering to evaluate and see where we can contribute! We're definitely interested in participating in that, so we'll keep an eye on the progress and see if there are any areas we can help. As you may know, we've had one of the top vSphere plugins for almost 15 years, and did the SMI-S integration for Microsoft with Hyper-V. We've got a lot of bumps and scars doing this kind of work, and would love to help where we can. I'll get the right folks signed up for the appropriate lists for comms. Thank you for the links!

You're welcome! Glad you wanna help out.

Our storage API and plugin infrastructure is relatively flexible, but it's starting to show its age. The current plan is to make it more developer-friendly and document everything nicely, perhaps even provide a full guide on how to write your own plugin. As I said, this will still take a while, but you can already work on your own plugin if you'd like - for that I recommend just getting familiar with our code and playing around with it. There are also the third-party plugins linstor-proxmox and freenas-proxmox, if you wanna have a look.

Also something that might be relevant: Our storage API has a mechanism to stay backward-compatible, so even if you were to cook up a plugin right now, it should still remain compatible with the brushed-up API.

 

[Bradomski]

@DatacenterDude We are currently an active professional Org dealing with some issues specifically between netapp NFS and Proxmox.

Would this be something that would be able to engage trying to sort through these issues.

 

[Falk R.]

What problems are you struggling with?
You can always get help here in the forum and the NetApp Discord is also an good place to exchange information.

 

[Bradomski]

Right now the issue is that we have NFS share mounted on pve, however when we either attempt to create or migrate to the Netapp share, PVE errors out with non specific errors.

Also despite the NFS being mounted, they show consistent disconnections, but this same Netapp and share are in production on VMWare with no issue

 

[bbgeek17]

@Bradomski , welcome to the forum.

How did you mount the NFS? Did you use PVE facilities or direct mount?
If former, what is the context of "cat /etc/pve/storage.cfg" ?
If the latter, what is the output of "mount" ?

Does VMware and PVE have the same network path/config?
Have you checked for good networking connectivity?
Are MTUs consistent?
Can you use "dd" to write/read to NFS share from the PVE host directly?
What version of PVE/Kernel are you running? (pveversion)

Keep in mind that PVE is based on Debian with Ubuntu-derived Kernel. Treat your PVE NFS troubleshooting as you would any Linux host. Get network traces if necessary.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Bradomski]

We utilized the GUI to mount. I can share the storage.cgf asap, but nothing stood out of th3 ordinary.
I have not tried dd yet but will asap
PVE is current, ie patched to latest today 8.x.x

MTUs are consistent between PVE/netapp/vmware

 

[vizerei]

Chiming in, I work with Brad. MTUs are 9216 everywhere on our "storage" VLAN. All shares were added via the GUI. We've tried disabling kerberos as suggested in another thread (and only disabling certain versions). We've also set individual share permissions to 777 but no change.

Initial add was weird as it would sit for a long time before showing the NFS share list, probably around 20-30 seconds, sometimes restarting the query via re-engaging the drop down. Once the list popped up it added immediately.

 

[bbgeek17]

Proxmox uses native Linux tools for NFS, ie rpcinfo and showmount: https://github.com/proxmox/pve-storage/blob/master/src/PVE/Storage/NFSPlugin.pm#L181

Run them manually, I'd expect you to see delays there. My advice - temporarily mount the NFS manually (ie /mnt/tmp) and exercise it to confirm that everything works.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[vizerei]

We can definitely try that, but I'm my experience that much delay indicates issues. What we DID do was exercise the shares at their mount points after the add and found regular delays in simple things like ls. Listing files in a directory should be zero time, not even taking in to account an enterprise grade 300tb ONTAP appliance and 25gbps dedicated network infrastructure.

 

[bbgeek17]

Truly sounds like a network problem to me. Possible packet loss and/or retransmits.

You could try older kernel. Again, there is no proprietary NFS stack in PVE - its Linux.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox