PVE Cluster with Netapp storage

P

paffeke

Member
Proxmox Subscriber
Jul 1, 2022
11
0
6
Italy
Hi all,
I need some advice for a new upcoming project.
I need to setup a 3 node cluster using NetApp as storage.
The cluster will start running only 8 VMs, details below:
- 6x - 64GB RAM, 32vCPU, 50/100GB OS Disk, 4TB Data Disk (i know it's a nightmare)
- 2x - 32GB RAM, 16vCPU, 30 OS Disk, 1TB Data Disk

Our customer wants to use an already running NetApp storage and here come my questions.
Each node reach NetApp through 2 10Gbps LACP link.
I was thinking to create an ISCSI target on NetApp and use it as a LVM storage on proxmox to store vdisks. Is this way correct or is there something i could do better?
Regarding 4TB vdisk, is possible to use PBS to snapshot the VMs?
Thank you all!
 
paffeke said:
Is this way correct or is there something i could do better?
Click to expand...
Maybe ask NetApp, if how their PVE integration plans are coming along. Fallback is always LVM thick on iSCSI (or FC).

paffeke said:
Regarding 4TB vdisk, is possible to use PBS to snapshot the VMs?
Click to expand...
Snapshots are done on the storage, or in your case with LVM-thick sadly not. PBS can however backup your machine and "feels" like a snapshot, yet it is not nearly as fast as a snapshot. With the backup method also names snapshot, there will be a memory snapshot of the disk and this disk is then sent off to the PBS.
 
  • Like
Reactions: Johannes S
And it's better to use multi channel with nfs or iscsi and not link aggregation ( lag). With iscsi u get more iops if u create a not so big iscsi lun. Better to have 4x 2TB LUN than 1x 8TB LUN.
 
Hi paffeka:
If you are using NetApp Filer (with NFS/CIFS protocol support), I agree with spirit's viewpoint, you should us NFS protocol instead of iSCSI. but if your NetApp Storage is E-series (Do not support NFS/CIFS protocol), then you can only use iSCSI either FC protocol(depends which your storage support it) and let Proxmox VE host utilize it as shared LVM storage.
For Proxmox VE's backup function, it works well with shared LVM because the snapshot will be handle on QEMU level. But for VM snapshot function, you can not use it with shared LVM because currently not supported. but you can consider to use E-Series' build-in snapshot function(althrough it may not flexible because you need additional steps to retrive any single VM's disk from them). But it's better than nothing.
By the way, I have done to backup a single 4TB's vdisk (which belongs to a VM) using PBS. I think that is ok.
 
Last edited:
  • Like
Reactions: Johannes S
paffeke said:
Each node reach NetApp through 2 10Gbps LACP link.
I was thinking to create an ISCSI target on NetApp and use it as a LVM storage on proxmox to store vdisks. Is this way correct or is there something i could do better?
Click to expand...
As others have mentioned, your choices are: iSCSI+LVM or NFS. Each comes with its own set of pluses and minuses.
paffeke said:
Regarding 4TB vdisk, is possible to use PBS to snapshot the VMs?
Click to expand...
You are conflating terminology here. PBS does backups , snapshot is different technology that PBS is not involved in.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: Johannes S
bbgeek17 said:
You are conflating terminology here. PBS does backups , snapshot is different technology that PBS is not involved in.
Click to expand...

To clear possible missunderstandings: PBS uses qemus/kvm (the hypervisor of the Linux kernel) level snapshots for backups. They are storage-agnostic and thus fine to ensure conistency for backups. If one don't need snapshots but arefine with restoring from a backup this can be a workaround for the lack of storage-level snapshots with LVM-thick. This is especially true if one combines it with the live-restore capability of ProxmoxVE and ProxmoxBackupServer. But of course this is not a "one size fit's it all" solution.
I know bbgeek17 as one of the storage professionals here knows that, but for david_tao and other people stumbling over this thread I thought this might help in understanding the (somehow confusing) terminology
 
Last edited:
  • Like
Reactions: UdoB and bbgeek17
you are correct, @Johannes S . It is about most common use of the term. An SQL server could be doing a brief internal Snapshot of its Journal for an internal batch run. Yet this would not be something that most people mean when they say : I took a snapshot of my database.

PBS "snapshot" is a brief lived process of making sure that you have a time consistent view of the data _while_ the backup is running and a particular block has not been backed up.

As an example, you have blocks A-Z. A backup is executed and blocks A-D are backed up. A write comes to block S. Block S will be saved to the "snapshot" until the backup process gets to it. No other blocks will be "snapshotted". The end-user does not have access to this "snapshot".
To sum up - its not what 99.999% mean when they discuss storage snapshot.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: UdoB and Johannes S
Thanks all for these precious answers.
In my mind, when i said "snapshot" i was thinking about "incremental backup" that is a feature which i can achieve with both PVE and PBS, sorry for this error.
Regarding LVM on ISCSI i guess is not what i want. Having a thin provisioning would be great, expecially for the 4TB disks (because i'm sure that 4TB is just a "max recomended sizing" that the "app vendor" wrote in some document for his application sizing)
If i understood well, "nfs multichannel" is pNFS".
I'll try to understand if the NetApp model supports it (sadly the project informations still lack at this point). In that case i'll configure the 2 10Gb sfps of each pve node to reach 2 NetApp interfaces and will see if i can make it works.
 
alexskysilk said:
/pedantic. nconnect has been supported on ONTAP since version 9.9.1 (nconnect IS multithreaded nfs.) Unfortunately since pvesm doesnt support nconnect you'd need to manually manage the mounts to use it.
Click to expand...
pvesm set <storage> --options options vers=4.1,nconnect=8
 
  • Like
Reactions: david_tao and alexskysilk
Meant nfs multi pathing, but multi channel is also a good feature for virtualization.
 
paffeke said:
Thank you all once again.
Next week I should have a chance to set up a 3 node cluster lab and try out the NetApp configuration. I'll try both NFS and SMB and i'll let you know.
Click to expand...
How are things progressing, anything interesting to share yet? Thanks :)
 
You must log in or register to reply here.
Top Bottom