Netapp as PBS-Storage

[OsvaldoP]

After an successfull migration from esxi to pve, one of our customer have an free netap and want to use them as pbs backupstorage
Currently active is an phy PBS with local SSD. Plan is, to use the netapp as second storage to hold multiple backups. Sync from local storage to netapp.

The following options should be available for connecting the NetApp:
NFS/SMB: I think this is the traditional approach that is often described, but not
optimal.

iSCSI: I can't say much about this.

S3: We already using S3, and it runs smoothly for VMs with several hundred GBs.

Which approach would you consider the best from a technical standpoint?
I think S3 would be attractive, we don't have ZFS on the NetApp, and PBS and NetApp would be separated.

Both systems are connected via 10 GbE.

kr
Roland

 

[robertlukan]

After an successfull migration from esxi to pve, one of our customer have an free netap and want to use them as pbs backupstorage
Currently active is an phy PBS with local SSD. Plan is, to use the netapp as second storage to hold multiple backups. Sync from local storage to netapp.

The following options should be available for connecting the NetApp:
NFS/SMB: I think this is the traditional approach that is often described, but not
optimal.

iSCSI: I can't say much about this.

S3: We already using S3, and it runs smoothly for VMs with several hundred GBs.

Which approach would you consider the best from a technical standpoint?
I think S3 would be attractive, we don't have ZFS on the NetApp, and PBS and NetApp would be separated.

Both systems are connected via 10 GbE.

kr
Roland

I have used NetApp as iSCSI storage for PBS in the past. It worked, but I was using an older NetApp where I had to use multiple LUNs and overlay them with ZFS to get a decent datastore for PBS. Frankly speaking I would not do that anymore as it is quite complicated, iSCSI, with multi pathing, ZFS on top.

I think S3 would be better option, I am not sure how is the performance.

I have not tried NFS, but I am not sure if that would even work properly as you have another layer of file system in between.

 

[OsvaldoP]

Hi,

I'm not a NetApp expert, so I want to keep the project straightforward, no ZFS on NetApp, no iSCSI, and so on.
I also agree that an NFS share is not best practice for PBS usage. Therefore, my idea is to use S3 storage.

Our customer is currently in communication with their NetApp expert. If possible, they will set up S3 storage, and we will connect PBS to it.

I will report on our experience.

 

[Johannes S]

The main drawback I see with using the NetApp filer as datastore are two things:
- The performance especially with operations needing high IOPS like verify or garbage collection. It get's worse with HDDs instead of SSDs. What storage are you planning to use?
- The larger attack surface since a bad actor getting access to the NetApp can alter/remove etc your customers backups. This is especially problematic with NFS if you don't use KerberosAuth but also true for S3 ( but to be fair less likely). How are you planning to do the offsite backup? A remote PBS can be configured to not be reachable from your local PBS but still be able to pull backups from the local one. But if you happen to use a cloud S3 ( e.g. aws, wasabi or hetzner) datastore on your local PBS a bad actor taking over the PBS can also alter the NetApps backups AND your offsite backup on some cloud s3 storage.

Regarding performance: Benchmarking NFS versus S3 with different storage media ( hdd/ssd) might give some ideas what to expect.

 

[OsvaldoP]

Hy

I also see the different filesystem-types and the network as biggest problem. But think it would shame not to use the NetApp anymore, so the idea was to integrate Netapp as additional storage.
To separate the PBS and the Netapp as best as possible, the idea would be to connect Netapp as an S3. So PBS does not need to handle the NetApp file system.

An second PBS will installed on Remotesite in an different Vlan. Connections only allowed from second to first pbs "pull.sync".

kr

 

[robertlukan]

There is one important aspect = performance. Initially I used older netapp with spinning disks, while it was fine for a few 10 VMs, it was not fine for MSSQL servers. I had to give PVE hosts fleecing disks otherwise SQL would freak out. That somehow resolved issue for a few weeks. But after having about 70TB of data, it was simply too slow. Operations like garbage collection took several hours, while verification was going on for 2 days.

At that stage I decided to dedicate one physical host with NVMe-s with ZFS. An appliance has about 20 disks of 15TB per disk in RAIDZ. And utilization is about 65%. Garbage collection is done in an about an hour, while verification takes about 16 hours(once a week).

I would like to see a performance comparison with S3. If it would be comparable, I might reconsider my approach to save money on those NVMe-s.

If you implement it, please share some stats.

 

[UriShay-CyberSnap]

+1 to robertlukan on the speed side. Ask the customer's NetApp person what disks sit behind that bucket before you commit, since verify and GC are where it shows first.

The second PBS in another VLAN pulling from the first is the part that buys you the most here, I'd keep that no matter how the filer gets attached.

One thing on the S3 route. Per the PBS docs the access key needs get, put, list and delete on the bucket, so whoever owns the first PBS or that key can empty it. If the filer is on ONTAP 9.16.1 or newer you can put a snapshot policy on the bucket itself. Those S3 snapshots show up as read only buckets (bucketname-s3snap-snapname) and delete is blocked over S3, so the PBS key can't touch them. Removing them needs an admin login on the filer, so keep that account out of the domain and give it its own MFA. The default policy is only 6 hourly, 2 daily and 2 weekly, which is short if something sits quietly for a few weeks before anyone notices, so I'd add a monthly schedule. They don't work on buckets with object locking or in a SnapMirror relationship, and they count against the bucket quota.

Also plan a small dedicated disk for the S3 datastore cache, the docs suggest 64 to 128 GiB.

Disclosure: I'm founder and CEO of CyberSnap.