PBS backups in Azure storage

[tlex]

I'm trying to find an efficient way to store backups in the Cloud using Azure storages.
In the cloud, I've setup a container, enabled stfp support on it; on the pbs server I've installed sshfs and mount the storage volume.
So far so good. But when I try to add that mount point as a local pbs datastore, I get the following error message :

(Console for SSHFS)
sshfs -p 22 Azure-Storage1:/ /mnt/Azure-Storage1 -d -o allow_other -o reconnect -o ServerAliveInterval=15 SSHFS version 3.7.1 executing <ssh> <-x> <-a> <-oClearAllForwardings=yes> <-oPort=22> <-oServerAliveInterval=15> <-2> <Azure-Storage1> <-s> <sftp> Server version: 3 [00001] SETSTAT [00001] STATUS 85bytes (13ms)

(Console adding Datastore)
proxmox-backup-manager datastore create azure-store1 /mnt/Azure-Storage1 TASK ERROR: EOPNOTSUPP: Operation not supported on transport endpoint

I can read/write/delete files from that mount manually in shell.
Any toughts on that ? Any better way that you could suggest to send long terms backups to the Azure ?
I don't think setting up a PBS instance in the Cloud would be a good solution since that would consume a lot of money for running a non native azure vm where all I really need is only storage isn't it ?

 

[gurubert]

This will not work. PBS needs a fast (and therefor local) storage to work.

 

[Dunuin]

PBS was designed with local SSDs as storage in mind because it needs a high IOPS performance and isn't doing sequential IO. So it was never meant to run on HDDs or remote filesystems (but atleast here NFS and HDDs run fine but slow). Best would be to have a local PBS server using SSDs and a second offsite PBS server and then use a sync job so backups get synced from the local to the remote PBS. That way you get fast backups/restores with downtime as low as possible and also the additional redundancy of a offsite backup.

 

[rwithd]

I'm in a similar boat, Testing PBS in AWS Elastic Block Storage With Cold HDD (sc1) Volumes As an offsite backup. Our local backup is NFS based right now as we re-organize but will be SSD in the near future. I tried something similar with S3, but You really need to use some sort of Block storage Not Object storage.

 

[flatpzero]

PBS was designed with local SSDs as storage in mind because it needs a high IOPS performance and isn't doing sequential IO. So it was never meant to run on HDDs or remote filesystems (but atleast here NFS and HDDs run fine but slow).

If that's true then PBS has not been properly designed for the purpose of performing backups at an enterprise level. Backups by design are meant to go to slow (or slower) storage where you pay for capacity, not speed. Granted that opens up the argument for backing up NVMe to SSD, but not SSD to SSD.

Being able to use an arbitrary block storage back end is a fundamental expectation of any backup solution. If that's not there, walk away and find another one that offers it.

 

[Dunuin]

The staff explained that once. PBS was designed with SSDs in mind because thats what the future will be. SSDs are getting cheaper and cheaper while HDD prices are more of less stagnating. So in the future everyone will use SSDs and no one wants to use HDDs anymore. And even now in a enterprise environment the cost of a downtime is way higher than the cost difference between HDDs and SSDs. So if you care about fast restores to limit downtime you should even today choose SSDs over HDDs. And stuff like deduplication just needs the IOPS and good random IO performance of SSDs. I totally understand why they designed it in that way. If you create a new backup software from scratch you don't want to put years of work into a oldschool backup concept that is optimized for a outdated storage technology that might be obsolete soon. I also would choose the design that is future proof.

 

[tlex]

The staff explained that once. PBS was designed with SSDs in mind because thats what the future will be. SSDs are getting cheaper and cheaper while HDD prices are more of less stagnating. So in the future everyone will use SSDs and no one wants to use HDDs anymore. And even now in a enterprise environment the cost of downtime is way higher than the cost difference between HDDs and SSDs. So if you care about fast restores to limit downtime you should even today choose SSDs over HDDs. And stuff like deduplication just needs the IOPS and good random IO performance of SSDs. I totally understand why they designed it in that way. If you create a new backup software from scratch you don't want to put years of work into a oldschool backup concept that is optimized for a outdated storage technology that might be obsolete soon. I also would choose the design that is future proof.

Without been so radical I still agree that backups should support slower devices / disks....
My other solution at this time ( And this is not realistic from the $ perspective) is to build a PBS vm in Azure and turn in ON/OFF based on the backup sync schedule with my on prem PBS instance. Still pretty expensive and requires way more maintenance than just cold storage....
Also agree that hdd will disappear to be replaced by SSD but the Cloud wont disappear.. (Unless Poutine tries some of his nuclear gear !!) and I think that Proxmox solutions should embrace it. For exemple, not the best but they could purpose a PBS version in the Azure marketplace... (I still prefer the idea of supporting azure / aws storage containers as cold storage....)

 

[flatpzero]

The staff explained that once. PBS was designed with SSDs in mind because thats what the future will be.

If you look at their screen shot here:

https://pbs.proxmox.com/wiki/index.php/Main_Page#/media/File:Proxmox-Backup-Server-Dashboard.png

"exos-zfs". To me that sounds like a HDD backup target using ZFS built on Segate Exos hard disk drives. The name "exos-zfs" shows up in a lot of screenshots for PBS, so it would seem to me like they're fairly keen on promoting HDD use.

There's a lot wrong with the rest of your comment but lets not derail the thread going into detail about that.

 

[Dunuin]

If you look at their screen shot here:

https://pbs.proxmox.com/wiki/index.php/Main_Page#/media/File:Proxmox-Backup-Server-Dashboard.png

"exos-zfs". To me that sounds like a HDD backup target using ZFS built on Segate Exos hard disk drives. The name "exos-zfs" shows up in a lot of screenshots for PBS, so it would seem to me like they're fairly keen on promoting HDD use.

There's a lot wrong with the rest of your comment but lets not derail the thread going into detail about that.

Just wrote how I remember it what the staff said why they designed PBS the way it is. Searched for the thread but wasn'T able to find it again to quote it.

Nothing is preventing anyone from using HDDs with PBS. It will work fine...just slow. It just by design requires high IOPS and random access performance and thats what HDDs are terrible but SSDs great at. So if you don't care about performance and just want it cheap use HDDs with SSDs for caching or metadata. Otherwise use SSD only.

Making it well working with HDDs too would require to start everything from scratch going to totally different direction where alot of great features usefull for SSD users wouldn't be possible. So double the work, because you would need to basically code two different PBS for SSD and HDD users. The Proxmox team isn't as big as Vmware, Veeam and Co. I don't think they have the capacities to do everything twice. Even if they wanted.

 

[flatpzero]

Just wrote how I remember it what the staff said why they designed PBS the way it is. Searched for the thread but wasn'T able to find it again to quote it.

There are two possibilities here:
1) maybe you misunderstood or misheard the staff person
2) the PBS staff realised their mistake and deleted that text

Making it optimized for HDDs would require to start everything from scratch going to totally different direction. So double the work.

Your badge doesn't say that you're Proxmox Staff, so I'm at a bit of a loss to understand how you're qualified to make a statement like that.

 

[Dunuin]

There are two possibilities here:
1) maybe you misunderstood or misheard the staff person
2) the PBS staff realised their mistake and deleted that text

Maybe I remeber details wrong. Like I said, I also would have prefered to quote the staff if I could find that thread again. You can search the forums if you want. Maybe you will find it. I read everything thats posted in the German and English PVE/PBS forums so it's not that easy to find something again if I don't bookmarked it as I read hundreds of posts each day. This isn't the first discussion about this topic and the Proxmox staff explained alot of times in detail how and why they decided to design PBS the way it is.

Your badge doesn't say that you're Proxmox Staff, so I'm at a bit of a loss to understand how you're qualified to make a statement like that.

I'm not and thats just my personal opinion. I've read most of the threads written in this forum, all the documentations and some source code. So thats my personal résumé based on all of that. But I'm sure the Proxmox staff will also answer soon and explain it again first hand.

 

[flatpzero]

I'm not and thats just my personal opinion. I've read most of the threads written in this forum, all the documentations and some source code. So thats my personal résumé based on all of that. But I'm sure the Proxmox staff will also answer soon and explain it again first hand.

When something is your opinion it is wise to include the phrase "in my opinion ..", otherwise your name may as well be Donald Trump who we all know is an expert on everything.

 

[Dunuin]

Some staff quotes:

https://forum.proxmox.com/threads/determining-backup-server-specs.101363/post-437248:

In my opinion, yes. Datastore maintenance tasks are optimized for flash storage.

Compared to the costs of a long downtime due to a slow backup/restore system, high end NVMe storage is "cheap".

https://forum.proxmox.com/threads/determining-backup-server-specs.101363/post-437300:

Tom, I love Proxmox, but think this is very bad design on the software side to require flash for backups.

It's no hard requirement, but it just makes sense to have for the most important server in a company if things go bad:

• much higher reliability (no moving parts), you want the backup data pool last for more than a few years.
• fast enough so that one can even verify the whole backup dataset (which is part of the "only tested backups are valid backups") in a reasonable time frame
• 15.36 TB SSD drives are available now for ~2300 to 2500 € (excl. VAT), and you probably do not need to scale up to 100 TB immediately, starting with 15 or 30 TB capacity now and adding mirror pairs in the future is a valid option too, not a bad investment for increased reliability and reduced time spent on administration and more importantly on recovery if things go bad - cheap 100 TB spinner only sound good as long as you do not need to wait days until, e.g., your main domain controller or mail server is recovered, which can cost a company easily more than the few thousands one saved on cheaper disks.

I made some other posts regarding HW and also advantages of SSDs in the past if you're interested:
https://forum.proxmox.com/threads/garbage-collector-too-slow.86726/#post-381027
https://forum.proxmox.com/threads/hardware-recommendations-for-larger-setup.100221/#post-432707

Also, if you think it's very bad design nobody stops you from using any other backup solution that fits your design needs better, you won't ever outrun basic limits like seek time and bandwidth though.

Large flash drives aren't comparable in price to HDD's of the same capacity. There is a reason Seagate and WD keep making HDD's bigger and bigger and haven't moved away from manufacturing larger drives each year. Flash just can't compare per TB to HDD's.

Same as (enterprise) spinners can't compare at all in terms of seek times, provided bandwidth and also reliability compared to (enterprise) flash. Remember that PBS uses a Content Addressable Storage to reduce the space usage of subsequent, or similar, backups - that nice advantage comes with the cost of having more files due to chunking and the requirement to check them, so flash fares better, but often you also do not need as much of it.

Note that, as said, nobody stops you from buying cheap(er) and slow(er), with things like ZFS special-devices pair for metadata you can even improve the setup speed quite a bit, for some people its fast enough then for their use case and PBS will work just fine, albeit, don't expect miracles as you'll get the bandwidth and seek times you paid for.

https://forum.proxmox.com/threads/a...anned-feature-in-the-future.96365/post-417774:

Proxmox Backup Server is designed to use high performance local storage.

You can configure nfs for cifs mounts manually but this is not recommended and you will see major performance drops.

https://forum.proxmox.com/threads/a...anned-feature-in-the-future.96365/post-417802:

16TB of ssd storage isn't exactly affordable/cheap....

No one said that SSDs are cheap.

You will benefit a lot from deduplication and fast operations by using SSDs.

Based on your IP, you are located in US so if you consider TCO and the average cost of your hourly work rates, the costs of the SSD are really low compared to the costs of the admin working hours in case of backup/restore operations.

If you choose reliable datacenter SSDs (around 200 per TB), you have a very reliable backup server for many years.

Post in thread 'Garbage collector too slow.' https://forum.proxmox.com/threads/garbage-collector-too-slow.86726/post-381027

 

[flatpzero]

Some staff quotes:

Post in thread 'Garbage collector too slow.' https://forum.proxmox.com/threads/garbage-collector-too-slow.86726/post-381027

Summary: PBS GC performance is garbage.

Sounds like they need to bring on someone that knows how to do application design rather than a graduate - the rest of the arguments are to make up for what appears to be weak product architecture. You'd be forgiven for thinking that deduplication wasn't possible without SSDs.

The comments around "seek time" for backup storage are an indication that the posters have a fundamental misunderstanding of backup storage as it is a situation where seek times are unimportant. In some of their examples they've included LTO, which is tape, and the seek times for tape are larger again than spinning disk. Seemingly some people at Proxmox get backup (examples include tape, HDDs), some people at Proxmox do not (justify using SSD.)

Personally, I think I'd prefer to see Proxmox focus on their virtualisation platform rather than try to diversify and own the whole solution. When it comes to backups, what I want to know is how does Proxmox integrate with veeam, commvault, veritas, acronis, etc.

 

[fabian]

Summary: PBS GC performance is garbage.

Sounds like they need to bring on someone that knows how to do application design rather than a graduate - the rest of the arguments are to make up for what appears to be weak product architecture. You'd be forgiven for thinking that deduplication wasn't possible without SSDs.

The comments around "seek time" for backup storage are an indication that the posters have a fundamental misunderstanding of backup storage as it is a situation where seek times are unimportant. In some of their examples they've included LTO, which is tape, and the seek times for tape are larger again than spinning disk. Seemingly some people at Proxmox get backup (examples include tape, HDDs), some people at Proxmox do not (justify using SSD.)

Personally, I think I'd prefer to see Proxmox focus on their virtualisation platform rather than try to diversify and own the whole solution. When it comes to backups, what I want to know is how does Proxmox integrate with veeam, commvault, veritas, acronis, etc.

you might want to read up on how PBS works under the hood before posting such things. nobody said that deduplication is not possible without SSDs, and PBS works perfectly fine with HDDs, it's just slower for obvious reasons. so if you want fast performance, especially on I/O intensive operations (GC, restore), having a storage that provides fast random access (which nowadays means at least SSDs for metadata caching, ideally all-flash storage) is a must. if you can live with longer restore times or having GC chug along for a few days on bigger datastores, you can use spinning rust

seek times do matter a lot for PBS, as the backup data is stored in a chunked fashion on a content-addressable storage, so accessing lots of chunks always implies lots of seeking. you can improve the situation somewhat by throwing lots of disks at it to spread the load, but flash storage plays in another league. of course there are other approaches to doing and storing backups that don't have this requirement, but they suffer from other problems that we can avoid with our architecture (things like inefficient/no deduplication, dependency between full and incremental/differential snapshots, ...).

 

[tlex]

... So maybe, just to recenter that thread to what I was originally looking for (and that was not a pseudo fight between hdd/sdd), finding a solution in order to take advantage of Cloud storage for long term / cold backups... Maybe that wouldn't be so hard to implement something to support such a thing ? I'm mean, without all the GC part, just like a dump deduplication process with various slow but sure cloud connectors for various storage solutions ? ie. for AWS, Azure, etc ?
I saw a few people doing that with debian and other linux environment and I'm sure I could do something (ugly) by scripting it but the cost that it would be for Proxmox to implement something like that embedded in the nice gui you already have for the backups would be great and I think a great "feature / addition" for many of us ? or at least great exposure to support the Cloud anyhow ?

 

[fabian]

there's an enhancement request open for S3 support for PBS already:
https://bugzilla.proxmox.com/show_bug.cgi?id=3169

 

[gurubert]

If S3 would be supported an on-premise Ceph cluster could also be used as backup storage.

 

[damo2929]

if your using AWS you could use the virtual tape library system with PBS
https://aws.amazon.com/blogs/aws/create-a-virtual-tape-library-using-the-aws-storage-gateway/

 

[dcsapak]

if your using AWS you could use the virtual tape library system with PBS
https://aws.amazon.com/blogs/aws/create-a-virtual-tape-library-using-the-aws-storage-gateway/

sadly aws tape gateway does not work, see here: https://forum.proxmox.com/threads/a...tus-failed-failed-to-fill-whole-buffer.93009/

EDIT: altough i did not try for quite some time, maybe they updated it. it would be nice if anyone could test it and post the result