PBS backups in Azure storage

tlex

Member
Mar 9, 2021
87
6
8
41
I'm trying to find an efficient way to store backups in the Cloud using Azure storages.
In the cloud, I've setup a container, enabled stfp support on it; on the pbs server I've installed sshfs and mount the storage volume.
So far so good. But when I try to add that mount point as a local pbs datastore, I get the following error message :

(Console for SSHFS)
sshfs -p 22 Azure-Storage1:/ /mnt/Azure-Storage1 -d -o allow_other -o reconnect -o ServerAliveInterval=15 SSHFS version 3.7.1 executing <ssh> <-x> <-a> <-oClearAllForwardings=yes> <-oPort=22> <-oServerAliveInterval=15> <-2> <Azure-Storage1> <-s> <sftp> Server version: 3 [00001] SETSTAT [00001] STATUS 85bytes (13ms)

(Console adding Datastore)
proxmox-backup-manager datastore create azure-store1 /mnt/Azure-Storage1 TASK ERROR: EOPNOTSUPP: Operation not supported on transport endpoint

I can read/write/delete files from that mount manually in shell.
Any toughts on that ? Any better way that you could suggest to send long terms backups to the Azure ?
I don't think setting up a PBS instance in the Cloud would be a good solution since that would consume a lot of money for running a non native azure vm where all I really need is only storage isn't it ?
 

Dunuin

Famous Member
Jun 30, 2020
8,966
2,303
156
Germany
PBS was designed with local SSDs as storage in mind because it needs a high IOPS performance and isn't doing sequential IO. So it was never meant to run on HDDs or remote filesystems (but atleast here NFS and HDDs run fine but slow). Best would be to have a local PBS server using SSDs and a second offsite PBS server and then use a sync job so backups get synced from the local to the remote PBS. That way you get fast backups/restores with downtime as low as possible and also the additional redundancy of a offsite backup.
 
Last edited:
Dec 8, 2021
14
0
1
29
USA
I'm in a similar boat, Testing PBS in AWS Elastic Block Storage With Cold HDD (sc1) Volumes As an offsite backup. Our local backup is NFS based right now as we re-organize but will be SSD in the near future. I tried something similar with S3, but You really need to use some sort of Block storage Not Object storage.
 
Mar 24, 2022
13
3
3
PBS was designed with local SSDs as storage in mind because it needs a high IOPS performance and isn't doing sequential IO. So it was never meant to run on HDDs or remote filesystems (but atleast here NFS and HDDs run fine but slow).

If that's true then PBS has not been properly designed for the purpose of performing backups at an enterprise level. Backups by design are meant to go to slow (or slower) storage where you pay for capacity, not speed. Granted that opens up the argument for backing up NVMe to SSD, but not SSD to SSD.

Being able to use an arbitrary block storage back end is a fundamental expectation of any backup solution. If that's not there, walk away and find another one that offers it.
 
  • Like
Reactions: rwithd and gurubert

Dunuin

Famous Member
Jun 30, 2020
8,966
2,303
156
Germany
The staff explained that once. PBS was designed with SSDs in mind because thats what the future will be. SSDs are getting cheaper and cheaper while HDD prices are more of less stagnating. So in the future everyone will use SSDs and no one wants to use HDDs anymore. And even now in a enterprise environment the cost of a downtime is way higher than the cost difference between HDDs and SSDs. So if you care about fast restores to limit downtime you should even today choose SSDs over HDDs. And stuff like deduplication just needs the IOPS and good random IO performance of SSDs. I totally understand why they designed it in that way. If you create a new backup software from scratch you don't want to put years of work into a oldschool backup concept that is optimized for a outdated storage technology that might be obsolete soon. I also would choose the design that is future proof.
 

tlex

Member
Mar 9, 2021
87
6
8
41
The staff explained that once. PBS was designed with SSDs in mind because thats what the future will be. SSDs are getting cheaper and cheaper while HDD prices are more of less stagnating. So in the future everyone will use SSDs and no one wants to use HDDs anymore. And even now in a enterprise environment the cost of downtime is way higher than the cost difference between HDDs and SSDs. So if you care about fast restores to limit downtime you should even today choose SSDs over HDDs. And stuff like deduplication just needs the IOPS and good random IO performance of SSDs. I totally understand why they designed it in that way. If you create a new backup software from scratch you don't want to put years of work into a oldschool backup concept that is optimized for a outdated storage technology that might be obsolete soon. I also would choose the design that is future proof.
Without been so radical I still agree that backups should support slower devices / disks....
My other solution at this time ( And this is not realistic from the $ perspective) is to build a PBS vm in Azure and turn in ON/OFF based on the backup sync schedule with my on prem PBS instance. Still pretty expensive and requires way more maintenance than just cold storage....
Also agree that hdd will disappear to be replaced by SSD but the Cloud wont disappear.. (Unless Poutine tries some of his nuclear gear !!) and I think that Proxmox solutions should embrace it. For exemple, not the best but they could purpose a PBS version in the Azure marketplace... (I still prefer the idea of supporting azure / aws storage containers as cold storage....)
 
Mar 24, 2022
13
3
3
The staff explained that once. PBS was designed with SSDs in mind because thats what the future will be.

If you look at their screen shot here:

https://pbs.proxmox.com/wiki/index.php/Main_Page#/media/File:Proxmox-Backup-Server-Dashboard.png

"exos-zfs". To me that sounds like a HDD backup target using ZFS built on Segate Exos hard disk drives. The name "exos-zfs" shows up in a lot of screenshots for PBS, so it would seem to me like they're fairly keen on promoting HDD use.

There's a lot wrong with the rest of your comment but lets not derail the thread going into detail about that.
 

Dunuin

Famous Member
Jun 30, 2020
8,966
2,303
156
Germany
If you look at their screen shot here:

https://pbs.proxmox.com/wiki/index.php/Main_Page#/media/File:Proxmox-Backup-Server-Dashboard.png

"exos-zfs". To me that sounds like a HDD backup target using ZFS built on Segate Exos hard disk drives. The name "exos-zfs" shows up in a lot of screenshots for PBS, so it would seem to me like they're fairly keen on promoting HDD use.

There's a lot wrong with the rest of your comment but lets not derail the thread going into detail about that.
Just wrote how I remember it what the staff said why they designed PBS the way it is. Searched for the thread but wasn'T able to find it again to quote it.

Nothing is preventing anyone from using HDDs with PBS. It will work fine...just slow. It just by design requires high IOPS and random access performance and thats what HDDs are terrible but SSDs great at. So if you don't care about performance and just want it cheap use HDDs with SSDs for caching or metadata. Otherwise use SSD only.

Making it well working with HDDs too would require to start everything from scratch going to totally different direction where alot of great features usefull for SSD users wouldn't be possible. So double the work, because you would need to basically code two different PBS for SSD and HDD users. The Proxmox team isn't as big as Vmware, Veeam and Co. I don't think they have the capacities to do everything twice. Even if they wanted.
 
Last edited:
Mar 24, 2022
13
3
3
Just wrote how I remember it what the staff said why they designed PBS the way it is. Searched for the thread but wasn'T able to find it again to quote it.

There are two possibilities here:
1) maybe you misunderstood or misheard the staff person
2) the PBS staff realised their mistake and deleted that text

Making it optimized for HDDs would require to start everything from scratch going to totally different direction. So double the work.

Your badge doesn't say that you're Proxmox Staff, so I'm at a bit of a loss to understand how you're qualified to make a statement like that.
 

Dunuin

Famous Member
Jun 30, 2020
8,966
2,303
156
Germany
There are two possibilities here:
1) maybe you misunderstood or misheard the staff person
2) the PBS staff realised their mistake and deleted that text
Maybe I remeber details wrong. Like I said, I also would have prefered to quote the staff if I could find that thread again. You can search the forums if you want. Maybe you will find it. I read everything thats posted in the German and English PVE/PBS forums so it's not that easy to find something again if I don't bookmarked it as I read hundreds of posts each day. This isn't the first discussion about this topic and the Proxmox staff explained alot of times in detail how and why they decided to design PBS the way it is.
Your badge doesn't say that you're Proxmox Staff, so I'm at a bit of a loss to understand how you're qualified to make a statement like that.
I'm not and thats just my personal opinion. I've read most of the threads written in this forum, all the documentations and some source code. So thats my personal résumé based on all of that. But I'm sure the Proxmox staff will also answer soon and explain it again first hand.
 
Last edited:
Mar 24, 2022
13
3
3
I'm not and thats just my personal opinion. I've read most of the threads written in this forum, all the documentations and some source code. So thats my personal résumé based on all of that. But I'm sure the Proxmox staff will also answer soon and explain it again first hand.

When something is your opinion it is wise to include the phrase "in my opinion ..", otherwise your name may as well be Donald Trump who we all know is an expert on everything.
 

Dunuin

Famous Member
Jun 30, 2020
8,966
2,303
156
Germany
Some staff quotes:

https://forum.proxmox.com/threads/determining-backup-server-specs.101363/post-437248:
In my opinion, yes. Datastore maintenance tasks are optimized for flash storage.

Compared to the costs of a long downtime due to a slow backup/restore system, high end NVMe storage is "cheap".

https://forum.proxmox.com/threads/determining-backup-server-specs.101363/post-437300:
Tom, I love Proxmox, but think this is very bad design on the software side to require flash for backups.
It's no hard requirement, but it just makes sense to have for the most important server in a company if things go bad:
  • much higher reliability (no moving parts), you want the backup data pool last for more than a few years.
  • fast enough so that one can even verify the whole backup dataset (which is part of the "only tested backups are valid backups") in a reasonable time frame
  • 15.36 TB SSD drives are available now for ~2300 to 2500 € (excl. VAT), and you probably do not need to scale up to 100 TB immediately, starting with 15 or 30 TB capacity now and adding mirror pairs in the future is a valid option too, not a bad investment for increased reliability and reduced time spent on administration and more importantly on recovery if things go bad - cheap 100 TB spinner only sound good as long as you do not need to wait days until, e.g., your main domain controller or mail server is recovered, which can cost a company easily more than the few thousands one saved on cheaper disks.
I made some other posts regarding HW and also advantages of SSDs in the past if you're interested:
https://forum.proxmox.com/threads/garbage-collector-too-slow.86726/#post-381027
https://forum.proxmox.com/threads/hardware-recommendations-for-larger-setup.100221/#post-432707

Also, if you think it's very bad design nobody stops you from using any other backup solution that fits your design needs better, you won't ever outrun basic limits like seek time and bandwidth though.
Large flash drives aren't comparable in price to HDD's of the same capacity. There is a reason Seagate and WD keep making HDD's bigger and bigger and haven't moved away from manufacturing larger drives each year. Flash just can't compare per TB to HDD's.
Same as (enterprise) spinners can't compare at all in terms of seek times, provided bandwidth and also reliability compared to (enterprise) flash. Remember that PBS uses a Content Addressable Storage to reduce the space usage of subsequent, or similar, backups - that nice advantage comes with the cost of having more files due to chunking and the requirement to check them, so flash fares better, but often you also do not need as much of it.

Note that, as said, nobody stops you from buying cheap(er) and slow(er), with things like ZFS special-devices pair for metadata you can even improve the setup speed quite a bit, for some people its fast enough then for their use case and PBS will work just fine, albeit, don't expect miracles as you'll get the bandwidth and seek times you paid for.

https://forum.proxmox.com/threads/a...anned-feature-in-the-future.96365/post-417774:
Proxmox Backup Server is designed to use high performance local storage.

You can configure nfs for cifs mounts manually but this is not recommended and you will see major performance drops.

https://forum.proxmox.com/threads/a...anned-feature-in-the-future.96365/post-417802:
16TB of ssd storage isn't exactly affordable/cheap....
No one said that SSDs are cheap.

You will benefit a lot from deduplication and fast operations by using SSDs.

Based on your IP, you are located in US so if you consider TCO and the average cost of your hourly work rates, the costs of the SSD are really low compared to the costs of the admin working hours in case of backup/restore operations.

If you choose reliable datacenter SSDs (around 200 per TB), you have a very reliable backup server for many years.

Post in thread 'Garbage collector too slow.' https://forum.proxmox.com/threads/garbage-collector-too-slow.86726/post-381027
 
Last edited:
  • Like
Reactions: nibblerrick
Mar 24, 2022
13
3
3
Some staff quotes:

Post in thread 'Garbage collector too slow.' https://forum.proxmox.com/threads/garbage-collector-too-slow.86726/post-381027

Summary: PBS GC performance is garbage.

Sounds like they need to bring on someone that knows how to do application design rather than a graduate - the rest of the arguments are to make up for what appears to be weak product architecture. You'd be forgiven for thinking that deduplication wasn't possible without SSDs.

The comments around "seek time" for backup storage are an indication that the posters have a fundamental misunderstanding of backup storage as it is a situation where seek times are unimportant. In some of their examples they've included LTO, which is tape, and the seek times for tape are larger again than spinning disk. Seemingly some people at Proxmox get backup (examples include tape, HDDs), some people at Proxmox do not (justify using SSD.)

Personally, I think I'd prefer to see Proxmox focus on their virtualisation platform rather than try to diversify and own the whole solution. When it comes to backups, what I want to know is how does Proxmox integrate with veeam, commvault, veritas, acronis, etc.
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
8,372
1,662
174
Summary: PBS GC performance is garbage.

Sounds like they need to bring on someone that knows how to do application design rather than a graduate - the rest of the arguments are to make up for what appears to be weak product architecture. You'd be forgiven for thinking that deduplication wasn't possible without SSDs.

The comments around "seek time" for backup storage are an indication that the posters have a fundamental misunderstanding of backup storage as it is a situation where seek times are unimportant. In some of their examples they've included LTO, which is tape, and the seek times for tape are larger again than spinning disk. Seemingly some people at Proxmox get backup (examples include tape, HDDs), some people at Proxmox do not (justify using SSD.)

Personally, I think I'd prefer to see Proxmox focus on their virtualisation platform rather than try to diversify and own the whole solution. When it comes to backups, what I want to know is how does Proxmox integrate with veeam, commvault, veritas, acronis, etc.

you might want to read up on how PBS works under the hood before posting such things. nobody said that deduplication is not possible without SSDs, and PBS works perfectly fine with HDDs, it's just slower for obvious reasons. so if you want fast performance, especially on I/O intensive operations (GC, restore), having a storage that provides fast random access (which nowadays means at least SSDs for metadata caching, ideally all-flash storage) is a must. if you can live with longer restore times or having GC chug along for a few days on bigger datastores, you can use spinning rust ;)

seek times do matter a lot for PBS, as the backup data is stored in a chunked fashion on a content-addressable storage, so accessing lots of chunks always implies lots of seeking. you can improve the situation somewhat by throwing lots of disks at it to spread the load, but flash storage plays in another league. of course there are other approaches to doing and storing backups that don't have this requirement, but they suffer from other problems that we can avoid with our architecture (things like inefficient/no deduplication, dependency between full and incremental/differential snapshots, ...).
 
  • Like
Reactions: nibblerrick

tlex

Member
Mar 9, 2021
87
6
8
41
... So maybe, just to recenter that thread to what I was originally looking for (and that was not a pseudo fight between hdd/sdd), finding a solution in order to take advantage of Cloud storage for long term / cold backups... Maybe that wouldn't be so hard to implement something to support such a thing ? I'm mean, without all the GC part, just like a dump deduplication process with various slow but sure cloud connectors for various storage solutions ? ie. for AWS, Azure, etc ?
I saw a few people doing that with debian and other linux environment and I'm sure I could do something (ugly) by scripting it but the cost that it would be for Proxmox to implement something like that embedded in the nice gui you already have for the backups would be great and I think a great "feature / addition" for many of us ? or at least great exposure to support the Cloud anyhow ?
 
  • Like
Reactions: gurubert and Dunuin

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
8,638
1,125
174
34
Vienna
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!