Correct way of taking high capacity vm disk

B

bsinha

Member
May 5, 2022
83
0
11
Hi,

I was wondering what are the best ways of taking snapshot backups of high-capacity VM disks. Is it even a standard practice?

We have lots of VMs running in our Proxmox environment. Among them, 2 VMs I want to talk about.

VM #1: It has got 2 VM disks. 100GB (For OS) and 4000GB (For application data).
VM #2: It has got 5 VM disks and increasing. 100GB (For OS) and the rest of the 4 drives are 1000GB each. All of them are configured with LVM. In total, I am getting a 4000GB volume, which stores the application data. Whenever we need to increase the storage, we add a 1000GB VM disk to the VM and add it to the existing LVM.

My questions are:
1) Is it ok to take the snapshot backups of these VMs every day? Please note, we cannot turn off the VM, since we need to maintain the uptime.
2) What can be the maximum size of a VM disk that is safe to take a snapshot backup? Does it even depend on the flavor of the operating system (Windows Server or Linux Server)?
 
Personally I think that is not the best way to architect the solution. I don't store any data on my Proxmox machines. I do all data storage via NFS either as shares mounted in the VM or through Docker volumes mounted with the NFS driver. I let TrueNAS handle all of the data security for me (including backups of my shares and Docker volumes, data scrubs, snapshots, VM backups, etc. ) This way if I need to rebuild a Proxmox node for whatever reason, it takes me 15 minutes to re-install proxmox and maybe another 30 minutes to restore my VMs. This makes it super easy to do hardware upgrades, refresh hardware, upgrade major version of proxmox, etc. Personally, I am a big fan of Proxmox, but TrueNAS does a much better job of handling the data.

That being said
1) Is it ok to take the snapshot backups of these VMs every day? Please note, we cannot turn off the VM, since we need to maintain the uptime.
Click to expand...

Sure, why not? I just wouldn't keep them all. Depending on what risks you are trying to mitigate and your RTO/RPO needs, maybe only keep 14 or so snapshots and delete the older ones. After the first snapshot, incremental snapshots will only be the size of the data that changed in the last 24 hours (based on doing one a day). Snapshots do not capture a full copy of all the data on a disk, only the changes since the last snapshot.

2) What can be the maximum size of a VM disk that is safe to take a snapshot backup? Does it even depend on the flavor of the operating system (Windows Server or Linux Server)?
Click to expand...
I am not 100% sure, but I don't think there is a practical size limitation beyond how much disk space you have. Also, do not confuse snapshots with backups. These are two very different functions in Proxmox. Backups are full copies of a VM's file system including the data, configuration, and settings, which can be stored separately from the VM (either in Proxmox or on another device like a NAS). Snapshots, are temporary copies of a VM's state at a specific point in time, including disk content and memory. They are stored on the same disk/same file system as the VM. If that disk dies, you lose your data. Backups are for long-term data protection and recovery, while snapshots are used for short-term testing, rollback, and quick recovery from minor issues.
 
  • Like
Reactions: Johannes S
bsinha said:
1) Is it ok to take the snapshot backups of these VMs every day? Please note, we cannot turn off the VM, since we need to maintain the uptime.
Click to expand...
Yes, as mentioned previously, daily snapshots are generally fine. However, depending on the rate of data changes, snapshots can grow significantly over time, potentially impacting data accessibility and system performance.

You haven't specified the type of backing storage you're using. This is important, as a snapshot of a QCOW2 disk located on NFS can introduce I/O "stutter" in the VM during the creation period which could affect your application. There is also the copy-on-write mechanism in snapshots (e.g., QCOW2), which adds additional read and write operations for every write initiated by the application.

Your question combined two different concepts: snapshots and backups. Without further detail, several important distinctions should be highlighted:

- Snapshots are not backups. If you lose your primary storage, snapshots will not help you recover the data.
- Proxmox Backup Server (PBS) does not rely on storage-level snapshots. Instead, it creates a dedicated QEMU-based snapshot, which is storage-agnostic.

If your goal is to ensure consistent backups with PBS, then the underlying storage snapshot mechanism is not relevant. PBS uses a QEMU block-dirty bitmap during backup to track changed blocks. If a block changes before it is backed up, it is temporarily held in memory or on auxiliary “fleecing” storage, ensuring a consistent backup set.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: Johannes S
For SQL databases I would even go so far that a specialiced database backup tool might be better suited in terms of consistency. But: I have no practical experience, only theoretical knowledge.

As far I know PBS doesn't have an upper size limit.

Regarding NFS instead of local storage: Managing the data might be easier but I would expect worse performance. Might still be enough though, depending on the usecase.
 
Last edited:
Johannes S said:
For SQL databases I would even go so far that a specialiced database backup tool might be better suited in terms of consistency. But: I have no practical experience, only theoretical knowledge.
Click to expand...

Following discussion might be of interest:

https://forum.restic.net/t/the-dream-mysql-backups/737/7


They use mysqldump for that though so you need enough temorary storage to save the mysqldump before backup. This is propably not feasible if a database is so large like in the OP.
 
  • Like
Reactions: bbgeek17
louie1961 said:
I am not 100% sure, but I don't think there is a practical size limitation beyond how much disk space you have. Also, do not confuse snapshots with backups. These are two very different functions in Proxmox. Backups are full copies of a VM's file system including the data, configuration, and settings, which can be stored separately from the VM (either in Proxmox or on another device like a NAS). Snapshots, are temporary copies of a VM's state at a specific point in time, including disk content and memory. They are stored on the same disk/same file system as the VM. If that disk dies, you lose your data. Backups are for long-term data protection and recovery, while snapshots are used for short-term testing, rollback, and quick recovery from minor issues.
Click to expand...
I think you got me wrong. I believe Snapshot and Snapshot backups are two different things. I meant the following actually. I wanted to know if a big VM disk (Say 4TB) can be backed up using the Proxmox backup server that it offers without facing any issue in the VM operation. 1747814032643.png

Another question is - if a 4000GB LVM is configured with 4 x1000GB VM disks in a guest linux system, and if that system gets backed up every day using Proxmox backup server without turning off the VM, will it cause any issue? Please note, later if we need to extend the LVM size, we would add another 1000GB disk, making the LVM size 5000 GB.
 
bbgeek17 said:
Yes, as mentioned previously, daily snapshots are generally fine. However, depending on the rate of data changes, snapshots can grow significantly over time, potentially impacting data accessibility and system performance.

You haven't specified the type of backing storage you're using. This is important, as a snapshot of a QCOW2 disk located on NFS can introduce I/O "stutter" in the VM during the creation period which could affect your application. There is also the copy-on-write mechanism in snapshots (e.g., QCOW2), which adds additional read and write operations for every write initiated by the application.

Your question combined two different concepts: snapshots and backups. Without further detail, several important distinctions should be highlighted:

- Snapshots are not backups. If you lose your primary storage, snapshots will not help you recover the data.
- Proxmox Backup Server (PBS) does not rely on storage-level snapshots. Instead, it creates a dedicated QEMU-based snapshot, which is storage-agnostic.

If your goal is to ensure consistent backups with PBS, then the underlying storage snapshot mechanism is not relevant. PBS uses a QEMU block-dirty bitmap during backup to track changed blocks. If a block changes before it is backed up, it is temporarily held in memory or on auxiliary “fleecing” storage, ensuring a consistent backup set.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...
I wanted to mean the snapshot backup feature of the Proxmox backup server. Let me put a scenario:

I have a VM of 4 x1000GB VM disks. The VM is configured with LVM. The size of that LVM is 4000 GB. If we need to increase the size of the LVM, we would add an additional 1000GB disk to it and make it 5000 GB.

As I know, the way Proxmox backup servers works is, it will first freeze disk 1, take the backup, then it will go for the 2nd disk, freeze it and take the backup of the 2nd disk, and so on.

Suppose there is an instance where PBS is taking the backup of the VM disk 3 (Which is one of the 4 LVM disks and provided the backup of VM disk 1 and 2 is complete already), and during that period, a 1GB file comes in to the VM for getting saved in the LVM. This file will be divided into chunks and spread across these 4 LVM disks.

Now, at this moment, the scenario is:
1) VM disk 1 has been backed up by PBS
2) VM disk 2 has been backed up by PBS
3) VM disk 3 is being backed up by PBS
4) VM disk 4 will be backed up by PBS

Let us assume the backup is complete, and for some reason, we need to restore this backup. My question is: Shall we only lose that 1GB file after the restoration, or will the entire filesystem that was running on LVM be impacted? What will be worst-case scenario here?
 
Johannes S said:
For SQL databases I would even go so far that a specialiced database backup tool might be better suited in terms of consistency. But: I have no practical experience, only theoretical knowledge.

As far I know PBS doesn't have an upper size limit.

Regarding NFS instead of local storage: Managing the data might be easier but I would expect worse performance. Might still be enough though, depending on the usecase.
Click to expand...
I would use percona xtrabackup for MySQL backup rather. But my question was never this one. I wanted to clarify a concept of how big a VM disk can be where it is safe to take a snapshot backup of by PBS. And if I use LVM, will it be impacted by the backup technology that is offered by PBS? You may refer to the scenario that I have just mentioned in comment #10.
 
You must log in or register to reply here.
Top Bottom