Proxmox: Saving Snapshots to External Drive?

[naupe]

I have a Dell PowerEdge R710 Server with Proxmox 6.4-13 installed as the Main OS. I got 4 VMs running it, one of which is an Nginx VM.

LV                                          VG  Attr       LSize   Pool Origin                                Data%  Meta%  Move Log Cpy%Sync Convert
  data                                        pve twi-aotz-- 701.80g                                            95.87  4.74
  root                                        pve -wi-ao----  96.00g
  snap_vm-100-disk-0_Backup_29Jan2021_Upgrade pve Vri---tz-k  11.00g data vm-100-disk-0
  snap_vm-101-disk-0_HTTPS_works              pve Vri---tz-k 500.00g data
  snap_vm-101-disk-0_Snapshot_27Dec2021       pve Vri---tz-k 500.00g data
  snap_vm-103-disk-0_Backup_29Jan2022         pve Vri---tz-k  50.00g data vm-103-disk-0
  snap_vm-104-disk-0_Upgrade_15June2021       pve Vri---tz-k  50.00g data vm-104-disk-0
  swap                                        pve -wi-ao----   8.00g
  vm-100-disk-0                               pve Vwi-aotz--  11.00g data                                       89.46
  vm-101-disk-0                               pve Vwi-aotz-- 500.00g data snap_vm-101-disk-0_Snapshot_27Dec2021 87.56
  vm-101-state-HTTPS_works                    pve Vwi-a-tz-- <16.50g data                                       23.24
  vm-103-disk-0                               pve Vwi-aotz--  50.00g data                                       63.32
  vm-104-disk-0                               pve Vwi-aotz--  50.00g data                                       98.88

Any recommended methods to lowering the LVM? I'm already assuming getting rid of the Snapshots should help a lot. I would really like to be able to keep my Snapshots, so I had a thought: is it possible to save Snapshots to an External Drive? I have a Mirrored Blade setup, so adding more Disks isn't a realistic solution for me.

If it is possible, then I have several questions:

1. Any recommended External Disk Drives for a Proxmox Server?
2. What would I have to do to have the External Drive to be readable by Proxmox?
3. How would I setup Snapshots to be saved on the External Drive and be read by it in the Proxmox GUI?
4. How would I setup VMs so they can be backed up to the External Drive in the Proxmox GUI? Might as well do that too.

 

[Dunuin]

If it is possible, then I have several questions:

1. Any recommended External Disk Drives for a Proxmox Server?

Make sure you don't get a SMR HDD.

1. What would I have to do to have the External Drive to be readable by Proxmox?

You can manually partition, format and mount the drive using the /etc/fstab. Then go to "Datacenter -> Storage -> Add" to add it as a storage.
Or use the WebUI to format/add it (YourNode -> Disks -> LVM/LVM-Thin/ZFS -> Create).

1. How would I setup Snapshots to be saved on the External Drive and be read by it in the Proxmox GUI?

Not sure how LVM handles snapshots, but with ZFS snapshots need to be on the same drive. Maybe someone else knows if such a thing is possible with LVM.

1. How would I setup VMs so they can be backed up to the External Drive in the Proxmox GUI? Might as well do that too.

Add a disk like described above and add it as a Storage with "Vzdump" set as content type. But keep in mind that a normal ZFS or LVM-Thin can't be used for storing backups. For that I would add a normal etx4/xfs formated partition as a "directory" storage.

 

[naupe]

Thank you for the reply Dunuin

Make sure you don't get a SMR HDD.

So a WD 4TB Elements Portable External HDD should do? It's not a very big Server, and I only suspect I'll need to backup at any one time 1.5 TB. I also plan to host Backups on a Backblaze B2 Cloud I have as a secondary backup.

You can manually partition, format and mount the drive using the /etc/fstab. Then go to "Datacenter -> Storage -> Add" to add it as a storage.
Or use the WebUI to format/add it (YourNode -> Disks -> LVM/LVM-Thin/ZFS -> Create).

I found this video on YouTube and I'm looking to follow it. Hopefully this will work. I don't have much experience with partitioning or formatting, and rather worried about screwing up Proxmox (I partitioned an old Linux PC wrong and lost everything on it ... don't want that happening again). Any videos you recommend?

Not sure how LVM handles snapshots, but with ZFS snapshots need to be on the same drive. Maybe someone else knows if such a thing is possible with LVM.

I linked this Topic to the Proxmox Reddit, and a Redditor was saying there he doesn't believe Snapshots can be saved outside the Drive the VM is saved on. A shame, but having Backups I can Restore is certainly better than nothing (and I'd only argue 1, maybe 2, of my 4 VMs have I used their Snapshots to restore).

 

[Dunuin]

So a WD 4TB Elements Portable External HDD should do? It's not a very big Server, and I only suspect I'll need to backup at any one time 1.5 TB. I also plan to host Backups on a Backblaze B2 Cloud I have as a secondary backup.

I would only buy an USB-Disk if you really know that it is not using SMR. SMR disks are cheaper than CMR disks so most USB-Disks these days below 8TB should use SMR disks. Especially 2.5" ones. And for 3.5" HDDs most manufacturers don't produce HDDs for these USB enclosures, they just put in what they got left over. So the disks used inside are always changing. If you really want to make sure that you don't get SMR disk, buy a bulk/retail SATA 3.5" HDD and an empty USB enclosure and put it together yourself.
According to some websites your WD 4TB (model "‎WDBU6Y0040BBK-WESN") is using SMR.

I linked this Topic to the Proxmox Reddit, and a Redditor was saying there he doesn't believe Snapshots can be saved outside the Drive the VM is saved on. A shame, but having Backups I can Restore is certainly better than nothing (and I'd only argue 1, maybe 2, of my 4 VMs have I used their Snapshots to restore).

I'm personally not using snapshots anymore after switching from vzdump to PBS backups. Saving backups using PBS isn't using more space than using snapshots, because of the deduplication. So it makes no big difference in used space if I keep 1 or 20 backups of the same VM. So it basically consumed the same amount of space compared to snapshots, but in contrast to snapshots I can store the backups on a cheap big HDD instead of an expensive small enterprise SSD and in case of a drive failure I got a proper backup. Snapshots won't help much in case of a drive failure, because you loose all your snapshots together with your VMs/LXCs.

 

[naupe]

Oh ... I didn't know about SMR vs CMR. So, from what I gather: CMR writes faster than SMR, both both read equally quickly?

Is there a "plug and play" CMR Drive you can recommend Dunuin? I do not have physical access to the Server needed an External Drive to backup to: I need something that the Server Owner (who isn't too Technical) can just plug into a Dell PowerEdge R710 USB Port and then I can remote in and format it.

I found this website going over CMR and SMR Devices, but all the CMR entries just lead to the Synology DS220+ 2 Bay NAS Enclosure ... which doesn't have any available Capacity options for purchase.

 

[Dunuin]

Oh ... I didn't know about SMR vs CMR. So, from what I gather: CMR writes faster than SMR, both both read equally quickly?

Jup. Reading is fast for both and SMR first look quick until you write some GB at once. As soon as the RAM-cache and CMR-cache-area gets full the SMR drives write performance drops down to low MBs/s or even kb/s and the drive becomes unresponsive so that ZFS for example thinks the drive is dead and reports errors because is disk is too slow to answer in time. Got a SMR disk here that goes from 20ms average response time to over one minute as soon as the cache gets full.

Is there a "plug and play" CMR Drive you can recommend Dunuin? I do not have physical access to the Server needed an External Drive to backup to: I need something that the Server Owner (who isn't too Technical) can just plug into a Dell PowerEdge R710 USB Port and then I can remote in and format it.

Basically all 2.5" USB disks are SMR. For 3.5" you can't be sure that they don't put a SMR drive in it if you buy a a drive below 10TB. So everything with 10TB or above should be fine because they don't produce SMR drives in that size so something like the WDBWLG0100HBK, WDBBGB0100HBK, STEB10000400 or STEL10000400 should be fine. But I guess you don't need that much capacity.

 

[naupe]

No, I don't think we need a lot, although it probably would be a good idea to get something bigger like a 6TB.

Are all the WD Elements Desktop Hard Drives (WDBWLG0060HBK-NESN) CMRs? Even if they're 4TB or 6TB?

Are all the Seagate Backup Plus Hub Desktop External Hard Drive (STEL10000400) CMRs? Even if they're 4TB or 6TB?

I'm assuming I'll just need to call both companies and get confirmation from them.

 

[Dunuin]

No, I don't think we need a lot, although it probably would be a good idea to get something bigger like a 6TB.

Are all the WD Elements Desktop Hard Drives (WDBWLG0060HBK-NESN) CMRs? Even if they're 4TB or 6TB?

Are all the Seagate Backup Plus Hub Desktop External Hard Drive (STEL10000400) CMRs? Even if they're 4TB or 6TB?

No, only the 10-18 TB ones should be definitely be CMR. As far as I know most 4 and 6 TB ones should be SMR and the newer (last 1 year) 8TB ones might also be SMR. As far as I can see there are no USB-HDDs below 10TB where you guaranteed get a CMR disk. You should check the datahoarder subreddit. They always discuss about what USB disks to buy to shuck to get cheap CMR disks. I personally shucked 8x WD 8TB disks, all the same modell number and bought at the same time and I got 4 different looking types of disks inside. They really put in whats left over at the factory. If you buy the big ones they can't put cheap SMR disks in there because they only produce CMR disks with this big sizes so you always get a more or less good disk. So its gambling if you get a SMR or CMR disk. But as I bought my 8TB disks back then they all should be CMR and after tests I could verify that they are. But that changed meanwhile so next HDD upgrade I also need to get 10TB+ disks.

 

[naupe]

I called WD last night ... and they were telling me they couldn't confirm whether any WD Elements Desktop Hard Drives (WDBWLG0060HBK-NESN) I buy would include a CMR or SMR. REALLY frustrating.

I'll check with Seagate tonight, but I suspect I'll be getting the same thing from them.

If I can't any kind of full confirmation, then I'll likely go for a WD Elements Desktop Hard Drives (WDBWLG0060HBK-NESN) 10TB just because it sounds like it's the most cost-effective option to get a CMR Plug-and-Play External Drive.

 

[Dunuin]

Jup, I was also searching the web and wasn't able to find a single USB HDD that was advertised as using CMR.
Your best bet is to look at the datahoarder subreddit where they discuss about shucking (they just buy the USB disks to remove the HDDs to get something similar to a WD Red Plus or HGST for the NAS which saves 1/3 or 2/3 of the cost compared to buying the plain SATA disks bulk or retail) and share their knowlage about it. There are some great tutorials like this one about my 8TB WD HDDs. There you can check if people report a USB disk to mainly contain CMR drives and then get one and of these and hope it is really CMR.
Then test it (CMR HDDs should report not to support trimming as only SMR disks need that and you can also run benchmarks (like writing 100GB to it) where you should see a big performance hit in case of a SMR after writing some dozens of GBs) after receiving and if it is a SMR return it and get another one.

As far as I can see WD and Seagate don't sell SMR SATA disks above 8TB so it should be very unlikely that 10+TB USB HDDs should contain SMR disks.

 

[naupe]

@Dunulin Well, I'm back and I bought a 10TB Drive ... and I can't backup to it at all.

See my new thread here.

 

[LostOnTheLine]

I would really like to be able to keep my Snapshots, so I had a thought: is it possible to save Snapshots to an External Drive? I have a Mirrored Blade setup, so adding more Disks isn't a realistic solution for me.

So you can backup snapshots, but it is not simple & not separatable.

Let me explain.

1. All snapshots are stored together in /var/lib/vz/images/#VMNUUMBER#/
   • /var/lib/vz/images/100/ will be all images in VM "100"
2. All snapshots of the same disk are saved in the same QCOW2file
   • vm-100-disk-0.qcow2 contains all images for "Disk 0"
   • vm-100-disk-1.qcow2 contains all images for "Disk 1"
3. You can save a backup of all your snapshots, but not just some of them
4. You can restore all the snapshots at the time of the backup, but you will loose any snapshots made after. You cannot mix them together

To do this kind of a backup you simply need to backup the /var/lib/vz/images/100/ folder to backup this timestopped version for VM100 right now

 

[deanfourie]

To do this kind of a backup you simply need to backup the /var/lib/vz/images/100/ folder to backup this timestopped version for VM100 right now

So, if I read this correctly,

We can just copy the entire VM directory, eg

/var/lib/vz/images/100

This is a significant way to take a entire backup of a VM for storing externally offsite, and can we restore this VM this way?

Thanks