Specify Dir for LVM Snapshots

[Proxygen]

The daily backups of running containers/VMs on LVM creates the LVM snapshots on the same disk (as it should), but due to the size of the machines this is using up my TBW (Total Bytes Written, which the warranty depends on) pretty fast.

This is not so much of a proxmox question, but is it possible to tell LVM to use another disk/VG to place the snapshots in (before it LZOs them)?

TIA

 

[dietmar]

I assume you talk about LVM-thin? Creating an LVM-thin snapshot does not use much space, so I am quite sure this will not use up the TBW.

 

[Proxygen]

I assume you talk about LVM-thin? Creating an LVM-thin snapshot does not use much space, so I am quite sure this will not use up the TBW.

Yep, LVM thin. Actual usage is over 800GB (1.6TB, ~50%+ in use).

At this rate the 5 yr warranty will expire in less than 1.5 yrs (the data usage is expected to double within a year).

If designating a different volume to store the snapshots is not possible, I will have to implement a differential/incremental backup (probably from within the container/VM. I'm thinking borgbackup)

 

[dcsapak]

a lvm thin snapshot initally uses no space for the data (only metadata), only the newly written blocks will use space (so this should not impact tbw at all)
vm backups do not even create storage snapshots but use a qemu internal snapshot mechanism which does not touch the vm disks...

what maybe happes is that for lvmthin, each time a new chunk has to be taken from the thinpool, it has to fully write over it with zeroes and the default size for those is 4mb afair
also, maybe something else is writing on your disks? (check with atop, iotop, etc..)

 

[Proxygen]

I took the avg of 24hr sample in 10 minutes intervals of the column KB/writ from atop.
avg is 70.86 KB written per second. That's 6.14 GB/day

TBW increases by much more than that, but I can't tell exactly how much. I just started logging TBW daily today. I will update.

I'm not very familiar with atop so I am posting the log here. Maybe my calculation is incorrect.

 

[Proxygen]

I did some more monitoring, and over the course of 72 hrs, 5.76GB was written to disk, but the TBW counter increased by 4700GB (4.7TB). This means (assuming TBW is showing the correct figure) that the applications are writing to disk then erasing (since the space used isn't increasing as fast) more than 27000 times the amount of data actually remaining on disk.

It is highly unlikely that any application is actually doing this. If it was the case, I'd see much higher I/O wait (it averages under 3% unless a backup is taking place).

While I appreciate any suggestions on figuring this out, I am going to try one thing (which will have relatively very little impact, but regardless would help): https://id-rsa.pub/post/proxmox-pve-lxc-noatime/. Is this still the proper way to mount containers with noatime?

What about vm.dirty (which, given enough RAM, would probably be capable of eliminating the problem), as in https://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performance-vm-dirty_ratio/? What are the recommended settings for proxmox?

 

[dcsapak]

I did some more monitoring, and over the course of 72 hrs, 5.76GB was written to disk, but the TBW counter increased by 4700GB (4.7TB). This means (assuming TBW is showing the correct figure) that the applications are writing to disk then erasing (since the space used isn't increasing as fast) more than 27000 times the amount of data actually remaining on disk.

if i calculate this 4700/5.76 ~ 816

this can be explained with write amplification: https://en.wikipedia.org/wiki/Write_amplification

 

[Proxygen]

not enough sleep. yes, I multiplied instead of dividing it. 816 is far from a reasonable write amplification number. An Intel employee mentions that it could go up to 10 here.

and is https://id-rsa.pub/post/proxmox-pve-lxc-noatime/ the proper way to implement noatime?

thanks dominik