Oops, no VM backups because pve-root is full!?

[Wouter Samaey]

Oops, turns out I have no backups since many months due to pve-root filling up during the backup process and it eventually fails.

My disk is:
pve-root: 90 GB
pve-data: everything else

I have 1 critical VM running.

How can I address this issue safely?
Honestly, I'm a bit new to Proxmox and LVM so I'm not sure how best to proceed.

I found a script that shrinks pve-data and grows pve-root, but is that smart? The script is here: https://gist.github.com/dergachev/6828967

Of should I create a new LVM-Thin called pve-backups, mount that and try to backup there?

The backups will get huge so I'm not sure if they're best put in pve-root.

PS: I have used Proxmox a lot in the past, but an other version (one which doesn't use LVM-Thin I guess?) Never ran into this issue.

Please help!

 

[Wouter Samaey]

After doing some more research, the best way to proceed seams to be creating a new volume in pve-data, mounting that and putting backups there.

These backups are only temporary and rsynced to a NAS later.

Is this a good idea?

 

[mbaldini]

Why not backup to NAS via NFS/CIFS ?

 

[Wouter Samaey]

Why not backup to NAS via NFS/CIFS ?

I want to backup/restore quickly using my SSDs on the local machine.

I've tried snapshots, by they lock the VM (I guess I don't have the proper storage backend). Using ext4 for Proxmox instead of ZFS (only learning about ZFS now).

For now, I just want to fix the main issue and get local backups working again.

 

[Faris Raouf]

Can you add an extra drive to your system? You can then use that for local backups. It doesn't have to be SSD - you can use a large capacity HDD instead. That's what we do, anyway. Then you can add a "Directory", say "backups" and backup there.

 

[Wouter Samaey]

Can you add an extra drive to your system?

This is a great idea, but I can't since this is not my server.

Should I go ahead and create a volume in the LVM-Thin and put the backups there? I'm a bit hesitant to do it, since I don't fully know / understand if Proxmox will like this.

If so, what commands should I use? (again not familiar with LVM and/or Thin setups).

 

[Faris Raouf]

You aren't the only one confused about these things. The moment I think I have a grasp on things, I learn there's more to it, or that I was wrong about an assumption that I'd made.

With that firmly in mind:

You can't create a volume within the existing LVM-Thin as it is a volume itself. And because it is "Thin", it can only be used to store actual disk images and not anything else (e.g. cannot be used to store backups). Nor can you add a "Directory" within it.

I know you know this, but I need to start from the bottom in order to better explain what I'm trying to say:

The default arrangement is:
A Volume Group called PVE
Within this Volume Group, two Logical Volumes are created:
1) LV "root" (100GB or so in size - 90GB in your case)
2) LV "data" (all the rest of the space)

The "root" LV is a "conventional" LVM2 volume with a filesystem. It contains the OS, and all the files and directories that you can see when you login via SSH.

Proxmox has an additional layer called a "Storage". One "Storage" type is a "Directory". You can create a Directory within a non-Thin volume containing a filesystem, for example within the "root" LV. Indeed, there is one such Directory created by default, called "Local", which is stored in /var/lib/vz within the "root" LV. When you create a Directory like this via the GUI, an actual file directory is created in the file system. It is just like doing "mkdir".

Your backups are currently stored in the "Local" directory, which is in /var/lib/vz, which is in the "root" LV.

You cannot add a Directory within the "data" LV, because this is a "Thin Pool", aka LVM-Thin. It has no conventional file structure within which to add a Directory, you see.

So how do you solve your problem?

Well, LVM is supposed to make changing volume sizes easy (hah!) I expect there is a way to reduce the size of the lvm-thin volume safely, but I do not know what it might be. The script you linked to uses the command resize2fs /dev/mapper/pve-data 70G but (with my limited knowledge) there is no filesystem there to resize in modern Proxmox (older proxmox didn't use lvm-thin) so I don't see how this can work. At the bottom of the page, there are some modifications noted for proxmox 5.3 (modern). But that seems to involve removing the "data" LV (and swap). What?

Someone with real, correct and full knowledge of LVM and thin pools will hopefully step in and inform us if this is possible at all. Increasing volume sizes is relatively easy (if there is space to do so). Reducing them is more difficult because you need to make sure there's no data in the area you are reducing.

I still think the solution is to add an extra disk. If this isn't possible due to cost, then surely the only option is a slower backup to a network filesystem of some kind? OR, get a new server, partition it more to your requirements, and move everything across. Then kill off the existing server.

 

[Wouter Samaey]

Actually, I think I just solved it (but maybe not best practise).

I found a tutorial about LVM-Thin. Basically you can create a volume inside LVM Thin and that's what I did. I created a 1TB volume and formatted that as ext4, then mounted that at /var/lib/vz/dump + put it in fstab.

The new volume is now mounted and empty.

I will wait till this evening to try to create a backup.

Some ideas going through my mind are:
- Why does Proxmox not create a bigger backup disk / location by default? Or why is this not in the setup wizard as a step "Where do you want your backups?"
- Am I doing something bad here? I know it's better to backup offsite etc etc, but in a 1 server scenario, is this acceptable?
- In the future, I will go the ZFS way so I can have snapshots. This is the thing I really wanted and didn't get because I was unaware of ZFS.

 

[Faris Raouf]

Well, since you can create a disk volume within lvm-thin, which can have a filesystem, then I don't know why I thought you could not do what you have done. D'oh. Sorry.

Unfortunately proxmox is mainly aimed at multi-server clusters with remote/shared filesystems, hence the default arrangement. Those of us not using such a setup sometimes feel a little left out :-(

 

[Wouter Samaey]

Hehe, I don't mind Proxmox being larger scale oriented. I was just surprised by the newer partitioning and didn't account for issues like this. With the older versions (before LVM-Thin) you could just backup out of the box and didn't need to worry about partitioning. That was better IMHO.

Will let you know how the backup went once it's finished.

 

[kazianik]

Actually, I think I just solved it (but maybe not best practise).

I found a tutorial about LVM-Thin. Basically you can create a volume inside LVM Thin and that's what I did. I created a 1TB volume and formatted that as ext4, then mounted that at /var/lib/vz/dump + put it in fstab.

The new volume is now mounted and empty.

I will wait till this evening to try to create a backup.

Some ideas going through my mind are:
- Why does Proxmox not create a bigger backup disk / location by default? Or why is this not in the setup wizard as a step "Where do you want your backups?"
- Am I doing something bad here? I know it's better to backup offsite etc etc, but in a 1 server scenario, is this acceptable?
- In the future, I will go the ZFS way so I can have snapshots. This is the thing I really wanted and didn't get because I was unaware of ZFS.

Thanks a lot. would you please provide the link for the solution?