LXC Fail to Backup

[liberodark]

Hi,

I have several proxmoxes at home. And I realized that there was a problem. That I was able to validate on several nodes. I have an LXC with 1TB + 100GB of RAM When I want to save it or take a snapshot, if it is at 10% utilization it fails it will just lock the CT and make it unavailable to me without showing me anything. And it can last a long time. I am on a 1TB x2 NVME SSD in LVM-Thin and I am backing up to PBS. If I stop the CT I can save it without worry and take snapshots without any problem.
1 Workaround is backup in mode stop but that make long unavailability.

Best regards

1 x PBS 2To
1 x Node 1To NVME x2 + 128 Go DDR4

 

[oguz]

hi,

could you please post the log of that failed/stalled backup?

if it works normally when the container is offline, then i would assume with the 100G ram the container has it can take a long time to make a snapshot so it might be stalling because of that.

can you also post the container configuration?

 

[liberodark]

Hi,

Another 10% point is only 10GB of Ram. This alone is already causing concern. The problem is that I cannot cut the CT I want it to be unavailable as short as possible.

Configuration of LXC :

arch: amd64
cores: 12
features: nesting=1
hostname: pterodactyl
memory: 102400
nameserver: 1.0.0.1
net0: name=eth0,bridge=vmbr0,gw=x.x.x.x,hwaddr=x:x:x:x:x:x,ip=x.x.x.x/32,type=veth
ostype: debian
rootfs: local-lvm:vm-100-disk-0,size=1000G
searchdomain: 1.1.1.1
swap: 2048
lxc.apparmor.profile: unconfined
lxc.cgroup.devices.allow: a
lxc.cap.drop:
lxc.mount.auto:

Configuration of VZdump :

# vzdump default settings
tmpdir: /tmp
#dumpdir: DIR
#storage: STORAGE_ID
#mode: snapshot|suspend|stop
#bwlimit: KBPS
#ionice: PRI
#lockwait: MINUTES
#stopwait: MINUTES
#size: MB
#stdexcludes: BOOLEAN
#mailto: ADDRESSLIST
#maxfiles: N
#script: FILENAME
#exclude-path: PATHLIST
#pigz: N

Log of VZdump :

INFO: starting new backup job: vzdump 100 --remove 0 --node ns3192436 --storage pbs --mode snapshot
INFO: Starting Backup of VM 100 (lxc)
INFO: Backup started at 2021-04-26 11:58:00
INFO: status = running
INFO: CT Name: pterodactyl
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'

 

[oguz]

tmpdir: /tmp

do you have enough space in /tmp to store 100G? that's where the snapshot temporary files are being held according to your vzdump configuration, so i guess your tmpfs fills up and the backup hangs? you can check this while running a backup and maybe change the tmpdir to somewhere with more space

 

[liberodark]

Have change on /var/lib/vz/tmp have 859Go
But need to test if that change anything.
My issue is lock i need to not lock longtime the CT.

 

[oguz]

you could also be running out of disk space since the container uses almost your whole disk, when you make a snapshot it could be causing the storage to get overprovisioned - so you might need more space around 1100G for it (%10 utilization)

 

[liberodark]

I had also tested on 4TB HDD I had the same problem with the same CT.
Basically my question how to save without downtime?
And the other problem is it is very very slow when I launched auto backups I had cases at 5am it was launched at 9am it still turned endless ... I can't afford to shut down the server. It's a big dilemma for me.
Except when the CT is off, I have no worries.

 

[liberodark]

Hello,
I did another test on a 4TB HDD this time in ZFS with the same CT. Still impossible to save except in stop mode. This is a real problem. if you can lean on it. Because it is not at all normal especially since there I tested SSD / HDD / LVM-Thin and ZFS. Even the snapshots we have to turn it off ... I launched a test snapshot after 6 hours, nothing has changed. On quemu or vmware this kind of problem is not present at all. And I also tested on two different Proxmox.
My last test is with proxmox 6.4 but is same

Best Regards