[SOLVED] LXC Backups to Backup Server Slow and larger than LXC Disk

B

bgsteiner

New Member
Jan 9, 2024
4
0
1
Florida, USA
I have been running into an issue with certain LXC containers on my server when backing them up to Proxmox Backup Server.
When I backup the LXC containers which have small disks from 8-24GB the backup to PBS will take an 30 minutes or more. It appears it is reading the entire LVM the LXC root disk is on. Each backup is reading about 445GB each time.

When I do a backup to an SMB share on an NAS or to directory storage on Proxmox the backup is really quick and only shows a size of the LXC of a few MB since the disks don't usually have that much data on them.

I know the actual backup will be much smaller on PBS but the backup time is the problem as it can take an entire day to backup the LXC containers on the server.
When the backups are running CPU and RAM usage is low and all packages are up to date on pve version 8.1.3

What may be causing this behavior with certain LXC containers and not others?
How can I reduce this backup time?
Should I be using different storage for LXC disks than LVM such as LVM-thin?

LXC Backup Log
Code: 
INFO: starting new backup job: vzdump 101 --remove 0 --node ds1 --mode snapshot --storage PBS1 --notification-mode auto --notes-template '{{guestname}}'
INFO: Starting Backup of VM 101 (lxc)
INFO: Backup started at 2024-01-09 10:14:51
INFO: status = running
INFO: CT Name: sdn
INFO: including mount point rootfs ('/') in backup
INFO: found old vzdump snapshot (force removal)
  Logical volume "snap_vm-101-disk-0_vzdump" successfully removed.
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
  Logical volume "snap_vm-101-disk-0_vzdump" created.
INFO: creating Proxmox Backup Server archive 'ct/101/2024-01-09T15:14:51Z'
INFO: run: lxc-usernsexec -m u:1502800000:1502810000:200000 -m g:1502800000:1502810000:200000 -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp2056880_101/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 101 --backup-time 1704813291 --repository ds1_backup@pbs@172.16.99.24:Local
INFO: Starting backup: ct/101/2024-01-09T15:14:51Z
INFO: Client name: ds1
INFO: Starting backup protocol: Tue Jan  9 10:14:54 2024
INFO: No previous manifest available.
INFO: Upload config file '/var/tmp/vzdumptmp2056880_101/etc/vzdump/pct.conf' to 'ds1_backup@pbs@172.16.99.24:8007:Local' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'ds1_backup@pbs@172.16.99.24:8007:Local' as root.pxar.didx
INFO: root.pxar: had to backup 2.707 GiB of 411.645 GiB (compressed 1.165 GiB) in 1969.02s
INFO: root.pxar: average backup speed: 1.408 MiB/s
INFO: root.pxar: backup was done incrementally, reused 408.938 GiB (99.3%)
INFO: Uploaded backup catalog (855.708 KiB)
INFO: Duration: 1969.25s
INFO: End Time: Tue Jan  9 10:47:43 2024
INFO: adding notes to backup
INFO: cleanup temporary 'vzdump' snapshot
  Logical volume "snap_vm-101-disk-0_vzdump" successfully removed.
INFO: Finished Backup of VM 101 (00:33:12)
INFO: Backup finished at 2024-01-09 10:48:03
INFO: Backup job finished successfully
INFO: notified via target `mail-to-root`
TASK OK


LXC Config
Code: 
arch: amd64
cores: 2
features: nesting=1
hostname: sdn
memory: 2048
nameserver: 172.16.99.1
net0: name=eth0,bridge=vmbr1,hwaddr=A6:1A:F9:3B:10:51,ip=dhcp,tag=99,type=veth
onboot: 1
ostype: ubuntu
rootfs: local-lvm:vm-101-disk-0,size=24G
searchdomain: orl.domain.int
startup: order=1,up=10
swap: 1024
unprivileged: 1
lxc.idmap: u 1502800000 1502810000 200000
lxc.idmap: g 1502800000 1502810000 200000
lxc.idmap: u 0 100000 65536
lxc.idmap: g 0 100000 65536

Code: 
proxmox-ve: 8.1.0 (running kernel: 6.5.11-7-pve)
pve-manager: 8.1.3 (running version: 8.1.3/b46aac3b42da5d15)
proxmox-kernel-helper: 8.1.0
pve-kernel-5.15: 7.4-7
proxmox-kernel-6.5: 6.5.11-7
proxmox-kernel-6.5.11-7-pve-signed: 6.5.11-7
proxmox-kernel-6.5.11-6-pve-signed: 6.5.11-6
proxmox-kernel-6.2.16-20-pve: 6.2.16-20
proxmox-kernel-6.2: 6.2.16-20
proxmox-kernel-6.2.16-15-pve: 6.2.16-15
pve-kernel-5.15.126-1-pve: 5.15.126-1
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx7
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.0.7
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.1.0
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.5
libpve-network-perl: 0.9.5
libpve-rs-perl: 0.8.7
libpve-storage-perl: 8.0.5
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve4
novnc-pve: 1.4.0-3
openvswitch-switch: 3.1.0-2
proxmox-backup-client: 3.1.2-1
proxmox-backup-file-restore: 3.1.2-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.2
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.1.3
pve-cluster: 8.0.5
pve-container: 5.0.8
pve-docs: 8.1.3
pve-edk2-firmware: 4.2023.08-2
pve-firewall: 5.0.3
pve-firmware: 3.9-1
pve-ha-manager: 4.0.3
pve-i18n: 3.1.5
pve-qemu-kvm: 8.1.2-6
pve-xtermjs: 5.3.0-3
qemu-server: 8.0.10
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.2-pve1
 
Last edited:
Hi,
this is an odd behavior. Could you please share your storage config by running cat /etc/pve/storage.cfg and the output of lvdisplay. Also, what is the output of df -h and mount from within the container?
 
Hello,

Here is my storage config.
Code: 
root@ds1:~# cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content iso,vztmpl
        shared 0

lvmthin: local-lvm
        thinpool data
        vgname pve
        content rootdir,images

lvm: lvm1
        vgname lvm1
        content rootdir,images
        nodes ds1
        shared 0

lvm: lvm2
        vgname lvm2
        content images,rootdir
        nodes ds1
        shared 0

cifs: nas
        path /mnt/pve/nas
        server 172.16.99.214
        share PVE_Backup
        content backup,iso
        prune-backups keep-all=1
        username pve

lvm: nvme
        vgname nvme
        content rootdir,images
        nodes lab
        shared 0

pbs: PBS1
        datastore Local
        server 172.16.99.24
        content backup
        fingerprint a6:6c:a6:af:cb:b3:5d:d2:ff:e6:12:ef:1a:74:c4:7b:62:6a:bb:d2:b6:9a:a8:12:c3:8e:fa:ba:48:cc:4e:6c
        prune-backups keep-all=1
        username ds1_backup@pbs

dir: temp
        path /mnt/pve/temp
        content rootdir,vztmpl,images,backup,iso,snippets
        is_mountpoint 1
        nodes ds1

Code: 
--- Logical volume ---
  LV Path                /dev/pve/vm-101-disk-0
  LV Name                vm-101-disk-0
  VG Name                pve
  LV UUID                MgD3OA-50Fp-M31D-77hd-9zr5-FGqo-pVkTXj
  LV Write Access        read/write
  LV Creation host, time ds1, 2023-10-13 12:13:43 -0400
  LV Pool name           data
  LV Status              available
  # open                 1
  LV Size                24.00 GiB
  Mapped size            33.82%
  Current LE             6144
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           252:27


And in the Container
Code: 
root@sdn:~# df -h
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/pve-vm--101--disk--0   24G  3.0G   20G  14% /
none                              492K  4.0K  488K   1% /dev
tmpfs                              32G     0   32G   0% /dev/shm
tmpfs                              13G  160K   13G   1% /run
tmpfs                             5.0M     0  5.0M   0% /run/lock
tmpfs                             6.3G     0  6.3G   0% /run/user/0

Code: 
root@sdn:~# mount
/dev/mapper/pve-vm--101--disk--0 on / type ext4 (rw,relatime,stripe=16)
none on /dev type tmpfs (rw,relatime,size=492k,mode=755,uid=100000,gid=100000,inode64)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
proc on /proc/sys/net type proc (rw,nosuid,nodev,noexec,relatime)
proc on /proc/sys type proc (ro,relatime)
proc on /proc/sysrq-trigger type proc (ro,relatime)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
sysfs on /sys/devices/virtual/net type sysfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
proc on /dev/.lxc/proc type proc (rw,relatime)
sys on /dev/.lxc/sys type sysfs (rw,relatime)
none on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
lxcfs on /proc/cpuinfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/diskstats type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/loadavg type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/meminfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/slabinfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/stat type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/swaps type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/uptime type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /sys/devices/system/cpu type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
udev on /dev/full type devtmpfs (rw,nosuid,relatime,size=32908920k,nr_inodes=8227230,mode=755,inode64)
udev on /dev/null type devtmpfs (rw,nosuid,relatime,size=32908920k,nr_inodes=8227230,mode=755,inode64)
udev on /dev/random type devtmpfs (rw,nosuid,relatime,size=32908920k,nr_inodes=8227230,mode=755,inode64)
udev on /dev/tty type devtmpfs (rw,nosuid,relatime,size=32908920k,nr_inodes=8227230,mode=755,inode64)
udev on /dev/urandom type devtmpfs (rw,nosuid,relatime,size=32908920k,nr_inodes=8227230,mode=755,inode64)
udev on /dev/zero type devtmpfs (rw,nosuid,relatime,size=32908920k,nr_inodes=8227230,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=100005,mode=620,ptmxmode=666,max=1026)
devpts on /dev/ptmx type devpts (rw,nosuid,noexec,relatime,gid=100005,mode=620,ptmxmode=666,max=1026)
devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=100005,mode=620,ptmxmode=666,max=1026)
devpts on /dev/tty1 type devpts (rw,nosuid,noexec,relatime,gid=100005,mode=620,ptmxmode=666,max=1026)
devpts on /dev/tty2 type devpts (rw,nosuid,noexec,relatime,gid=100005,mode=620,ptmxmode=666,max=1026)
none on /proc/sys/kernel/random/boot_id type tmpfs (ro,nosuid,nodev,noexec,relatime,size=492k,mode=755,uid=100000,gid=100000,inode64)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,uid=100000,gid=100000,inode64)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=13177036k,nr_inodes=819200,mode=755,uid=100000,gid=100000,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,uid=100000,gid=100000,inode64)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=6588516k,nr_inodes=1647129,mode=700,uid=100000,gid=100000,inode64)
 
Also here is the lvdisplay for all of the containers it was larger than the forum would allow
lvdisplay on pastebin


One container that is not showing this behavior is 302 which is a simmilar ubuntu CT running postgresql with teh root disk on local-lvm and the database mount on lvm2. Its backups are currently only about 1GB and take 30 seconds to do.
 
This does appear to be the issue. It looks like the deadly combination for me is using freeipa-client and sssd. Since the UIDs are bing remapped to work on the LXC its causing a very large and sparse /var/log/lastlog. Because proxmox backup server will handle large sparse files its taking a while to handle the zero space in the file.

I'll have to look at the best way to handle this. It looks as if editing the backup jobs or vzdump to exclude certain files may be the way to go.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom