I'm trying to backup containers to a PBS server but the backup seems to hang after uploading one container backup. It gets stuck after
I've tried letting it run for a day but it made no progress. CPU usage on the backup server is low.
While the backup job hangs, the PBS web interface gives a connection error overlay. Restarting the proxmox-backup-proxy service lets the task continue:
Is there anything I can do to debug this?
INFO: End Time: 2020-08-08T15:14:58+02:00I've tried letting it run for a day but it made no progress. CPU usage on the backup server is low.
Code:
INFO: starting new backup job: vzdump 131 132 135 133 140 100 124 --quiet 1 --all 0 --mailnotification failure --mode snapshot --compress zstd --node pistol --storage zombie-internal1 --mailto <removed>
INFO: Starting Backup of VM 124 (lxc)
INFO: Backup started at 2020-08-08 14:31:03
INFO: status = running
INFO: CT Name: <removed>
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
/dev/rbd6
INFO: creating Proxmox Backup Server archive 'ct/124/2020-08-08T12:31:03Z'
INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=encrypt --keyfd=14 pct.conf:/var/tmp/vzdumptmp1984578/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --backup-type ct --backup-id 124 --backup-time 1596889863 --repository <removed>@pbs@<removed>:internal1
INFO: Starting backup: ct/124/2020-08-08T12:31:03Z
INFO: Client name: pistol
INFO: Starting protocol: 2020-08-08T14:31:05+02:00
INFO: Upload config file '/var/tmp/vzdumptmp1984578/etc/vzdump/pct.conf' to '<removed>@pbs@<removed>:internal1' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to '<removed>@pbs@<removed>:internal1' as root.pxar.didx
INFO: root.pxar: had to upload 39.45 MiB of 93.68 GiB in 2631.80s, avgerage speed 15.35 KiB/s).
INFO: root.pxar: backup was done incrementally, reused 93.65 GiB (100.0%)
INFO: Uploaded backup catalog (1.10 MiB)
INFO: Duration: PT2632.556921212S
INFO: End Time: 2020-08-08T15:14:58+02:00
Code:
Aug 08 15:14:57 zombie proxmox-backup-proxy[23074]: server checksum [109, 63, 118, 77, 128, 77, 176, 161, 252, 57, 167, 184, 173, 50, 223, 136, 22, 30, 134, 154, 139, 221, 104,
Aug 08 15:14:57 zombie proxmox-backup-proxy[23074]: Upload statistics for 'root.pxar.didx'
Aug 08 15:14:57 zombie proxmox-backup-proxy[23074]: UUID: 98aba390fc164b3aaeeeb2ab5dddfdb8
Aug 08 15:14:57 zombie proxmox-backup-proxy[23074]: Checksum: 6d3f764d804db0a1fc39a7b8ad32df88161e869a8bdd68dd8fe9ebab5fc90209
Aug 08 15:14:57 zombie proxmox-backup-proxy[23074]: Size: 100592722923
Aug 08 15:14:57 zombie proxmox-backup-proxy[23074]: Chunk count: 25623
Aug 08 15:14:57 zombie proxmox-backup-proxy[23074]: Upload size: 41365581 (0%)
Aug 08 15:14:57 zombie proxmox-backup-proxy[23074]: Duplicates: 25616+0 (99%)
Aug 08 15:14:57 zombie proxmox-backup-proxy[23074]: Compression: 46%
Aug 08 15:14:57 zombie proxmox-backup-proxy[23074]: successfully closed dynamic index 2
Aug 08 15:14:58 zombie proxmox-backup-proxy[23074]: server checksum [40, 63, 164, 240, 153, 202, 72, 215, 152, 170, 222, 155, 10, 40, 202, 229, 197, 167, 184, 32, 196, 211, 67,
Aug 08 15:14:58 zombie proxmox-backup-proxy[23074]: Upload statistics for 'catalog.pcat1.didx'
Aug 08 15:14:58 zombie proxmox-backup-proxy[23074]: UUID: c967099f9d284c17a0313e1ead9d6e4d
Aug 08 15:14:58 zombie proxmox-backup-proxy[23074]: Checksum: 283fa4f099ca48d798aade9b0a28cae5c5a7b820c4d3436825bc5afcf2e62fe7
Aug 08 15:14:58 zombie proxmox-backup-proxy[23074]: Size: 1156072
Aug 08 15:14:58 zombie proxmox-backup-proxy[23074]: Chunk count: 4
Aug 08 15:14:58 zombie proxmox-backup-proxy[23074]: Upload size: 1156072 (100%)
Aug 08 15:14:58 zombie proxmox-backup-proxy[23074]: Duplicates: 0+2 (50%)
Aug 08 15:14:58 zombie proxmox-backup-proxy[23074]: Compression: 31%
Aug 08 15:14:58 zombie proxmox-backup-proxy[23074]: successfully closed dynamic index 1
Aug 08 15:14:58 zombie proxmox-backup-proxy[23074]: add blob "/internal1/ct/124/2020-08-08T12:31:03Z/index.json.blob" (411 bytes, comp: 411)
Aug 08 15:14:58 zombie proxmox-backup-proxy[23074]: successfully finished backup
Aug 08 15:14:58 zombie proxmox-backup-proxy[23074]: backup finished successfully
Aug 08 15:14:58 zombie proxmox-backup-proxy[23074]: TASK OK
Aug 08 15:14:58 zombie proxmox-backup-proxy[23074]: Detected stopped UPID UPID:zombie:00005A22:00D31DDA:00000001:5F2E9B09:backup:internal1_ct_124:<removed>@pbs:
Aug 08 15:14:58 zombie proxmox-backup-proxy[23074]: unregister workerWhile the backup job hangs, the PBS web interface gives a connection error overlay. Restarting the proxmox-backup-proxy service lets the task continue:
Code:
INFO: remove vzdump snapshot
2020-08-08 17:15:18.211 7f23ceffd700 -1 librbd::object_map::InvalidateRequest: 0x7f23d8008360 should_complete: r=0
Removing snap: 100% complete...done.
INFO: Finished Backup of VM 124 (02:44:16)
INFO: Backup finished at 2020-08-08 17:15:19
INFO: Starting Backup of VM 131 (lxc)
...Is there anything I can do to debug this?
Code:
root@shandy:~# pveversion -v
proxmox-ve: 6.2-1 (running kernel: 5.4.44-2-pve)
pve-manager: 6.2-10 (running version: 6.2-10/a20769ed)
pve-kernel-5.4: 6.2-4
pve-kernel-helper: 6.2-4
pve-kernel-5.3: 6.1-6
pve-kernel-5.0: 6.0-11
pve-kernel-5.4.44-2-pve: 5.4.44-2
pve-kernel-5.4.44-1-pve: 5.4.44-1
pve-kernel-5.4.41-1-pve: 5.4.41-1
pve-kernel-4.15: 5.4-6
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.13-1-pve: 5.3.13-1
pve-kernel-5.3.10-1-pve: 5.3.10-1
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.21-3-pve: 5.0.21-7
pve-kernel-5.0.21-2-pve: 5.0.21-7
pve-kernel-4.15.18-18-pve: 4.15.18-44
pve-kernel-4.13.13-2-pve: 4.13.13-33
ceph: 14.2.10-pve1
ceph-fuse: 14.2.10-pve1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve2
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.4
libpve-access-control: 6.1-2
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-5
libpve-guest-common-perl: 3.1-2
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-5
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
openvswitch-switch: 2.12.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-9
pve-cluster: 6.1-8
pve-container: 3.1-12
pve-docs: 6.2-5
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-3
pve-qemu-kvm: 5.0.0-11
pve-xtermjs: 4.3.0-1
qemu-server: 6.2-11
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.4-pve1
Last edited:
