Proxmox failed to backup VMs

S

samlm0

Member
Feb 16, 2021
9
0
6
25
Hello.

We are tesing backup with proxmox backup server, but it seem failed on starting backup progress

But no error when backup to local storage.

Proxmox backup server version:
1.0-8

The task log:
Code: 
INFO: starting new backup job: vzdump 100 --storage backup --compress zstd
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2021-02-16 14:55:33
INFO: status = running
INFO: VM Name: KVM100
INFO: include disk 'virtio0' 'zfs:vm-100-disk-1' 50G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/100/2021-02-16T19:55:33Z'
INFO: skipping guest-agent 'fs-freeze', agent configured but not running?
INFO: enabling encryption
ERROR: VM 100 qmp command 'backup' failed - unable to read file '/var/tmp/vzdumptmp4737_100/qemu-server.conf'
INFO: aborting backup job
ERROR: Backup of VM 100 failed - VM 100 qmp command 'backup' failed - unable to read file '/var/tmp/vzdumptmp4737_100/qemu-server.conf'
INFO: Failed at 2021-02-16 14:55:38
INFO: Backup job finished with errors

TASK ERROR: job errors

PVE Version
Code: 
proxmox-ve: 6.3-1 (running kernel: 5.4.78-2-pve)
pve-manager: 6.3-3 (running version: 6.3-3/eee5f901)
pve-kernel-5.4: 6.3-3
pve-kernel-helper: 6.3-3
pve-kernel-5.4.78-2-pve: 5.4.78-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.0-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
ksmtuned: 4.20150325+b1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.0.7
libproxmox-backup-qemu0: 1.0.2-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-3
libpve-guest-common-perl: 3.1-4
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-6
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
openvswitch-switch: residual config
proxmox-backup-client: 1.0.8-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-5
pve-cluster: 6.2-1
pve-container: 3.3-3
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-1
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-8
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-5
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.6-1~bpo10+1
 
If VM state is "stopped". the backup task send to pbs server is successful done.
Code: 
()
INFO: starting new backup job: vzdump 197 --remove 0 --node debian --mode snapshot --storage backup
INFO: Starting Backup of VM 197 (qemu)
INFO: Backup started at 2021-02-16 15:17:10
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: KVM197
INFO: include disk 'virtio0' 'zfs:vm-197-disk-0' 50G
INFO: creating Proxmox Backup Server archive 'vm/197/2021-02-16T20:17:10Z'
INFO: starting kvm to execute backup task
Use of uninitialized value in split at /usr/share/perl5/PVE/QemuServer/Cloudinit.pm line 100.
generating cloud-init ISO
INFO: enabling encryption
INFO: started backup task '871ff742-d70f-4e28-83d9-6a6cdc5663f6'
INFO: virtio0: dirty-bitmap status: created new
INFO:  14% (7.4 GiB of 50.0 GiB) in 3s, read: 2.5 GiB/s, write: 0 B/s
INFO:  29% (14.6 GiB of 50.0 GiB) in 6s, read: 2.4 GiB/s, write: 0 B/s
INFO:  43% (21.9 GiB of 50.0 GiB) in 9s, read: 2.4 GiB/s, write: 0 B/s
INFO:  57% (28.9 GiB of 50.0 GiB) in 12s, read: 2.3 GiB/s, write: 0 B/s
INFO:  72% (36.1 GiB of 50.0 GiB) in 15s, read: 2.4 GiB/s, write: 0 B/s
INFO:  86% (43.3 GiB of 50.0 GiB) in 18s, read: 2.4 GiB/s, write: 0 B/s
INFO: 100% (50.0 GiB of 50.0 GiB) in 21s, read: 2.2 GiB/s, write: 0 B/s
INFO: backup is sparse: 50.00 GiB (100%) total zero data
INFO: backup was done incrementally, reused 50.00 GiB (100%)
INFO: transferred 50.00 GiB in 22 seconds (2.3 GiB/s)
INFO: stopping kvm after backup task
trying to acquire lock...
 OK
INFO: Finished Backup of VM 197 (00:00:27)
INFO: Backup finished at 2021-02-16 15:17:37
Using encryption key from file descriptor
INFO: Backup job finished successfully
TASK OK
 
Hi,
The problem could be that you do not have the qemu-guest-agent [1] enabled in the VM, which would prevent the system from freezing the VM before making the backup. Could you enable the qemu-guest-agent with systemctl start qemu-guest-agent, and see if that fixes the issue?

[1] https://pve.proxmox.com/wiki/Qemu-guest-agent
 
dylanw said:
Hi,
The problem could be that you do not have the qemu-guest-agent [1] enabled in the VM, which would prevent the system from freezing the VM before making the backup. Could you enable the qemu-guest-agent with systemctl start qemu-guest-agent, and see if that fixes the issue?

[1] https://pve.proxmox.com/wiki/Qemu-guest-agent
Click to expand...
Hi

qemu-guest-agent did not fix the issue. This VM (VM 201) has same issue. The host as same as VM 100

Here is the task log

Code: 
INFO: starting new backup job: vzdump 201 --storage backup --mode snapshot --remove 0 --node debian
INFO: Starting Backup of VM 201 (qemu)
INFO: Backup started at 2021-02-17 07:01:29
INFO: status = running
INFO: VM Name: KVM201
INFO: include disk 'virtio0' 'zfs:vm-201-disk-0' 50G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/201/2021-02-17T12:01:29Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: enabling encryption
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 201 qmp command 'backup' failed - unable to read file '/var/tmp/vzdumptmp17416_201/qemu-server.conf'
INFO: aborting backup job
ERROR: Backup of VM 201 failed - VM 201 qmp command 'backup' failed - unable to read file '/var/tmp/vzdumptmp17416_201/qemu-server.conf'
INFO: Failed at 2021-02-17 07:01:31
INFO: Backup job finished with errors
TASK ERROR: job errors
 
Could you post the exact command that you are using and also the contents of /etc/vzdump.conf?
 
dylanw said:
Could you post the exact command that you are using and also the contents of /etc/vzdump.conf?
Click to expand...
Using Web GUI to backup VM, the backup parameters :

Code: 
Storage: backup (pbs)
Mode: Snapshot


/etc/vzdump.conf is untouch
Code: 
root@debian:~# cat /etc/vzdump.conf
# vzdump default settings

#tmpdir: /tmp
#dumpdir: DIR
#storage: STORAGE_ID
#mode: snapshot|suspend|stop
#bwlimit: KBPS
#ionice: PRI
#lockwait: MINUTES
#stopwait: MINUTES
#size: MB
#stdexcludes: BOOLEAN
#mailto: ADDRESSLIST
#maxfiles: N
#script: FILENAME
#exclude-path: PATHLIST
#pigz: N

And VM Config: qm config 201
Code: 
root@debian:~# qm config 201
acpi: 1
agent: 1
args: -vnc 127.0.0.1:10201
boot: order=ide1;virtio0
cicustom: user=pve:snippets/201_user.yaml,network=pve:snippets/201_network.yaml
citype: nocloud
cores: 1
cpu: cputype=host
cpulimit: 1
description: d1ca3de51d935c45faf03ece87425954
ide0: zfs:vm-201-cloudinit,media=cdrom
ide1: none,media=cdrom
ide2: none,media=cdrom
kvm: 1
memory: 1024
name: KVM201
net0: virtio=00:50:56:c2:c0:ec,bridge=vmbr0,firewall=1,queues=2,rate=125
onboot: 1
ostype: l26
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=056e6fc7-eb47-46b1-8905-34d063213851
sockets: 1
virtio0: zfs:vm-201-disk-1,size=50G
vmgenid: 590c076c-c307-4a6b-bccc-14435a64ea99
 
samlm0 said:
ERROR: VM 201 qmp command 'backup' failed - unable to read file '/var/tmp/vzdumptmp17416_201/qemu-server.conf'
Click to expand...
Unfortunately, this doesn't provide an obvious reason for failing. Could you try changing the tmpdir options in the /etc/vzdump.conf to a path that exists, has enough storage for the backup, and is writable? Could you also check the permissions on /var/tmp?

Could you also check the PBS is on the latest version?
Is this happening with all VMs?
 
Last edited:
dylanw said:
Unfortunately, this doesn't provide an obvious reason for failing. Could you try changing the tmpdir options in the /etc/vzdump.conf to a path that exists, has enough storage for the backup, and is writable? Could you also check the permissions on /var/tmp?

Could you also check the PBS is on the latest version?
Is this happening with all VMs?
Click to expand...
PBS was newest version
Code: 
root@backup1:~# apt list | grep proxmox-backup-server

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

proxmox-backup-server-dbgsym/stable 1.0.8-1 amd64
proxmox-backup-server/stable,now 1.0.8-1 amd64 [installed]
root@backup1:~#

We are trying to change to /tmp and still reporting same error

Is happening with all "Running" VM (Online backup), Backup offline VM is fine

Try to build new node to test backup server, to test this issue belong to pbs or pve
 
/var/tmp is writeable
Code: 
root@debian:~# stat /var/tmp
  File: /var/tmp
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: 900h/2304d      Inode: 100664365   Links: 9
Access: (1777/drwxrwxrwt)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-02-19 09:57:11.024599533 -0500
Modify: 2021-02-19 09:59:50.373319931 -0500
Change: 2021-02-19 09:59:50.373319931 -0500
 Birth: -

and left some space for backup (VM disk is 50Gb)

Code: 
root@debian:~# df -Ph /var/tmp
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        187G   33G  154G  18% /
 
Update:

Move the vzdump to another /tmp or /var/tmp locaction can solve this issue

Maybe some systemd bug?
 
You must log in or register to reply here.
Top Bottom