ERROR: interrupted by signal

devis

Member
Mar 2, 2023
33
2
13
Good afternoon, recently we have started to observe the presence of the error ERROR: interrupted by signal, when performing a backup using Proxmox Backup Server, this problem does not occur for all virtual machines, but for certain ones, these machines are not connected to each other. Moreover, the problem occurs on different hypervisors, at different times, and on different clusters. And everywhere the error is the same, we checked for the possibility of stopping the backup, for example, by the user, but users except administrators do not have access to the web interface and API, we also checked for jumps in RAM usage in the monitoring system, but memory consumption is always approximately the same. During an attempt to debug the process, we found that something forcibly terminates the backup process, the error log from the web interface as well as the processor trace are provided within the framework of this message.

Error:
Bash:
Proxmox Virtual Environment 8.4.1
Virtual Machine 100824 (laidjqznrqcurc) on node 'hv51'
INFO: starting new backup job: vzdump 100824 --mode snapshot --quiet 1 --storage bkp.cloud.local-storage1 --notes-template autobackup_hcc --prune-backups 'keep-daily=1,keep-monthly=0,keep-weekly=1'
INFO: Starting Backup of VM 100824 (qemu)
INFO: Backup started at 2025-05-27 03:15:05
INFO: status = running
INFO: VM Name: laidjqznrqcurc
INFO: include disk 'virtio0' 'onapp-utvwbhhizvjosv:vm-100824-disk-0' 355G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/100824/2025-05-27T00:15:05Z'
INFO: skipping guest-agent 'fs-freeze', agent configured but not running?
INFO: started backup task 'f7e90832-2484-49b4-9c66-df87bd0b82a5'
INFO: resuming VM again
INFO: virtio0: dirty-bitmap status: created new
INFO:   0% (768.0 MiB of 355.0 GiB) in 3s, read: 256.0 MiB/s, write: 121.3 MiB/s
INFO:   1% (3.6 GiB of 355.0 GiB) in 17s, read: 209.1 MiB/s, write: 168.3 MiB/s
INFO:   2% (7.3 GiB of 355.0 GiB) in 36s, read: 197.9 MiB/s, write: 166.9 MiB/s
INFO:   3% (10.8 GiB of 355.0 GiB) in 54s, read: 199.3 MiB/s, write: 172.4 MiB/s
ERROR: interrupted by signal
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 100824 failed - interrupted by signal
INFO: Failed at 2025-05-27 03:16:09
ERROR: Backup job failed - interrupted by signal
INFO: skipping disabled matcher 'default-matcher'
TASK ERROR: interrupted by signal

Trace:
Bash:
pselect6(24, [18], [18], NULL, {tv_sec=599, tv_nsec=998259000}, NULL) = 1 (out [18], left {tv_sec=599, tv_nsec=998256792})
write(18, "{\"arguments\":{},\"execute\":\"query"..., 61) = 61
pselect6(24, [18], [], NULL, {tv_sec=599, tv_nsec=998026000}, NULL) = 1 (in [18], left {tv_sec=599, tv_nsec=997160204})
read(18, "{\"return\": {\"total\": 64424509440"..., 8192) = 340
close(18)                               = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, {tv_sec=0, tv_nsec=502936536}) = ? ERESTART_RESTARTBLOCK (Interrupted by signal)
--- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=3379162, si_uid=0} ---
rt_sigreturn({mask=[]})                 = -1 EINTR (Interrupted system call)
rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [TERM], NULL, 8) = 0
newfstatat(AT_FDCWD, "/etc/localtime", {st_mode=S_IFREG|0644, st_size=1321, ...}, 0) = 0
newfstatat(AT_FDCWD, "/etc/localtime", {st_mode=S_IFREG|0644, st_size=1321, ...}, 0) = 0
newfstatat(AT_FDCWD, "/etc/localtime", {st_mode=S_IFREG|0644, st_size=1321, ...}, 0) = 0
write(2, "ERROR: interrupted by signal\n", 29) = 29
newfstatat(AT_FDCWD, "/etc/localtime", {st_mode=S_IFREG|0644, st_size=1321, ...}, 0) = 0
newfstatat(AT_FDCWD, "/etc/localtime", {st_mode=S_IFREG|0644, st_size=1321, ...}, 0) = 0
newfstatat(AT_FDCWD, "/etc/localtime", {st_mode=S_IFREG|0644, st_size=1321, ...}, 0) = 0
write(2, "INFO: aborting backup job\n", 26) = 26

Versions:
Bash:
proxmox-ve: 8.4.0 (running kernel: 6.8.12-10-pve)
pve-manager: 8.4.1 (running version: 8.4.1/2a5fa54a8503f96d)
proxmox-kernel-helper: 8.1.1
proxmox-kernel-6.8.12-10-pve-signed: 6.8.12-10
proxmox-kernel-6.8: 6.8.12-10
proxmox-kernel-6.8.12-8-pve-signed: 6.8.12-8
ceph-fuse: 16.2.15+ds-0+deb12u1
corosync: 3.1.9-pve1
criu: 3.17.1-2+deb12u1
frr-pythontools: 10.2.2-1+pve1
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.30-pve2
libproxmox-acme-perl: 1.6.0
libproxmox-backup-qemu0: 1.5.1
libproxmox-rs-perl: 0.3.5
libpve-access-control: 8.2.2
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.1.0
libpve-cluster-perl: 8.1.0
libpve-common-perl: 8.3.1
libpve-guest-common-perl: 5.2.2
libpve-http-server-perl: 5.2.2
libpve-network-perl: 0.11.2
libpve-rs-perl: 0.9.4
libpve-storage-perl: 8.3.6
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.6.0-2
proxmox-backup-client: 3.4.1-1
proxmox-backup-file-restore: 3.4.1-1
proxmox-firewall: 0.7.1
proxmox-kernel-helper: 8.1.1
proxmox-mail-forward: 0.3.2
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.3.10
pve-cluster: 8.1.0
pve-container: 5.2.6
pve-docs: 8.4.0
pve-edk2-firmware: 4.2025.02-3
pve-esxi-import-tools: 0.7.4
pve-firewall: 5.1.1
pve-firmware: 3.15-3
pve-ha-manager: 4.0.7
pve-i18n: 3.4.2
pve-qemu-kvm: 9.2.0-5
pve-xtermjs: 5.5.0-2
qemu-server: 8.3.12
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.7-pve2
 
Hi,
anything of interest in the systemd journal of the PVE node the VM is running on at the time of the backup? Do you have high memory or io pressure? cat /proc/pressure/{memory,io,cpu}
 
Hi,
anything of interest in the systemd journal of the PVE node the VM is running on at the time of the backup? Do you have high memory or io pressure? cat /proc/pressure/{memory,io,cpu}
CPU and RAM usage is low, but IO reaches around 30% of maximum utilization during backup.
Bash:
cat /proc/pressure/{memory,io,cpu}
some avg10=0.00 avg60=0.00 avg300=0.00 total=38
full avg10=0.00 avg60=0.00 avg300=0.00 total=38
some avg10=0.00 avg60=0.00 avg300=0.00 total=1747702693
full avg10=0.00 avg60=0.00 avg300=0.00 total=1596699386
some avg10=0.34 avg60=0.40 avg300=0.40 total=3798235504
full avg10=0.00 avg60=0.00 avg300=0.00 total=0

Bash:
uptime
 16:55:09 up 7 days, 20:26,  3 users,  load average: 9.64, 9.33, 8.45

2025-05-29_16-49.png
1 hour graphics (During this period a backup was made, which received an error)Снимок экрана от 2025-05-29 16-57-36.png
At the time of backup