ERROR: interrupted by signal

devis

Member
Mar 2, 2023
45
2
13
Good afternoon, recently we have started to observe the presence of the error ERROR: interrupted by signal, when performing a backup using Proxmox Backup Server, this problem does not occur for all virtual machines, but for certain ones, these machines are not connected to each other. Moreover, the problem occurs on different hypervisors, at different times, and on different clusters. And everywhere the error is the same, we checked for the possibility of stopping the backup, for example, by the user, but users except administrators do not have access to the web interface and API, we also checked for jumps in RAM usage in the monitoring system, but memory consumption is always approximately the same. During an attempt to debug the process, we found that something forcibly terminates the backup process, the error log from the web interface as well as the processor trace are provided within the framework of this message.

Error:
Bash:
Proxmox Virtual Environment 8.4.1
Virtual Machine 100824 (laidjqznrqcurc) on node 'hv51'
INFO: starting new backup job: vzdump 100824 --mode snapshot --quiet 1 --storage bkp.cloud.local-storage1 --notes-template autobackup_hcc --prune-backups 'keep-daily=1,keep-monthly=0,keep-weekly=1'
INFO: Starting Backup of VM 100824 (qemu)
INFO: Backup started at 2025-05-27 03:15:05
INFO: status = running
INFO: VM Name: laidjqznrqcurc
INFO: include disk 'virtio0' 'onapp-utvwbhhizvjosv:vm-100824-disk-0' 355G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/100824/2025-05-27T00:15:05Z'
INFO: skipping guest-agent 'fs-freeze', agent configured but not running?
INFO: started backup task 'f7e90832-2484-49b4-9c66-df87bd0b82a5'
INFO: resuming VM again
INFO: virtio0: dirty-bitmap status: created new
INFO:   0% (768.0 MiB of 355.0 GiB) in 3s, read: 256.0 MiB/s, write: 121.3 MiB/s
INFO:   1% (3.6 GiB of 355.0 GiB) in 17s, read: 209.1 MiB/s, write: 168.3 MiB/s
INFO:   2% (7.3 GiB of 355.0 GiB) in 36s, read: 197.9 MiB/s, write: 166.9 MiB/s
INFO:   3% (10.8 GiB of 355.0 GiB) in 54s, read: 199.3 MiB/s, write: 172.4 MiB/s
ERROR: interrupted by signal
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 100824 failed - interrupted by signal
INFO: Failed at 2025-05-27 03:16:09
ERROR: Backup job failed - interrupted by signal
INFO: skipping disabled matcher 'default-matcher'
TASK ERROR: interrupted by signal

Trace:
Bash:
pselect6(24, [18], [18], NULL, {tv_sec=599, tv_nsec=998259000}, NULL) = 1 (out [18], left {tv_sec=599, tv_nsec=998256792})
write(18, "{\"arguments\":{},\"execute\":\"query"..., 61) = 61
pselect6(24, [18], [], NULL, {tv_sec=599, tv_nsec=998026000}, NULL) = 1 (in [18], left {tv_sec=599, tv_nsec=997160204})
read(18, "{\"return\": {\"total\": 64424509440"..., 8192) = 340
close(18)                               = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, {tv_sec=0, tv_nsec=502936536}) = ? ERESTART_RESTARTBLOCK (Interrupted by signal)
--- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=3379162, si_uid=0} ---
rt_sigreturn({mask=[]})                 = -1 EINTR (Interrupted system call)
rt_sigprocmask(SIG_BLOCK, [TERM], [], 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [TERM], NULL, 8) = 0
newfstatat(AT_FDCWD, "/etc/localtime", {st_mode=S_IFREG|0644, st_size=1321, ...}, 0) = 0
newfstatat(AT_FDCWD, "/etc/localtime", {st_mode=S_IFREG|0644, st_size=1321, ...}, 0) = 0
newfstatat(AT_FDCWD, "/etc/localtime", {st_mode=S_IFREG|0644, st_size=1321, ...}, 0) = 0
write(2, "ERROR: interrupted by signal\n", 29) = 29
newfstatat(AT_FDCWD, "/etc/localtime", {st_mode=S_IFREG|0644, st_size=1321, ...}, 0) = 0
newfstatat(AT_FDCWD, "/etc/localtime", {st_mode=S_IFREG|0644, st_size=1321, ...}, 0) = 0
newfstatat(AT_FDCWD, "/etc/localtime", {st_mode=S_IFREG|0644, st_size=1321, ...}, 0) = 0
write(2, "INFO: aborting backup job\n", 26) = 26

Versions:
Bash:
proxmox-ve: 8.4.0 (running kernel: 6.8.12-10-pve)
pve-manager: 8.4.1 (running version: 8.4.1/2a5fa54a8503f96d)
proxmox-kernel-helper: 8.1.1
proxmox-kernel-6.8.12-10-pve-signed: 6.8.12-10
proxmox-kernel-6.8: 6.8.12-10
proxmox-kernel-6.8.12-8-pve-signed: 6.8.12-8
ceph-fuse: 16.2.15+ds-0+deb12u1
corosync: 3.1.9-pve1
criu: 3.17.1-2+deb12u1
frr-pythontools: 10.2.2-1+pve1
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.30-pve2
libproxmox-acme-perl: 1.6.0
libproxmox-backup-qemu0: 1.5.1
libproxmox-rs-perl: 0.3.5
libpve-access-control: 8.2.2
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.1.0
libpve-cluster-perl: 8.1.0
libpve-common-perl: 8.3.1
libpve-guest-common-perl: 5.2.2
libpve-http-server-perl: 5.2.2
libpve-network-perl: 0.11.2
libpve-rs-perl: 0.9.4
libpve-storage-perl: 8.3.6
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.6.0-2
proxmox-backup-client: 3.4.1-1
proxmox-backup-file-restore: 3.4.1-1
proxmox-firewall: 0.7.1
proxmox-kernel-helper: 8.1.1
proxmox-mail-forward: 0.3.2
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.3.10
pve-cluster: 8.1.0
pve-container: 5.2.6
pve-docs: 8.4.0
pve-edk2-firmware: 4.2025.02-3
pve-esxi-import-tools: 0.7.4
pve-firewall: 5.1.1
pve-firmware: 3.15-3
pve-ha-manager: 4.0.7
pve-i18n: 3.4.2
pve-qemu-kvm: 9.2.0-5
pve-xtermjs: 5.5.0-2
qemu-server: 8.3.12
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.7-pve2
 
Hi,
anything of interest in the systemd journal of the PVE node the VM is running on at the time of the backup? Do you have high memory or io pressure? cat /proc/pressure/{memory,io,cpu}
 
Hi,
anything of interest in the systemd journal of the PVE node the VM is running on at the time of the backup? Do you have high memory or io pressure? cat /proc/pressure/{memory,io,cpu}
CPU and RAM usage is low, but IO reaches around 30% of maximum utilization during backup.
Bash:
cat /proc/pressure/{memory,io,cpu}
some avg10=0.00 avg60=0.00 avg300=0.00 total=38
full avg10=0.00 avg60=0.00 avg300=0.00 total=38
some avg10=0.00 avg60=0.00 avg300=0.00 total=1747702693
full avg10=0.00 avg60=0.00 avg300=0.00 total=1596699386
some avg10=0.34 avg60=0.40 avg300=0.40 total=3798235504
full avg10=0.00 avg60=0.00 avg300=0.00 total=0

Bash:
uptime
 16:55:09 up 7 days, 20:26,  3 users,  load average: 9.64, 9.33, 8.45

2025-05-29_16-49.png
1 hour graphics (During this period a backup was made, which received an error)Снимок экрана от 2025-05-29 16-57-36.png
At the time of backup
 
Additionally, when receiving an error, we see this on the proxmox side ve

Bash:
--- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=1037351, si_uid=0} ---
root@hv1:/tmp# cat ps_aux.txt | grep 1037351
root     1037351  0.0  0.0 379768 143324 ?       S    17:48   0:00 pvescheduler

And from the backup server side, this is the information
Bash:
2025-06-09T01:39:06+03:00: download 'index.json.blob' from previous backup 'vm/106397/2025-06-05T11:46:25Z'.
2025-06-09T01:39:06+03:00: register chunks in 'drive-scsi0.img.fidx' from previous backup 'vm/106397/2025-06-05T11:46:25Z'.
2025-06-09T01:39:06+03:00: download 'drive-scsi0.img.fidx' from previous backup 'vm/106397/2025-06-05T11:46:25Z'.
2025-06-09T01:39:06+03:00: created new fixed index 1 ("ns/BY-BTK-1/vm/106397/2025-06-08T22:39:00Z/drive-scsi0.img.fidx")
2025-06-09T01:39:06+03:00: register chunks in 'drive-scsi2.img.fidx' from previous backup 'vm/106397/2025-06-05T11:46:25Z'.
2025-06-09T01:39:06+03:00: download 'drive-scsi2.img.fidx' from previous backup 'vm/106397/2025-06-05T11:46:25Z'.
2025-06-09T01:39:07+03:00: created new fixed index 2 ("ns/BY-BTK-1/vm/106397/2025-06-08T22:39:00Z/drive-scsi2.img.fidx")
2025-06-09T01:39:07+03:00: register chunks in 'drive-scsi3.img.fidx' from previous backup 'vm/106397/2025-06-05T11:46:25Z'.
2025-06-09T01:39:07+03:00: download 'drive-scsi3.img.fidx' from previous backup 'vm/106397/2025-06-05T11:46:25Z'.
2025-06-09T01:39:07+03:00: created new fixed index 3 ("ns/BY-BTK-1/vm/106397/2025-06-08T22:39:00Z/drive-scsi3.img.fidx")
2025-06-09T01:39:07+03:00: add blob "/mnt/storage/Backups-Storage1/ns/BY-BTK-1/vm/106397/2025-06-08T22:39:00Z/qemu-server.conf.blob" (499 bytes, comp: 499)
2025-06-09T01:39:07+03:00: add blob "/mnt/storage/Backups-Storage1/ns/BY-BTK-1/vm/106397/2025-06-08T22:39:00Z/fw.conf.blob" (192 bytes, comp: 192)
2025-06-09T01:45:00+03:00: backup ended and finish failed: backup ended but finished flag is not set.
2025-06-09T01:45:00+03:00: removing unfinished backup
2025-06-09T01:45:00+03:00: removing backup snapshot "/mnt/storage/Backups-Storage1/ns/BY-BTK-1/vm/106397/2025-06-08T22:39:00Z"
2025-06-09T01:45:00+03:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: stream closed because of a broken pipe
2025-06-09T01:45:00+03:00: TASK ERROR: backup ended but finished flag is not set.
2025-06-09T01:45:00+03:00: POST /fixed_chunk: 400 Bad Request: backup already marked as finished.
 
We also tried changing the backup destination from Proxmox Backup Server to an NFS volume connected to Proxmox VE, the result is the same.
 
We also tried changing the backup destination from Proxmox Backup Server to an NFS volume connected to Proxmox VE, the result is the same.
Could you be more specific? What is the same? The error you are getting on the PVE side? Please post the full task log also for the backup using the NFS storage as backup target.

Edit: Also include the systemd journal from the PVE side from the time when the backup job fails.
 
Last edited:
Could you be more specific? What is the same? The error you are getting on the PVE side? Please post the full task log also for the backup using the NFS storage as backup target.

Edit: Also include the systemd journal from the PVE side from the time when the backup job fails.
Got the error ERROR: interrupted by signal

Here I backed up the machine both on Proxmox Backup Server and on NFS
Proxmox Backup Server:
Bash:
()
INFO: GUEST HOOK: job-init
INFO: starting new backup job: vzdump 106844 --remove 0 --storage bkp3.cloud.hoster.by-storage1 --mode snapshot --protected 1 --node hv45 --notes-template 5214941
INFO: GUEST HOOK: job-start
INFO: Starting Backup of VM 106844 (qemu)
INFO: Backup started at 2025-06-22 19:20:58
INFO: status = running
INFO: VM Name: hcc-8863
INFO: include disk 'scsi0' 'onapp-utvwbhhizvjosv:vm-106844-disk-0' 24G
INFO: include disk 'scsi1' 'onapp-bwjmfgovnrdmyy:vm-106844-disk-0' 1400G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: GUEST HOOK: backup-start snapshot 106844
INFO: Backup Hook | 2025-06-22 19:21:00 | INFO | Checking config file: /etc/.hcc.
INFO: Backup Hook | 2025-06-22 19:21:00 | INFO | Found config file: /etc/.hcc.
INFO: Backup Hook | 2025-06-22 19:21:00 | INFO | Config file /etc/.hcc loaded successfully.
INFO: Backup Hook | 2025-06-22 19:21:00 | INFO | Sending request to https://api.cloud.hoster.by/api/backups/auto/106844/backupstart.
INFO: Backup Hook | 2025-06-22 19:21:00 | INFO | Response code: 200
INFO: Backup Hook | 2025-06-22 19:21:00 | INFO | Backup state successfully sent: VMID 106844 - State backupstart.
INFO: GUEST HOOK: pre-stop snapshot 106844
INFO: GUEST HOOK: pre-restart snapshot 106844
INFO: GUEST HOOK: post-restart snapshot 106844
INFO: creating Proxmox Backup Server archive 'vm/106844/2025-06-22T16:20:58Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task 'beba9ab4-85c1-4977-99d4-56b40165b9f1'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: scsi1: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO:   0% (664.0 MiB of 1.4 TiB) in 3s, read: 221.3 MiB/s, write: 221.3 MiB/s
INFO:   1% (14.3 GiB of 1.4 TiB) in 1m 14s, read: 196.7 MiB/s, write: 196.7 MiB/s
INFO:   2% (28.5 GiB of 1.4 TiB) in 2m 29s, read: 194.0 MiB/s, write: 194.0 MiB/s
INFO:   3% (42.9 GiB of 1.4 TiB) in 3m 40s, read: 207.4 MiB/s, write: 207.4 MiB/s
INFO:   4% (57.1 GiB of 1.4 TiB) in 4m 49s, read: 211.5 MiB/s, write: 211.5 MiB/s
INFO:   5% (71.3 GiB of 1.4 TiB) in 6m 2s, read: 198.7 MiB/s, write: 198.7 MiB/s
INFO:   6% (85.5 GiB of 1.4 TiB) in 7m 15s, read: 199.4 MiB/s, write: 199.4 MiB/s
INFO:   7% (99.8 GiB of 1.4 TiB) in 8m 29s, read: 198.3 MiB/s, write: 198.3 MiB/s
INFO:   8% (114.0 GiB of 1.4 TiB) in 9m 43s, read: 196.1 MiB/s, write: 196.1 MiB/s
INFO:   9% (128.3 GiB of 1.4 TiB) in 10m 57s, read: 197.2 MiB/s, write: 197.2 MiB/s
INFO:  10% (142.5 GiB of 1.4 TiB) in 12m 11s, read: 197.4 MiB/s, write: 197.4 MiB/s
INFO:  11% (156.8 GiB of 1.4 TiB) in 13m 25s, read: 197.9 MiB/s, write: 197.9 MiB/s
INFO:  12% (171.0 GiB of 1.4 TiB) in 14m 42s, read: 188.1 MiB/s, write: 188.1 MiB/s
INFO:  13% (185.2 GiB of 1.4 TiB) in 15m 57s, read: 194.4 MiB/s, write: 194.4 MiB/s
INFO:  14% (199.5 GiB of 1.4 TiB) in 17m 11s, read: 197.6 MiB/s, write: 197.6 MiB/s
INFO:  15% (213.7 GiB of 1.4 TiB) in 18m 23s, read: 202.2 MiB/s, write: 201.4 MiB/s
INFO:  16% (227.9 GiB of 1.4 TiB) in 19m 22s, read: 246.9 MiB/s, write: 240.7 MiB/s
INFO:  17% (242.3 GiB of 1.4 TiB) in 20m 22s, read: 245.3 MiB/s, write: 238.8 MiB/s
INFO:  18% (256.5 GiB of 1.4 TiB) in 21m 25s, read: 230.4 MiB/s, write: 209.8 MiB/s
INFO:  19% (270.6 GiB of 1.4 TiB) in 22m 38s, read: 198.6 MiB/s, write: 197.3 MiB/s
INFO:  20% (284.9 GiB of 1.4 TiB) in 23m 53s, read: 194.5 MiB/s, write: 194.5 MiB/s
INFO:  21% (299.1 GiB of 1.4 TiB) in 25m 2s, read: 210.6 MiB/s, write: 210.6 MiB/s
INFO:  22% (313.3 GiB of 1.4 TiB) in 26m 8s, read: 221.3 MiB/s, write: 221.3 MiB/s
INFO:  23% (327.7 GiB of 1.4 TiB) in 27m 22s, read: 198.6 MiB/s, write: 198.6 MiB/s
INFO:  24% (341.8 GiB of 1.4 TiB) in 28m 33s, read: 203.0 MiB/s, write: 203.0 MiB/s
INFO:  25% (356.0 GiB of 1.4 TiB) in 29m 45s, read: 202.4 MiB/s, write: 202.4 MiB/s
INFO:  26% (370.4 GiB of 1.4 TiB) in 30m 57s, read: 204.9 MiB/s, write: 204.9 MiB/s
INFO:  27% (384.5 GiB of 1.4 TiB) in 32m 9s, read: 201.0 MiB/s, write: 201.0 MiB/s
INFO:  28% (398.8 GiB of 1.4 TiB) in 33m 20s, read: 204.9 MiB/s, write: 204.9 MiB/s
INFO:  29% (413.0 GiB of 1.4 TiB) in 34m 33s, read: 199.6 MiB/s, write: 199.6 MiB/s
INFO:  30% (427.3 GiB of 1.4 TiB) in 35m 47s, read: 198.4 MiB/s, write: 198.4 MiB/s
INFO:  31% (441.6 GiB of 1.4 TiB) in 37m 2s, read: 195.3 MiB/s, write: 195.3 MiB/s
INFO:  32% (455.8 GiB of 1.4 TiB) in 38m 14s, read: 201.4 MiB/s, write: 201.4 MiB/s
INFO:  33% (470.0 GiB of 1.4 TiB) in 39m 27s, read: 199.3 MiB/s, write: 199.0 MiB/s
INFO:  34% (484.3 GiB of 1.4 TiB) in 40m 43s, read: 192.2 MiB/s, write: 185.8 MiB/s
INFO:  35% (498.6 GiB of 1.4 TiB) in 41m 57s, read: 198.4 MiB/s, write: 195.7 MiB/s
INFO:  36% (512.9 GiB of 1.4 TiB) in 43m 6s, read: 212.2 MiB/s, write: 195.8 MiB/s
INFO:  37% (527.0 GiB of 1.4 TiB) in 44m 19s, read: 198.1 MiB/s, write: 196.9 MiB/s
INFO:  38% (541.2 GiB of 1.4 TiB) in 45m 36s, read: 189.3 MiB/s, write: 189.3 MiB/s
INFO:  39% (555.5 GiB of 1.4 TiB) in 46m 45s, read: 211.6 MiB/s, write: 211.6 MiB/s
INFO:  40% (569.7 GiB of 1.4 TiB) in 47m 54s, read: 210.3 MiB/s, write: 210.3 MiB/s
INFO:  41% (583.9 GiB of 1.4 TiB) in 49m 9s, read: 194.0 MiB/s, write: 194.0 MiB/s
INFO:  42% (598.1 GiB of 1.4 TiB) in 50m 25s, read: 191.3 MiB/s, write: 191.3 MiB/s
INFO:  43% (612.5 GiB of 1.4 TiB) in 51m 39s, read: 199.2 MiB/s, write: 199.2 MiB/s
INFO:  44% (626.6 GiB of 1.4 TiB) in 52m 51s, read: 200.7 MiB/s, write: 200.7 MiB/s
INFO:  45% (640.9 GiB of 1.4 TiB) in 54m 6s, read: 196.1 MiB/s, write: 196.1 MiB/s
INFO:  46% (655.2 GiB of 1.4 TiB) in 55m 21s, read: 194.5 MiB/s, write: 194.5 MiB/s
INFO:  47% (669.4 GiB of 1.4 TiB) in 56m 52s, read: 160.4 MiB/s, write: 160.4 MiB/s
INFO:  48% (683.7 GiB of 1.4 TiB) in 58m 19s, read: 167.9 MiB/s, write: 167.9 MiB/s
INFO:  49% (697.8 GiB of 1.4 TiB) in 59m 39s, read: 180.1 MiB/s, write: 180.1 MiB/s
INFO:  50% (712.1 GiB of 1.4 TiB) in 1h 1m 30s, read: 132.3 MiB/s, write: 132.3 MiB/s
INFO:  51% (726.3 GiB of 1.4 TiB) in 1h 3m 7s, read: 149.9 MiB/s, write: 148.2 MiB/s
INFO:  52% (740.6 GiB of 1.4 TiB) in 1h 4m 40s, read: 157.5 MiB/s, write: 155.4 MiB/s
INFO:  53% (754.9 GiB of 1.4 TiB) in 1h 6m 8s, read: 165.5 MiB/s, write: 163.0 MiB/s
INFO:  54% (769.0 GiB of 1.4 TiB) in 1h 7m 48s, read: 144.4 MiB/s, write: 143.9 MiB/s
ERROR: interrupted by signal
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 106844 failed - interrupted by signal
INFO: Failed at 2025-06-22 20:30:04
INFO: GUEST HOOK: backup-abort snapshot 106844
INFO: Backup Hook | 2025-06-22 20:30:05 | INFO | Checking config file: /etc/.hcc.
INFO: Backup Hook | 2025-06-22 20:30:05 | INFO | Found config file: /etc/.hcc.
INFO: Backup Hook | 2025-06-22 20:30:05 | INFO | Config file /etc/.hcc loaded successfully.
INFO: Backup Hook | 2025-06-22 20:30:05 | INFO | Sending request to https://api.cloud.hoster.by/api/backups/auto/106844/backupabort.
INFO: Backup Hook | 2025-06-22 20:30:05 | INFO | Response code: 200
INFO: Backup Hook | 2025-06-22 20:30:05 | INFO | Backup state successfully sent: VMID 106844 - State backupabort.
INFO: GUEST HOOK: log-end snapshot 106844
INFO: Backup Hook | 2025-06-22 20:30:06 | INFO | Checking config file: /etc/.hcc.
INFO: Backup Hook | 2025-06-22 20:30:06 | INFO | Found config file: /etc/.hcc.
INFO: Backup Hook | 2025-06-22 20:30:06 | INFO | Config file /etc/.hcc loaded successfully.
INFO: Backup Hook | 2025-06-22 20:30:06 | INFO | Sending request to https://api.cloud.hoster.by/api/backups/auto/106844/logend.
INFO: Backup Hook | 2025-06-22 20:30:06 | INFO | Response code: 200
INFO: Backup Hook | 2025-06-22 20:30:06 | INFO | Backup state successfully sent: VMID 106844 - State logend.
INFO: GUEST HOOK: job-abort
ERROR: Backup job failed - interrupted by signal
INFO: notified via target `mail-to-root`
TASK ERROR: interrupted by signal
 
Last edited:
NFS without Hooks
df -Th
10.0.14.40:/mnt/images nfs4 3.5T 0 3.3T 0% /mnt/pve/images.cloud.hoster.by

Bash:
()
INFO: starting new backup job: vzdump 106844 --storage images.cloud.hoster.by --notification-mode auto --remove 0 --notes-template test --compress zstd --node hv45 --mode snapshot
INFO: Starting Backup of VM 106844 (qemu)
INFO: Backup started at 2025-06-23 11:16:57
INFO: status = running
INFO: VM Name: hcc-8863
INFO: include disk 'scsi0' 'onapp-utvwbhhizvjosv:vm-106844-disk-0' 24G
INFO: include disk 'scsi1' 'onapp-bwjmfgovnrdmyy:vm-106844-disk-0' 1400G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/images.cloud.hoster.by/dump/vzdump-qemu-106844-2025_06_23-11_16_56.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task 'f5020362-0a63-4ccb-8d7b-55afdc415d6d'
INFO: resuming VM again
INFO:   0% (739.5 MiB of 1.4 TiB) in 3s, read: 246.5 MiB/s, write: 246.5 MiB/s
INFO:   1% (14.4 GiB of 1.4 TiB) in 1m 13s, read: 199.9 MiB/s, write: 199.9 MiB/s
INFO:   2% (28.6 GiB of 1.4 TiB) in 2m 30s, read: 188.5 MiB/s, write: 188.5 MiB/s
INFO:   3% (42.8 GiB of 1.4 TiB) in 3m 21s, read: 286.1 MiB/s, write: 286.1 MiB/s
INFO:   4% (57.1 GiB of 1.4 TiB) in 4m 9s, read: 305.1 MiB/s, write: 305.1 MiB/s
INFO:   5% (71.3 GiB of 1.4 TiB) in 5m 12s, read: 229.8 MiB/s, write: 229.8 MiB/s
INFO:   6% (85.5 GiB of 1.4 TiB) in 6m 16s, read: 228.5 MiB/s, write: 228.5 MiB/s
INFO:   7% (99.7 GiB of 1.4 TiB) in 7m 20s, read: 227.4 MiB/s, write: 227.4 MiB/s
INFO:   8% (114.1 GiB of 1.4 TiB) in 8m 24s, read: 230.1 MiB/s, write: 230.1 MiB/s
INFO:   9% (128.2 GiB of 1.4 TiB) in 9m 29s, read: 221.8 MiB/s, write: 221.8 MiB/s
INFO:  10% (142.5 GiB of 1.4 TiB) in 10m 35s, read: 221.5 MiB/s, write: 221.5 MiB/s
INFO:  11% (156.6 GiB of 1.4 TiB) in 11m 47s, read: 201.4 MiB/s, write: 201.4 MiB/s
INFO:  12% (170.9 GiB of 1.4 TiB) in 13m 1s, read: 197.3 MiB/s, write: 197.3 MiB/s
INFO:  13% (185.3 GiB of 1.4 TiB) in 14m 12s, read: 206.9 MiB/s, write: 206.9 MiB/s
INFO:  14% (199.5 GiB of 1.4 TiB) in 15m 26s, read: 197.7 MiB/s, write: 197.7 MiB/s
INFO:  15% (213.8 GiB of 1.4 TiB) in 16m 21s, read: 266.2 MiB/s, write: 262.1 MiB/s
INFO:  16% (228.2 GiB of 1.4 TiB) in 16m 52s, read: 473.7 MiB/s, write: 459.0 MiB/s
INFO:  17% (242.1 GiB of 1.4 TiB) in 17m 30s, read: 376.3 MiB/s, write: 361.9 MiB/s
INFO:  18% (256.3 GiB of 1.4 TiB) in 18m 31s, read: 238.4 MiB/s, write: 209.7 MiB/s
INFO:  19% (270.7 GiB of 1.4 TiB) in 19m 45s, read: 198.8 MiB/s, write: 198.8 MiB/s
INFO:  20% (284.9 GiB of 1.4 TiB) in 20m 55s, read: 208.0 MiB/s, write: 208.0 MiB/s
INFO:  21% (299.1 GiB of 1.4 TiB) in 21m 39s, read: 330.3 MiB/s, write: 330.3 MiB/s
INFO:  22% (313.4 GiB of 1.4 TiB) in 22m 33s, read: 270.7 MiB/s, write: 270.7 MiB/s
INFO:  23% (327.7 GiB of 1.4 TiB) in 23m 40s, read: 218.1 MiB/s, write: 218.1 MiB/s
INFO:  24% (341.8 GiB of 1.4 TiB) in 24m 45s, read: 222.0 MiB/s, write: 222.0 MiB/s
INFO:  25% (356.1 GiB of 1.4 TiB) in 25m 54s, read: 213.3 MiB/s, write: 213.3 MiB/s
INFO:  26% (370.4 GiB of 1.4 TiB) in 27m, read: 222.0 MiB/s, write: 222.0 MiB/s
INFO:  27% (384.7 GiB of 1.4 TiB) in 28m 4s, read: 228.0 MiB/s, write: 228.0 MiB/s
INFO:  28% (398.9 GiB of 1.4 TiB) in 29m 11s, read: 217.2 MiB/s, write: 217.2 MiB/s
INFO:  29% (413.0 GiB of 1.4 TiB) in 30m 23s, read: 200.9 MiB/s, write: 200.9 MiB/s
INFO:  30% (427.4 GiB of 1.4 TiB) in 31m 38s, read: 195.7 MiB/s, write: 195.7 MiB/s
INFO:  31% (441.5 GiB of 1.4 TiB) in 32m 51s, read: 197.9 MiB/s, write: 197.9 MiB/s
INFO:  32% (455.8 GiB of 1.4 TiB) in 34m 3s, read: 203.2 MiB/s, write: 203.2 MiB/s
INFO:  33% (470.0 GiB of 1.4 TiB) in 35m 23s, read: 182.8 MiB/s, write: 180.1 MiB/s
INFO:  34% (484.3 GiB of 1.4 TiB) in 36m 40s, read: 190.2 MiB/s, write: 187.1 MiB/s
INFO:  35% (498.5 GiB of 1.4 TiB) in 38m 1s, read: 178.6 MiB/s, write: 175.7 MiB/s
INFO:  36% (512.7 GiB of 1.4 TiB) in 39m 9s, read: 215.0 MiB/s, write: 199.0 MiB/s
INFO:  37% (527.0 GiB of 1.4 TiB) in 40m 21s, read: 202.6 MiB/s, write: 202.6 MiB/s
INFO:  38% (541.3 GiB of 1.4 TiB) in 41m 21s, read: 243.6 MiB/s, write: 243.6 MiB/s
INFO:  39% (555.7 GiB of 1.4 TiB) in 42m 3s, read: 352.4 MiB/s, write: 352.4 MiB/s
INFO:  40% (569.7 GiB of 1.4 TiB) in 43m 3s, read: 238.9 MiB/s, write: 238.9 MiB/s
INFO:  41% (583.9 GiB of 1.4 TiB) in 44m 7s, read: 226.5 MiB/s, write: 226.5 MiB/s
INFO:  42% (598.1 GiB of 1.4 TiB) in 45m 10s, read: 231.0 MiB/s, write: 231.0 MiB/s
INFO:  43% (612.4 GiB of 1.4 TiB) in 46m 13s, read: 232.9 MiB/s, write: 232.9 MiB/s
INFO:  44% (626.7 GiB of 1.4 TiB) in 47m 18s, read: 224.8 MiB/s, write: 224.8 MiB/s
INFO:  45% (641.0 GiB of 1.4 TiB) in 48m 23s, read: 225.2 MiB/s, write: 225.2 MiB/s
INFO:  46% (655.2 GiB of 1.4 TiB) in 49m 34s, read: 205.2 MiB/s, write: 205.2 MiB/s
INFO:  47% (669.4 GiB of 1.4 TiB) in 50m 48s, read: 195.7 MiB/s, write: 195.7 MiB/s
INFO:  48% (683.6 GiB of 1.4 TiB) in 52m, read: 202.6 MiB/s, write: 202.6 MiB/s
INFO:  49% (697.9 GiB of 1.4 TiB) in 53m 14s, read: 198.5 MiB/s, write: 198.5 MiB/s
INFO:  50% (712.0 GiB of 1.4 TiB) in 54m 28s, read: 195.1 MiB/s, write: 194.3 MiB/s
INFO:  51% (726.3 GiB of 1.4 TiB) in 55m 55s, read: 167.5 MiB/s, write: 164.9 MiB/s
INFO:  52% (740.6 GiB of 1.4 TiB) in 57m 26s, read: 161.1 MiB/s, write: 158.5 MiB/s
INFO:  53% (754.7 GiB of 1.4 TiB) in 58m 54s, read: 164.5 MiB/s, write: 162.2 MiB/s
INFO:  54% (769.1 GiB of 1.4 TiB) in 1h 14s, read: 183.8 MiB/s, write: 183.8 MiB/s
INFO:  55% (783.3 GiB of 1.4 TiB) in 1h 1m 36s, read: 176.9 MiB/s, write: 176.9 MiB/s
INFO:  56% (797.5 GiB of 1.4 TiB) in 1h 2m 25s, read: 297.2 MiB/s, write: 297.1 MiB/s
INFO:  57% (811.9 GiB of 1.4 TiB) in 1h 3m 22s, read: 258.6 MiB/s, write: 258.5 MiB/s
INFO:  58% (826.1 GiB of 1.4 TiB) in 1h 4m 31s, read: 210.7 MiB/s, write: 210.6 MiB/s
INFO:  59% (840.3 GiB of 1.4 TiB) in 1h 5m 43s, read: 203.1 MiB/s, write: 203.1 MiB/s
INFO:  60% (854.5 GiB of 1.4 TiB) in 1h 6m 52s, read: 210.2 MiB/s, write: 210.2 MiB/s
INFO:  61% (868.7 GiB of 1.4 TiB) in 1h 8m 1s, read: 211.2 MiB/s, write: 211.2 MiB/s
INFO:  62% (883.0 GiB of 1.4 TiB) in 1h 9m 15s, read: 197.8 MiB/s, write: 197.8 MiB/s
INFO:  63% (897.2 GiB of 1.4 TiB) in 1h 10m 28s, read: 198.4 MiB/s, write: 198.4 MiB/s
INFO:  64% (911.4 GiB of 1.4 TiB) in 1h 11m 47s, read: 184.5 MiB/s, write: 184.5 MiB/s
INFO:  65% (925.7 GiB of 1.4 TiB) in 1h 13m 9s, read: 178.9 MiB/s, write: 178.9 MiB/s
INFO:  66% (939.9 GiB of 1.4 TiB) in 1h 14m 28s, read: 183.4 MiB/s, write: 183.4 MiB/s
INFO:  67% (954.2 GiB of 1.4 TiB) in 1h 15m 49s, read: 181.5 MiB/s, write: 181.5 MiB/s
INFO:  68% (968.8 GiB of 1.4 TiB) in 1h 16m 46s, read: 262.3 MiB/s, write: 259.8 MiB/s
INFO:  69% (982.9 GiB of 1.4 TiB) in 1h 17m 20s, read: 422.8 MiB/s, write: 416.6 MiB/s
INFO:  70% (996.9 GiB of 1.4 TiB) in 1h 17m 58s, read: 377.6 MiB/s, write: 372.3 MiB/s
INFO:  71% (1011.3 GiB of 1.4 TiB) in 1h 18m 39s, read: 360.2 MiB/s, write: 355.4 MiB/s
INFO:  72% (1.0 TiB of 1.4 TiB) in 1h 19m 18s, read: 369.7 MiB/s, write: 365.2 MiB/s
INFO:  73% (1.0 TiB of 1.4 TiB) in 1h 19m 58s, read: 369.8 MiB/s, write: 365.3 MiB/s
INFO:  74% (1.0 TiB of 1.4 TiB) in 1h 20m 34s, read: 408.1 MiB/s, write: 402.7 MiB/s
INFO:  75% (1.0 TiB of 1.4 TiB) in 1h 21m 23s, read: 292.4 MiB/s, write: 289.6 MiB/s
INFO:  76% (1.1 TiB of 1.4 TiB) in 1h 22m 20s, read: 254.1 MiB/s, write: 251.2 MiB/s
ERROR: interrupted by signal
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 106844 failed - interrupted by signal
INFO: Failed at 2025-06-23 12:44:34
ERROR: Backup job failed - interrupted by signal
INFO: notified via target `mail-to-root`
TASK ERROR: interrupted by signal
 
Last edited:
ERROR: Backup of VM 106844 failed - interrupted by signal INFO: Failed at 2025-06-23 12:44:34
Your systemd journal excerpt does not cover the interesting part (the failure), please provide a longer timeframe

Edit: based on outdated information.
 
Last edited:
Your systemd journal excerpt does not cover the interesting part (the failure), please provide a longer timeframe
Please tell me what time frame you are interested in, I can provide from 11:00 to 14:00 if that works
 
Please tell me what time frame you are interested in, I can provide from 11:00 to 14:00 if that works
Best would be a window which covers the whole backup runtime, so from 11:00 to 13:00 should be enough. But also more does not hurt (if you are not restricted by upload size). Best is to attach the log as a file however.

Edit: Please ignore my request from above, since you split it into two posts i didn't see the second part in time since not having a refreshed view of the thread...
 
Last edited:
2 logs for 06/22 and 06/23, problems with intervals

in the log for 06/22 there is information about the backup on Proxmox Backup Server, in the log for 06/23 there is information about the backup on the NFS share, in both cases I received the error ERROR: interrupted by signal
 

Attachments

Last edited:
--- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=3379162, si_uid=0} ---
Can you still correlate the si_pid in the strace output above with a process id in the systemd journal from the time the trace was generated? It does not match anything in the already provided logs.
 
Can you still correlate the si_pid in the strace output above with a process id in the systemd journal from the time the trace was generated? It does not match anything in the already provided logs.
Yes, everything is correct, the trace is from another backup process, I added this to the main problem as our observations, we can re-make 2 types of backups both on Proxmox Backup Server and on NFS for this particular virtual machine and provide both the logs of these processes from syslog and trace the processes and provide the trace log.
 
Yes, everything is correct, the trace is from another backup process, I added this to the main problem as our observations, we can re-make 2 types of backups both on Proxmox Backup Server and on NFS for this particular virtual machine and provide both the logs of these processes from syslog and trace the processes and provide the trace log.
That would be great, could point to which process is sending the signal. Maybe also generate an process tree output via ps auxwf right after starting the backup job.
 
Chris

While we are preparing relevant logs and logs of tracked processes, I will supplement another observation can somehow help in the investigation of the problem, we also noticed that if you turn off all the tasks in the /etc/pve/jobs.cfg file and start the backup that constantly gets the error error: interrupted by signal - then the backup for this machine is already made without it. Errors, however, if you then turn on all the tasks in this file after that and, for example, in a day, start the backup again, then the backup begins to get an error again, at this moment, having turned off the tasks again and repeating the backup, it is performed normally. Here is the conclusion of the LS team on this file:
-rw-r----- 1 root www-data 144K Jun 25 08:15 /etc/pve/jobs.cfg
 
then the backup for this machine is already made without it.
So you have probably another backup job started by some script or the like then? The VM config is locked however during backup, so concurrent backups should not happen unless you have something forcefully removing the lock, e.g. via a qm unlock <vmid>... Check your cron jobs and systemd timers and investigate the output of ps auxwf for running backup jobs and their parent process tree, maybe that tells you were this backup job comes from...
 
I checked for duplicate tasks in the file for the virtual machine, in the file /etc/pve/jobs.cfg there are no tasks at all for the machine that serves as an example of the problem (I conduct all tests with virtual machine 106844, it constantly has an error), we start the backup through the backup menu in the section of the virtual machine itself, but we have a cluster of 20 nodes and the total number of running machines is 775, while almost all machines are backed up according to the schedule in the file /etc/pve/jobs.cfg and only 7 machines cannot perform their backup, receiving the error ERROR: interrupted by signal, for this machine that we use as a test and confirmation of the problem, there are no tasks in /etc/pve/jobs.cfg at all. And the rest of the machines that receive the error, that is, 6 machines, they are backed up according to the file /etc/pve/jobs.cfg, while the backup time is different for all of them. Only I have access to cluster management, and I do not unlock machines and do not interfere with backups on the cluster. We do not run anything on cron except for the script for running puppet

Bash:
~# crontab -l
# HEADER: This file was autogenerated at 2025-02-27 11:42:55 +0300 by puppet.
# HEADER: While it can still be managed manually, it is definitely not recommended.
# HEADER: Note particularly that the comments starting with 'Puppet Name' should
# HEADER: not be deleted, as doing so could cause duplicate cron jobs.
# Puppet Name: Cache patching data
32 * * * * /usr/local/bin/os_patching_fact_generation.sh >/dev/null 2>/dev/null
# Puppet Name: Cache patching data at reboot
@reboot /usr/local/bin/os_patching_fact_generation.sh >/dev/null 2>/dev/null
 
Last edited:
Not sure if it makes sense to share the full trace output, the idea was to identify the process which sends the signal leading to the backup failing (the full trace might also contain sensitive information). Can you identify which process it is which sends the SIGTERM signal, given by si_pid? Can you correlate this to a process of the ps output? And if so, please post corresponding section of the process tree and and excerpt from the trace surrounding the signal entry... That is of interest for the moment.