Hello.
I'm running a "pve-manager/7.3-6/723bb6ec" and since about two days while backuping my VMs the host crashes.
This is the same for backuping to a NAS as well for backuping to an internal ZFS storage in mirror mode. I got enough space on both backup targets. So this can't be the error.
To be honest I got no clue why this starts happening now.
The memory and disks are completely new so I would say it is no hardware error. And when disabling the backups I got no crashes.
Perhaps someone got an idea why this happens? Do the disks get too hot? (ZFS storage contains the VMs) I'm using Intel Enterprise SSDs.
Any help would be appreciated. Thank you very much!
Denis
I'm running a "pve-manager/7.3-6/723bb6ec" and since about two days while backuping my VMs the host crashes.
Code:
Header
Proxmox
Virtual Environment 7.3-6
Search
Node 'pve'
Week (average)
CPU usage 0.59% of 6 CPU(s)
IO delay 0.00%
Load average 0.09,0.14,0.08
RAM usage 15.84% (4.94 GiB of 31.15 GiB)
KSM sharing 0 B
/ HD space 12.66% (50.66 GiB of 400.19 GiB)
SWAP usage N/A
CPU(s) 6 x Intel(R) Core(TM) i5-8500 CPU @ 3.00GHz (1 Socket)
Kernel Version Linux 5.15.85-1-pve #1 SMP PVE 5.15.85-1 (2023-02-01T00:00Z)
PVE Manager Version pve-manager/7.3-6/723bb6ec
Repository Status Proxmox VE updates Non production-ready repository enabled!
Server View
Logs
()
INFO: starting new backup job: vzdump --notes-template '{{guestname}}' --storage synology_nas --prune-backups 'keep-last=3,keep-monthly=1,keep-weekly=1,keep-yearly=1' --quiet 1 --compress zstd --mode snapshot --exclude 100,302,401 --all 1 --mailnotification always
INFO: Starting Backup of VM 301 (qemu)
INFO: Backup started at 2023-02-27 00:00:00
INFO: status = running
INFO: VM Name: ubuntu-vm1-influxdb
INFO: include disk 'scsi0' 'local-zfs:vm-301-disk-0' 50G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/synology_nas/dump/vzdump-qemu-301-2023_02_27-00_00_00.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '464c99a2-9a94-48be-a4be-1249d30357c0'
INFO: resuming VM again
INFO: 5% (2.7 GiB of 50.0 GiB) in 3s, read: 935.7 MiB/s, write: 213.7 MiB/s
INFO: 12% (6.2 GiB of 50.0 GiB) in 6s, read: 1.2 GiB/s, write: 172.3 MiB/s
INFO: 21% (10.8 GiB of 50.0 GiB) in 9s, read: 1.5 GiB/s, write: 180.7 MiB/s
INFO: 23% (11.8 GiB of 50.0 GiB) in 12s, read: 358.9 MiB/s, write: 213.7 MiB/s
INFO: 28% (14.1 GiB of 50.0 GiB) in 15s, read: 793.9 MiB/s, write: 188.2 MiB/s
INFO: 33% (16.5 GiB of 50.0 GiB) in 18s, read: 815.1 MiB/s, write: 175.9 MiB/s
ERROR: job failed with err -5 - Input/output error
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 301 failed - job failed with err -5 - Input/output error
INFO: Failed at 2023-02-27 00:00:37
INFO: Starting Backup of VM 303 (qemu)
INFO: Backup started at 2023-02-27 00:00:37
INFO: status = running
INFO: VM Name: haos-vm1-homeassistant
INFO: include disk 'scsi0' 'local-zfs:vm-303-disk-1' 40G
INFO: include disk 'efidisk0' 'local-zfs:vm-303-disk-0' 4M
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/synology_nas/dump/vzdump-qemu-303-2023_02_27-00_00_37.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '06fd39d2-0bff-41e5-bf5b-72be4fccd64e'
INFO: resuming VM again
INFO: 2% (1.2 GiB of 40.0 GiB) in 3s, read: 404.8 MiB/s, write: 226.5 MiB/s
INFO: 4% (1.9 GiB of 40.0 GiB) in 6s, read: 239.2 MiB/s, write: 233.3 MiB/s
INFO: 6% (2.6 GiB of 40.0 GiB) in 9s, read: 227.2 MiB/s, write: 217.3 MiB/s
INFO: 7% (2.9 GiB of 40.0 GiB) in 11s, read: 163.2 MiB/s, write: 122.4 MiB/s
ERROR: job failed with err -5 - Input/output error
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 303 failed - job failed with err -5 - Input/output error
INFO: Failed at 2023-02-27 00:01:06
INFO: Backup job finished with errors
TASK ERROR: job errors
Code:
Feb 27 00:00:00 pve pvescheduler[1436066]: <root@pam> starting task UPID:pve:0015E9A3:079011E2:63FBE470:vzdump::root@pam:
Feb 27 00:00:00 pve pvescheduler[1436067]: INFO: starting new backup job: vzdump --notes-template '{{guestname}}' --storage synology_nas --prune-backups 'keep-last=3,keep-monthly=1,keep-weekly=1,keep-yearly=1' --quiet 1 --compress zstd --mode snapshot --exclude 100,302,401 --all 1 --mailnotification always
Feb 27 00:00:00 pve pvescheduler[1436067]: INFO: Starting Backup of VM 301 (qemu)
Feb 27 00:00:15 pve systemd[1]: Starting Rotate log files...
Feb 27 00:00:15 pve systemd[1]: Starting Daily man-db regeneration...
Feb 27 00:00:15 pve systemd[1]: Stopping Proxmox VE firewall logger...
Feb 27 00:00:15 pve pvefw-logger[28204]: received terminate request (signal)
Feb 27 00:00:15 pve pvefw-logger[28204]: stopping pvefw logger
Feb 27 00:00:15 pve systemd[1]: man-db.service: Succeeded.
Feb 27 00:00:15 pve systemd[1]: Finished Daily man-db regeneration.
Feb 27 00:00:15 pve systemd[1]: pvefw-logger.service: Succeeded.
Feb 27 00:00:15 pve systemd[1]: Stopped Proxmox VE firewall logger.
Feb 27 00:00:15 pve systemd[1]: pvefw-logger.service: Consumed 5.824s CPU time.
Feb 27 00:00:15 pve systemd[1]: Starting Proxmox VE firewall logger...
Feb 27 00:00:15 pve pvefw-logger[1438833]: starting pvefw logger
Feb 27 00:00:15 pve systemd[1]: Started Proxmox VE firewall logger.
Feb 27 00:00:15 pve systemd[1]: logrotate.service: Succeeded.
Feb 27 00:00:15 pve systemd[1]: Finished Rotate log files.
Feb 27 00:00:37 pve pvescheduler[1436067]: ERROR: Backup of VM 301 failed - job failed with err -5 - Input/output error
Feb 27 00:00:37 pve pvescheduler[1436067]: INFO: Starting Backup of VM 303 (qemu)
Feb 27 00:01:06 pve pvescheduler[1436067]: ERROR: Backup of VM 303 failed - job failed with err -5 - Input/output error
Feb 27 00:01:06 pve pvescheduler[1436067]: INFO: Backup job finished with errors
Feb 27 00:01:06 pve pvescheduler[1436067]: job errors
Feb 27 00:03:04 pve smartd[1304]: Device: /dev/sdb [SAT], SMART Usage Attribute: 190 Drive_Temperature changed from 63 to 62
Feb 27 00:17:01 pve CRON[1452370]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Feb 27 00:17:01 pve CRON[1452371]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Feb 27 00:17:01 pve CRON[1452370]: pam_unix(cron:session): session closed for user root
Feb 27 00:33:04 pve smartd[1304]: Device: /dev/sdb [SAT], SMART Usage Attribute: 190 Drive_Temperature changed from 62 to 63
Feb 27 00:35:15 pve systemd[1]: Starting Discard unused blocks on filesystems from /etc/fstab...
Feb 27 00:35:15 pve systemd[1]: fstrim.service: Succeeded.
Feb 27 00:35:15 pve systemd[1]: Finished Discard unused blocks on filesystems from /etc/fstab.
Feb 27 01:03:04 pve smartd[1304]: Device: /dev/sdb [SAT], SMART Usage Attribute: 190 Drive_Temperature changed from 63 to 62
-- Reboot --This is the same for backuping to a NAS as well for backuping to an internal ZFS storage in mirror mode. I got enough space on both backup targets. So this can't be the error.
To be honest I got no clue why this starts happening now.
The memory and disks are completely new so I would say it is no hardware error. And when disabling the backups I got no crashes.
Perhaps someone got an idea why this happens? Do the disks get too hot? (ZFS storage contains the VMs) I'm using Intel Enterprise SSDs.
Any help would be appreciated. Thank you very much!
Denis
Last edited:
