[SOLVED] Backup to PBS failed

ukro

Member
May 16, 2021
122
13
23
38
Any idea why this one VM out of 9 in total had this error? Previously for 8days it was working okay.
When i try manualy backup to PBS its still giving error. All other VMs normaly backedup.
Manualy backup to different location (NON PBS) of this one VM work okay but to PBS no.

Code:
INFO: starting new backup job: vzdump 102 --mode snapshot --notes-template '{{guestname}}' --storage pbs --remove 0 --node node
INFO: Starting Backup of VM 102 (qemu)
INFO: Backup started at 2023-02-10 22:10:57
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: win10
INFO: include disk 'virtio0' 'vms:vm-102-disk-1' 70G
INFO: include disk 'efidisk0' 'vms:vm-102-disk-0' 1M
INFO: creating Proxmox Backup Server archive 'vm/102/2023-02-10T21:10:57Z'
INFO: starting kvm to execute backup task
ERROR: Backup of VM 102 failed - start failed: command '/usr/bin/kvm -id 102 -name 'win10,debug-threads=on' -no-shutdown -chardev 'socket,id=qmp,path=/var/run/qemu-server/102.qmp,server=on,wait=off' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/102.pid -daemonize -smbios 'type=1,uuid=079631b6-1c98-4e2d-ae5a-287a0845cfd3' -drive 'if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CODE_4M.secboot.fd' -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,size=540672,file=/dev/zvol/vms/vm-102-disk-0' -smp '4,sockets=1,cores=4,maxcpus=4' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc 'unix:/var/run/qemu-server/102.vnc,password=on' -no-hpet -cpu 'host,hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff,hv_stimer,hv_synic,hv_time,hv_vapic,hv_vpindex,kvm=off,+kvm_pv_eoi,+kvm_pv_unhalt,+pcid' -m 8128 -object 'iothread,id=iothread-virtio0' -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg -device 'vmgenid,guid=162955d9-f3fd-47c6-b8fd-c278dd96040d' -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=0000:03:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on,romfile=/usr/share/kvm/gt7101gnew.bin' -device 'vfio-pci,host=0000:03:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1' -device 'VGA,id=vga,bus=pcie.0,addr=0x1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:b61aadf3b61e' -drive 'file=/dev/zvol/vms/vm-102-disk-1,if=none,id=drive-virtio0,format=raw,cache=none,aio=io_uring,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,iothread=iothread-virtio0,bootindex=100' -device 'ahci,id=ahci0,multifunction=on,bus=pci.0,addr=0x7' ,if=none,id=drive-sata0,media=cdrom,aio=io_uring' -device 'ide-cd,bus=ahci0.0,drive=drive-sata0,id=sata0,bootindex=102' -netdev 'type=tap,id=net0,ifname=tap102i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=3E:A9:68:F3:78:28,netdev=net0,bus=pci.0,addr=0x12,id=net0' -netdev 'type=tap,id=net1,ifname=tap102i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=EE:6A:4C:D3:19:AE,netdev=net1,bus=pci.0,addr=0x13,id=net1' -rtc 'driftfix=slew,base=localtime' -machine 'type=pc-q35-6.2+pve0' -global 'kvm-pit.lost_tick_policy=discard' -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off' -machine 'max-ram-below-4g=1G,kernel_irqchip=on' -S' failed: got timeout
INFO: Failed at 2023-02-10 22:11:28
INFO: Backup job finished with errors
TASK ERROR: job errors

P.S. If its important this VM is turned off
 
And now it just worked, IDK....
Code:
INFO: starting new backup job: vzdump 102 --node node --storage pbs --notes-template '{{guestname}}' --remove 0 --mode snapshot
INFO: Starting Backup of VM 102 (qemu)
INFO: Backup started at 2023-02-10 22:25:33
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: win10
INFO: include disk 'virtio0' 'vms:vm-102-disk-1' 70G
INFO: include disk 'efidisk0' 'vms:vm-102-disk-0' 1M
INFO: creating Proxmox Backup Server archive 'vm/102/2023-02-10T21:25:33Z'
INFO: starting kvm to execute backup task
INFO: started backup task '5d322c12-b9d9-4415-9f0e-05602c3b6c85'
INFO: efidisk0: dirty-bitmap status: created new
INFO: virtio0: dirty-bitmap status: created new
INFO:   1% (884.0 MiB of 70.0 GiB) in 3s, read: 294.7 MiB/s, write: 0 B/s
INFO:   2% (1.5 GiB of 70.0 GiB) in 6s, read: 233.3 MiB/s, write: 0 B/s
INFO:   3% (2.2 GiB of 70.0 GiB) in 9s, read: 237.3 MiB/s, write: 0 B/s
INFO:   4% (2.9 GiB of 70.0 GiB) in 12s, read: 226.7 MiB/s, write: 0 B/s
INFO:   5% (3.6 GiB of 70.0 GiB) in 15s, read: 237.3 MiB/s, write: 0 B/s
INFO:   6% (4.3 GiB of 70.0 GiB) in 18s, read: 237.3 MiB/s, write: 0 B/s
INFO:   7% (5.0 GiB of 70.0 GiB) in 21s, read: 236.0 MiB/s, write: 0 B/s
INFO:   8% (5.7 GiB of 70.0 GiB) in 24s, read: 234.7 MiB/s, write: 0 B/s
INFO:   9% (6.3 GiB of 70.0 GiB) in 27s, read: 224.0 MiB/s, write: 0 B/s
INFO:  10% (7.2 GiB of 70.0 GiB) in 31s, read: 225.0 MiB/s, write: 0 B/s
INFO:  11% (7.9 GiB of 70.0 GiB) in 34s, read: 232.0 MiB/s, write: 0 B/s
INFO:  12% (8.6 GiB of 70.0 GiB) in 37s, read: 226.7 MiB/s, write: 0 B/s
INFO:  13% (9.2 GiB of 70.0 GiB) in 40s, read: 232.0 MiB/s, write: 0 B/s
INFO:  14% (9.9 GiB of 70.0 GiB) in 43s, read: 217.3 MiB/s, write: 0 B/s
INFO:  15% (10.6 GiB of 70.0 GiB) in 46s, read: 236.0 MiB/s, write: 0 B/s
INFO:  16% (11.3 GiB of 70.0 GiB) in 49s, read: 241.3 MiB/s, write: 0 B/s
INFO:  17% (11.9 GiB of 70.0 GiB) in 52s, read: 225.3 MiB/s, write: 0 B/s
INFO:  18% (12.6 GiB of 70.0 GiB) in 56s, read: 179.0 MiB/s, write: 0 B/s
INFO:  19% (13.5 GiB of 70.0 GiB) in 59s, read: 310.7 MiB/s, write: 0 B/s
INFO:  20% (14.2 GiB of 70.0 GiB) in 1m 2s, read: 225.3 MiB/s, write: 0 B/s
INFO:  24% (16.9 GiB of 70.0 GiB) in 1m 5s, read: 908.0 MiB/s, write: 0 B/s
INFO:  25% (17.6 GiB of 70.0 GiB) in 1m 8s, read: 236.0 MiB/s, write: 0 B/s
INFO:  26% (18.2 GiB of 70.0 GiB) in 1m 11s, read: 228.0 MiB/s, write: 0 B/s
INFO:  27% (18.9 GiB of 70.0 GiB) in 1m 14s, read: 238.7 MiB/s, write: 0 B/s
INFO:  28% (19.7 GiB of 70.0 GiB) in 1m 17s, read: 276.0 MiB/s, write: 0 B/s
INFO:  29% (20.4 GiB of 70.0 GiB) in 1m 20s, read: 230.7 MiB/s, write: 0 B/s
INFO:  30% (21.1 GiB of 70.0 GiB) in 1m 23s, read: 233.3 MiB/s, write: 0 B/s
INFO:  31% (21.7 GiB of 70.0 GiB) in 1m 26s, read: 225.3 MiB/s, write: 0 B/s
INFO:  32% (22.6 GiB of 70.0 GiB) in 1m 30s, read: 211.0 MiB/s, write: 0 B/s
INFO:  33% (23.2 GiB of 70.0 GiB) in 1m 33s, read: 210.7 MiB/s, write: 0 B/s
INFO:  34% (23.9 GiB of 70.0 GiB) in 1m 36s, read: 226.7 MiB/s, write: 0 B/s
INFO:  35% (24.7 GiB of 70.0 GiB) in 1m 40s, read: 217.0 MiB/s, write: 0 B/s
INFO:  36% (25.3 GiB of 70.0 GiB) in 1m 43s, read: 210.7 MiB/s, write: 0 B/s
INFO:  37% (25.9 GiB of 70.0 GiB) in 1m 46s, read: 210.7 MiB/s, write: 0 B/s
INFO:  38% (26.8 GiB of 70.0 GiB) in 1m 50s, read: 219.0 MiB/s, write: 0 B/s
INFO:  39% (27.4 GiB of 70.0 GiB) in 1m 53s, read: 225.3 MiB/s, write: 0 B/s
INFO:  40% (28.1 GiB of 70.0 GiB) in 1m 56s, read: 216.0 MiB/s, write: 0 B/s
INFO:  41% (28.7 GiB of 70.0 GiB) in 1m 59s, read: 216.0 MiB/s, write: 0 B/s
INFO:  42% (29.6 GiB of 70.0 GiB) in 2m 3s, read: 220.0 MiB/s, write: 0 B/s
INFO:  43% (30.7 GiB of 70.0 GiB) in 2m 6s, read: 392.0 MiB/s, write: 0 B/s
INFO:  44% (31.4 GiB of 70.0 GiB) in 2m 9s, read: 224.0 MiB/s, write: 0 B/s
INFO:  45% (32.0 GiB of 70.0 GiB) in 2m 12s, read: 216.0 MiB/s, write: 0 B/s
INFO:  46% (32.6 GiB of 70.0 GiB) in 2m 15s, read: 217.3 MiB/s, write: 0 B/s
INFO:  47% (33.3 GiB of 70.0 GiB) in 2m 18s, read: 226.7 MiB/s, write: 0 B/s
INFO:  48% (33.9 GiB of 70.0 GiB) in 2m 21s, read: 217.3 MiB/s, write: 0 B/s
INFO:  49% (34.6 GiB of 70.0 GiB) in 2m 24s, read: 222.7 MiB/s, write: 0 B/s
INFO:  50% (35.3 GiB of 70.0 GiB) in 2m 27s, read: 228.0 MiB/s, write: 0 B/s
INFO:  51% (35.9 GiB of 70.0 GiB) in 2m 30s, read: 220.0 MiB/s, write: 0 B/s
INFO:  52% (36.6 GiB of 70.0 GiB) in 2m 33s, read: 221.3 MiB/s, write: 0 B/s
INFO:  53% (37.3 GiB of 70.0 GiB) in 2m 36s, read: 241.3 MiB/s, write: 0 B/s
INFO:  54% (37.9 GiB of 70.0 GiB) in 2m 39s, read: 229.3 MiB/s, write: 0 B/s
INFO:  55% (38.6 GiB of 70.0 GiB) in 2m 42s, read: 230.7 MiB/s, write: 0 B/s
INFO:  56% (39.3 GiB of 70.0 GiB) in 2m 45s, read: 236.0 MiB/s, write: 0 B/s
INFO:  57% (40.0 GiB of 70.0 GiB) in 2m 48s, read: 236.0 MiB/s, write: 0 B/s
INFO:  58% (40.7 GiB of 70.0 GiB) in 2m 51s, read: 246.7 MiB/s, write: 0 B/s
INFO:  61% (43.2 GiB of 70.0 GiB) in 2m 54s, read: 829.3 MiB/s, write: 0 B/s
INFO:  75% (52.8 GiB of 70.0 GiB) in 2m 58s, read: 2.4 GiB/s, write: 0 B/s
INFO:  90% (63.1 GiB of 70.0 GiB) in 3m 1s, read: 3.4 GiB/s, write: 0 B/s
INFO:  99% (69.4 GiB of 70.0 GiB) in 3m 4s, read: 2.1 GiB/s, write: 0 B/s
INFO: 100% (70.0 GiB of 70.0 GiB) in 3m 7s, read: 210.8 MiB/s, write: 0 B/s
INFO: Waiting for server to finish backup validation...
INFO: backup is sparse: 32.11 GiB (45%) total zero data
INFO: backup was done incrementally, reused 70.00 GiB (100%)
INFO: transferred 70.00 GiB in 190 seconds (377.3 MiB/s)
INFO: stopping kvm after backup task
INFO: adding notes to backup
storing login ticket failed: $XDG_RUNTIME_DIR must be set
INFO: Finished Backup of VM 102 (00:03:41)
INFO: Backup finished at 2023-02-10 22:29:14
storing login ticket failed: $XDG_RUNTIME_DIR must be set
INFO: Backup job finished successfully
TASK OK
 
The automatic backup failed again, will turn on the VM and report back
EDIT:
Okay i can't turn on the VM, help please <3
Code:
Web searchCopy
TASK ERROR: start failed: command '/usr/bin/kvm -id 102 -name 'win10,debug-threads=on' -no-shutdown -chardev 'socket,id=qmp,path=/var/run/qemu-server/102.qmp,server=on,wait=off' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/102.pid -daemonize -smbios 'type=1,uuid=079631b6-1c98-4e2d-ae5a-287a0845cfd3' -drive 'if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CODE_4M.secboot.fd' -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,size=540672,file=/dev/zvol/vms/vm-102-disk-0' -smp '4,sockets=1,cores=4,maxcpus=4' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc 'unix:/var/run/qemu-server/102.vnc,password=on' -no-hpet -cpu 'host,hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff,hv_stimer,hv_synic,hv_time,hv_vapic,hv_vpindex,kvm=off,+kvm_pv_eoi,+kvm_pv_unhalt,+pcid' -m 8128 -object 'iothread,id=iothread-virtio0' -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg -device 'vmgenid,guid=162955d9-f3fd-47c6-b8fd-c278dd96040d' -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=0000:03:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on,romfile=/usr/share/kvm/gt7101gnew.bin' -device 'vfio-pci,host=0000:03:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1' -device 'VGA,id=vga,bus=pcie.0,addr=0x1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:b61aadf3b61e' -drive 'file=/dev/zvol/vms/vm-102-disk-1,if=none,id=drive-virtio0,format=raw,cache=none,aio=io_uring,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,iothread=iothread-virtio0,bootindex=100' -device 'ahci,id=ahci0,multifunction=on,bus=pci.0,addr=0x7' -device 'ide-cd,bus=ahci0.0,drive=drive-sata0,id=sata0,bootindex=102' -netdev 'type=tap,id=net0,ifname=tap102i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=3E:A9:68:F3:78:28,netdev=net0,bus=pci.0,addr=0x12,id=net0' -netdev 'type=tap,id=net1,ifname=tap102i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=EE:6A:4C:D3:19:AE,netdev=net1,bus=pci.0,addr=0x13,id=net1' -rtc 'driftfix=slew,base=localtime' -machine 'type=pc-q35-6.2+pve0' -global 'kvm-pit.lost_tick_policy=discard' -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off' -machine 'max-ram-below-4g=1G,kernel_irqchip=on'' failed: got timeout

EDIT2:
So the problem is that i have 2x GPUs that are the same but different IMMOU group.
One is connected to linux VM, working correctly. But this windows VM with another GPU its not turning on with the PCI passthrough and not backing up with the PCI in the Hardware
 
Last edited:
Code:
IOMMU Group 57 03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 710] [10de:128b] (rev a1)
IOMMU Group 58 03:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
IOMMU Group 59 06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 710] [10de:128b] (rev a1)
IOMMU Group 60 06:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
Code:
softdep nouveauu pre: vfio_pci
options vfio-pci ids=10de:128b,10de:0e0f disable_vga=1 disable_idle_d3=1
Windows VM102 was working before i inserted the second GPU
Code:
06:00.0 VGA compatible controller: NVIDIA Corporation GK208B [GeForce GT 710] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: ASUSTeK Computer Inc. GT710-SL-1GD5
        Physical Slot: 2
        Flags: bus master, fast devsel, latency 0, IRQ 87, NUMA node 0, IOMMU group 59
        Memory at 93000000 (32-bit, non-prefetchable) [size=16M]
        Memory at 39ff0000000 (64-bit, prefetchable) [size=128M]
        Memory at 39ff8000000 (64-bit, prefetchable) [size=32M]
        I/O ports at 2000 [size=128]
        Expansion ROM at 94080000 [disabled] [size=512K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Legacy Endpoint, MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Kernel driver in use: vfio-pci
        Kernel modules: nvidiafb, nouveau
Code:
03:00.0 VGA compatible controller: NVIDIA Corporation GK208B [GeForce GT 710] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: Micro-Star International Co., Ltd. [MSI] GK208B [GeForce GT 710]
        Physical Slot: 1
        Flags: fast devsel, IRQ 16, NUMA node 0, IOMMU group 57
        Memory at 95000000 (32-bit, non-prefetchable) [size=16M]
        Memory at 39fe0000000 (64-bit, prefetchable) [size=128M]
        Memory at 39fe8000000 (64-bit, prefetchable) [size=32M]
        I/O ports at 3000 [size=128]
        Expansion ROM at 96080000 [virtual] [disabled] [size=512K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Legacy Endpoint, MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Kernel driver in use: vfio-pci
        Kernel modules: nvidiafb, nouveau
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!