Backups failing due to timeout

[banana999]

My normally reliable weekly backup has failed the last 2 times, seems to a timeout of something?

INFO: starting new backup job: vzdump 100 102 --compress zstd --mode snapshot --mailto xxxxxx --mailnotification failure --quiet 1 --storage backup
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2021-06-05 05:00:02
INFO: status = running
INFO: VM Name: ubuntu-server
INFO: include disk 'virtio0' 'local-lvm:vm-100-disk-0' 64G
  /dev/sdb: open failed: No medium found
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/backup/dump/vzdump-qemu-100-2021_06_05-05_00_02.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
closing with write buffer at /usr/share/perl5/IO/Multiplex.pm line 928.
INFO: issuing guest-agent 'fs-thaw' command
ERROR: got timeout
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 100 failed - got timeout
INFO: Failed at 2021-06-05 05:03:09

Any ideas? The backup works fine when I run it manually later on

Some more details:

Spoiler

proxmox-ve: 6.4-1 (running kernel: 5.4.114-1-pve)
proxmox-backup-client: 1.1.6-2
pve-qemu-kvm: 5.2.0-6

Backup Mode: Snapshot

VM: Ubuntu 20.04.2 LTS
agent: 1
bootdisk: virtio0
cores: 4
cpu: host
hostpci0: 01:00.0
ide2: none,media=cdrom
memory: 10240
name: ubuntu-server
net0: virtio=56:48:3D:2F:EF:B1,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
protection: 1
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=775bd3af-03a5-491b-8c29-025bd6418942
sockets: 1
tablet: 0
usb0: host=1058:25ee,usb3=1
usb1: host=1a6e:089a
virtio0: local-lvm:vm-100-disk-0,size=64G
vmgenid: 750878c3-fbfc-458c-9e8d-767654a69c1a

QEMU Guest Agent installed and enabled
Package: qemu-system-x86
Version: 1:4.2-3ubuntu6.16

 

[Dominic]

Hi,

what is on /dev/sdb?

 

[banana999]

Hi,

what is on /dev/sdb?

Hi!

I'm not sure what's on sdb to be honest, I've pasted the output of lsblk and df -h here.
I am doing the backup onto a mounted network share which might be what it is, but my other VM backup works fine just 2 minutes before this one runs?
Though the network share is being provided by the ubuntu VM itself, but just weird because this issue hasn't happened before and it's been running happily for quite a while.

lsblk
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
nvme0n1                      259:0    0 238.5G  0 disk
├─nvme0n1p1                  259:1    0  1007K  0 part
├─nvme0n1p2                  259:2    0   512M  0 part /boot/efi
└─nvme0n1p3                  259:3    0   238G  0 part
  ├─pve-swap                 253:0    0     7G  0 lvm  [SWAP]
  ├─pve-root                 253:1    0  59.3G  0 lvm  /
  ├─pve-data_tmeta           253:2    0   1.6G  0 lvm
  │ └─pve-data-tpool         253:4    0 152.6G  0 lvm
  │   ├─pve-data             253:5    0 152.6G  0 lvm
  │   ├─pve-vm--101--disk--0 253:6    0    32G  0 lvm
  │   ├─pve-vm--100--disk--0 253:7    0    64G  0 lvm
  │   ├─pve-vm--102--disk--1 253:8    0    32G  0 lvm
  │   └─pve-vm--102--disk--0 253:9    0     4M  0 lvm
  └─pve-data_tdata           253:3    0 152.6G  0 lvm
    └─pve-data-tpool         253:4    0 152.6G  0 lvm
      ├─pve-data             253:5    0 152.6G  0 lvm
      ├─pve-vm--101--disk--0 253:6    0    32G  0 lvm
      ├─pve-vm--100--disk--0 253:7    0    64G  0 lvm
      ├─pve-vm--102--disk--1 253:8    0    32G  0 lvm
      └─pve-vm--102--disk--0 253:9    0     4M  0 lvm
     
df -h
Filesystem            Size  Used Avail Use% Mounted on
udev                  7.8G     0  7.8G   0% /dev
tmpfs                 1.6G  128M  1.5G   8% /run
/dev/mapper/pve-root   59G   16G   40G  29% /
tmpfs                 7.9G   43M  7.8G   1% /dev/shm
tmpfs                 5.0M  4.0K  5.0M   1% /run/lock
tmpfs                 7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/nvme0n1p2        511M  312K  511M   1% /boot/efi
/dev/fuse              30M   24K   30M   1% /etc/pve
//server.lan/proxmox  7.3T  146G  7.1T   2% /mnt/pve/backup
tmpfs                 1.6G     0  1.6G   0% /run/user/0

 

[Dominic]

/dev/sdb should appear in lsblk, so I guess there is none. Is it somewhere in your /etc/pve/storage.cfg?

Searching for sdb in the PVE configuration files might also be interesting

grep -r /etc/pve -e sdb -C10

You cannot recover a backup of your Ubuntu VM when the backup is inside that VM and the whole VM is broken?

 

[banana999]

/dev/sdb should appear in lsblk, so I guess there is none. Is it somewhere in your /etc/pve/storage.cfg?

Searching for sdb in the PVE configuration files might also be interesting

grep -r /etc/pve -e sdb -C10

You cannot recover a backup of your Ubuntu VM when the backup is inside that VM and the whole VM is broken?

The grep command didn't return anything, weird!

Sorry I should have mentioned, the Ubuntu VM just shares my external backup HDD connected via USB, just set up the network share inside of Ubuntu instead of on Proxmox.