ERROR: Backup of VM xxx failed - no such volume 'local-lvm:vm-xxx-disk-XX

Nick_M.C. · Apr 25, 2022

Hello,

We have one Proxmox node in standalone, we have backup scheduled on weekly basis and everything was working correctly till last 2 weekends.

We haven't changed anything in the configuration point of view.

Also this week we got multiple vm`s with failed backups due to no such volume 'NewDrive1:vm-xxx-disk-0':

2022-04-23 20:34:03 INFO: Starting Backup of VM xxx (qemu)
2022-04-23 20:34:03 INFO: status = running
2022-04-23 20:34:03 INFO: VM Name: vm-backup-vmname
2022-04-23 20:34:03 INFO: include disk 'scsi0' 'NewDrive1:vm-xxx-disk-0' 20G
2022-04-23 20:34:51 ERROR: Backup of VM xxx failed - no such volume 'NewDrive1:vm-xxx-disk-0'

mira · Apr 26, 2022

Please provide the output of pveversion -v and the VM config qm config <VMID>.

Nick_M.C. · Apr 26, 2022

Hello, please find bellow the requested information:

Code:

/etc/pve/storage.cfg
dir: local
    path /var/lib/vz
    content vztmpl,backup,iso

lvmthin: local-lvm
    thinpool data
    vgname pve
    content images,rootdir

lvm: NewDrive1
    vgname storage01
    content rootdir,images
    shared 0

lvm: NewDrive2
    vgname storage02
    content images,rootdir
    shared 0

Code:

sudo qm config 117
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = (unset),
    LC_ADDRESS = "de_AT.UTF-8",
    LC_NAME = "de_AT.UTF-8",
    LC_MONETARY = "de_AT.UTF-8",
    LC_PAPER = "de_AT.UTF-8",
    LC_IDENTIFICATION = "de_AT.UTF-8",
    LC_TELEPHONE = "de_AT.UTF-8",
    LC_MEASUREMENT = "de_AT.UTF-8",
    LC_TIME = "de_AT.UTF-8",
    LC_NUMERIC = "de_AT.UTF-8",
    LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
agent: 1
boot: order=scsi0;ide2;net0
cores: 2
ide2: local:iso/ubuntu-20.04.1-live-server-amd64.iso,media=cdrom
memory: 4000
name: vm-nfs-prod
net0: virtio=16:B4:98:96:B6:C0,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: NewDrive2:vm-117-disk-0,size=500G
scsihw: virtio-scsi-pci
smbios1: uuid=bba70400-2ab0-41e7-8ccb-1881b9a19d55
sockets: 2
vmgenid: 36bc26a5-7088-47dd-bcd8-7d989ee2638c


pveversion -v
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = (unset),
    LC_ADDRESS = "de_AT.UTF-8",
    LC_NAME = "de_AT.UTF-8",
    LC_MONETARY = "de_AT.UTF-8",
    LC_PAPER = "de_AT.UTF-8",
    LC_IDENTIFICATION = "de_AT.UTF-8",
    LC_TELEPHONE = "de_AT.UTF-8",
    LC_MEASUREMENT = "de_AT.UTF-8",
    LC_TIME = "de_AT.UTF-8",
    LC_NUMERIC = "de_AT.UTF-8",
    LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
proxmox-ve: 6.4-1 (running kernel: 5.4.106-1-pve)
pve-manager: not correctly installed (running version: 6.4-14/15e2bf61)
pve-kernel-5.4.106-1-pve: 5.4.106-1
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph: not correctly installed
ceph-fuse: not correctly installed
corosync: not correctly installed
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: not correctly installed
libproxmox-acme-perl: not correctly installed
libproxmox-backup-qemu0: not correctly installed
libpve-access-control: not correctly installed
libpve-apiclient-perl: 3.1-3
libpve-common-perl: not correctly installed
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: not correctly installed
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: not correctly installed
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.6-2
pve-cluster: 6.4-1
pve-container: not correctly installed
pve-docs: not correctly installed
pve-edk2-firmware: 2.20200531-1
pve-firewall: not correctly installed
pve-firmware: 3.3-2
pve-ha-manager: not correctly installed
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: not correctly installed

Please let us know as soon as possible if there is something else needed.

Thank you

stra4d · Apr 29, 2022

Over the last while we have intermittently experienced this as well. Local drives (not NFS) and the disk is in fact there. The backup will run fine the next time. I can provide details as well but did not want to hijack the thread.

mira · Apr 29, 2022

What kind of backing storage do you use for those disks?
What kind of disks are those? (HDDs, SSDs, NVMes? Which Model?)

Code:

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = (unset),
    LC_ADDRESS = "de_AT.UTF-8",
    LC_NAME = "de_AT.UTF-8",
    LC_MONETARY = "de_AT.UTF-8",
    LC_PAPER = "de_AT.UTF-8",
    LC_IDENTIFICATION = "de_AT.UTF-8",
    LC_TELEPHONE = "de_AT.UTF-8",
    LC_MEASUREMENT = "de_AT.UTF-8",
    LC_TIME = "de_AT.UTF-8",
    LC_NUMERIC = "de_AT.UTF-8",
    LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").

Those should be fixed.

Is there anything in the journal around the time a backup fails with this error?
Could you provide the journal (+10 mins on both ends) for a failed backup?

sshami · May 2, 2022

Thanks for reply @mira to us.

we have 2 disks as OS (SATA SSD) RAID 1 configured and 2 additional disks (2x SAS SSD) for storage.

Do we have commands for journal above request ?

mira · May 2, 2022

journalctl --since "YYYY-MM-DD HH:MM:SS" --until "YYYY-MM-DD HH:MM:SS" > journal.txt where you set the year, month, day, hours, minutes and seconds. The output will be saved to journal.txt.

sshami · May 4, 2022

Hi Mira, Thanks for the exact command.

here is the requested file attached here.

Backup was started on Apr 23 20:01 and end Apr 23 20:34. with VM backup issue.
All VMs backup failed except vzdump-qemu-109 and vzdump-qemu-113

Code:

-rw-r--r-- 1 mgms-admin bacula         333 Apr 23 20:01 vzdump-qemu-100-2022_04_23-20_00_04.log
-rw-r--r-- 1 mgms-admin bacula         332 Apr 23 20:02 vzdump-qemu-103-2022_04_23-20_01_41.log
-rw-r--r-- 1 mgms-admin bacula         415 Apr 23 20:03 vzdump-qemu-104-2022_04_23-20_02_29.log
-rw-r--r-- 1 mgms-admin bacula         335 Apr 23 20:04 vzdump-qemu-106-2022_04_23-20_03_49.log
-rw-r--r-- 1 mgms-admin bacula         326 Apr 23 20:05 vzdump-qemu-107-2022_04_23-20_04_44.log
-rw-r--r-- 1 mgms-admin bacula         326 Apr 23 20:06 vzdump-qemu-108-2022_04_23-20_05_33.log
-rw-r--r-- 1 mgms-admin bacula       10107 Apr 23 20:12 vzdump-qemu-109-2022_04_23-20_06_17.log
-rw-r--r-- 1 mgms-admin bacula 13079116872 Apr 23 20:32 vzdump-qemu-109-2022_04_23-20_06_17.vma.zst
-rw-r--r-- 1 mgms-admin bacula         417 Apr 23 20:32 vzdump-qemu-110-2022_04_23-20_32_08.log
-rw-r--r-- 1 mgms-admin bacula         328 Apr 23 20:34 vzdump-qemu-111-2022_04_23-20_32_56.log
-rw-r--r-- 1 mgms-admin bacula       11230 Apr 25 17:28 vzdump-qemu-113-2022_04_25-15_49_37.log
-rw-r--r-- 1 mgms-admin bacula 39178704840 Apr 25 17:28 vzdump-qemu-113-2022_04_25-15_49_37.vma.zst
-rw-r--r-- 1 mgms-admin bacula         331 Apr 23 20:34 vzdump-qemu-123-2022_04_23-20_34_03.log

mira · May 4, 2022

Thank you for the journal!

Is /dev/sdd part of the storage your VM disks are on?

Code:

Apr 23 20:02:04 vie01-proxmox03 kernel: sd 1:0:0:0: [sdd] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_SENSE Apr 23 20:02:04 vie01-proxmox03 kernel: sd 1:0:0:0: [sdd] tag#0 Sense Key : Hardware Error [current] Apr 23 20:02:04 vie01-proxmox03 kernel: sd 1:0:0:0: [sdd] tag#0 Add. Sense: No additional sense information Apr 23 20:02:04 vie01-proxmox03 kernel: sd 1:0:0:0: [sdd] tag#0 CDB: Read(10) 28 00 00 00 08 00 00 01 00 00 Apr 23 20:02:04 vie01-proxmox03 kernel: blk_update_request: I/O error, dev sdd, sector 2048 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 0

This would indicate that there is an issue with the hardware.

sshami · May 4, 2022

Hi MIra, thanks for finding even we thought its hdd issue.

One question, there is one prod vm is running and we don't have backup for this VM, we would like to migrate this vm to another Proxmox but we dont have cluster, all are running standalone mode. Do we have something to migrate HDD to other Proxmox without loosing data.

mira · May 4, 2022

It seems sometimes backups do work? Could you try manually running it a few times?

Other than that, you could copy the VM config and VM disks off of the host.
If you just plan on moving it to another PVE host, you could try pvesm export on the source host followed by pvesm import on the destination host. You could use SSH for the connection to the destination host.
See man pvesm for more information.

sshami · May 4, 2022

mira said:
It seems sometimes backups do work? Could you try manually running it a few times?

Other than that, you could copy the VM config and VM disks off of the host.
If you just plan on moving it to another PVE host, you could try pvesm export on the source host followed by pvesm import on the destination host. You could use SSH for the connection to the destination host.
See man pvesm for more information.

Where we can find those two files: VM config and VM disks ?
could u please give exact commands if possible . Thanks !

sshami · May 5, 2022

HI @mira,

I run below command :

Code:

sudo pvesm export NewDrive1:vm-115-disk-0 raw+size 115.vmdk

i tried vmdk+size it's not working:
volume export format tar+size not available for PVE::Storage::LVMPlugin

and found file created as folllows:
26843545608 May 5 09:21 115.vmdk

Does above file (raw+size) will work for transferring to other host, if yes what step should i need to follow ?

Thanks!

mira · May 5, 2022

If it is a file, I'd recommend just copying the file with scp.
scp <path/to/file> root@<target-node-ip>:<path-to-target-storage>
The same command can be used for the VM config, just replace both the source file and the target path accordingly.

The VM config can be found in /etc/pve/qemu-server/<VMID>.conf.

Nick_M.C. · May 18, 2022

Hi, thank you for your support, the problem is with the hardware, the disks are damaged and it was not allowing the data to be managed anymore.

Search

Search

ERROR: Backup of VM xxx failed - no such volume 'local-lvm:vm-xxx-disk-XX

Nick_M.C.

New Member

mira

Proxmox Staff Member

Nick_M.C.

New Member

stra4d

Well-Known Member

mira

Proxmox Staff Member

sshami

New Member

mira

Proxmox Staff Member

sshami

New Member

Attachments

mira

Proxmox Staff Member

sshami

New Member

mira

Proxmox Staff Member

sshami

New Member

sshami

New Member

mira

Proxmox Staff Member

Nick_M.C.

New Member