pbs change data disk and import datastore

[abranca]

Hi guys,
i'm testing pbs and i must say i'm very happy so far.
one of the tests i'm running is the following:
- backup of 1 ct on pbs with nfs drive
- backup of 1 vm on pbs with nfs drive

For safety I bring on mega the folders of the backups with rclone.
Now I simulate the breaking of the data disk by creating a new nfs folder and go to download from mega my backups.
Browsing the contents of my datastore I get the error

Bad Request (400)
unable to detect file type

I thought it was an ownership issue because I noticed that another backup had .chunck vm ct and .lock owned by the backup user while the one restored from mega belongs to root.
With chown backup:backup to the folders and files just mentioned I can browse the contents of the datastore and restore the machine but I can't make new backups of either ct or vm.

Regardless of where I move a copy of the backups, not using mega but a second hard disk, how is it possible to import the datastore in case of failure of the main data disk?

I post the error here.

Proxmox
Virtual Environment 6.3-3
Search
Container 113 (test) on node 'pve'
local-bk
Filter VMID
Server View
Logs
()
INFO: starting new backup job: vzdump 113 --mode snapshot --node pve --storage testds --remove 0
INFO: Starting Backup of VM 113 (lxc)
INFO: Backup started at 2020-12-10 19:42:28
INFO: status = running
INFO: CT Name: test
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
  /dev/sdc: open failed: No medium found
  Logical volume "snap_vm-113-disk-0_vzdump" created.
  /dev/sdd: open failed: No medium found
  /dev/sdc: open failed: No medium found
  /dev/sdd: open failed: No medium found
INFO: creating Proxmox Backup Server archive 'ct/113/2020-12-10T18:42:28Z'
INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp15911_113/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 113 --backup-time 1607625748 --repository root@pam@192.168.0.210:testds
INFO: Starting backup: ct/113/2020-12-10T18:42:28Z
INFO: Client name: pve
INFO: Starting backup protocol: Thu Dec 10 19:42:29 2020
INFO: Couldn't download previous manifest - no valid previous backup
INFO: Upload config file '/var/tmp/vzdumptmp15911_113/etc/vzdump/pct.conf' to 'root@pam@192.168.0.210:8007:testds' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'root@pam@192.168.0.210:8007:testds' as root.pxar.didx
INFO: catalog upload error - channel closed
INFO: Error: No such file or directory (os error 2)
INFO: cleanup temporary 'vzdump' snapshot
  /dev/sdc: open failed: No medium found
  Logical volume "snap_vm-113-disk-0_vzdump" successfully removed
  /dev/sdd: open failed: No medium found
ERROR: Backup of VM 113 failed - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp15911_113/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 113 --backup-time 1607625748 --repository root@pam@192.168.0.210:testds' failed: exit code 255
INFO: Failed at 2020-12-10 19:42:43
INFO: Backup job finished with errors
TASK ERROR: job errors

Proxmox VE version

root@pve:~# pveversion -v
proxmox-ve: 6.3-1 (running kernel: 5.4.78-1-pve)
pve-manager: 6.3-3 (running version: 6.3-3/eee5f901)
pve-kernel-5.4: 6.3-3
pve-kernel-helper: 6.3-3
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.78-1-pve: 5.4.78-1
pve-kernel-5.4.73-1-pve: 5.4.73-1
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.18-2-pve: 5.3.18-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libproxmox-backup-qemu0: 1.0.2-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.3-3
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.5-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-3
pve-cluster: 6.2-1
pve-container: 3.3-1
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-7
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-2
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.5-pve1

Proxmox backup server version

proxmox-backup: 1.0-4
proxmox-backup-server: 1.0.1-1
pve-kernel-5.4: 6.2-7
pve-kernel-helper: 6.2-7
pve-kernel-5.4.65-1-pve: 5.4.65-1
ifupdown2: 3.0.0-1+pve3
libjs-extjs: 6.0.1-10
proxmox-backup-docs: 1.0.1-1
proxmox-backup-client: 1.0.1-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.3-10
pve-xtermjs: 4.7.0-2
smartmontools: 7.1-pve2
zfsutils-linux: 0.8.4-pve2

 

[oguz]

hi,

I thought it was an ownership issue because I noticed that another backup had .chunck vm ct and .lock owned by the backup user while the one restored from mega belongs to root.
With chown backup:backup to the folders and files just mentioned I can browse the contents of the datastore and restore the machine but I can't make new backups of either ct or vm.

have you tried running chown recursively with the -R flag? like so: chown -R backup:backup /your/dir. if you haven't run it recursively then you might still be lacking write access to the directory

 

[abranca]

Hello and thank you for your response.
My mount point is /mnt/testds.
I actually gave the chown -R command only to the contents of the mount point (.lock .chunk ct vm) but not to the mount point itself (/mnt/testds).

I'll redo the test and tell you more. Thanks for your reply.

 

[abranca]

I tested again with a few gig machine but did not solve the problem.

The original folder from my first backup was copied to external/remote device.

I deleted the original folder (simulating a disk failure or replacement), restored the copy of the original folder by mounting it in /mnt/pb-firewall.
I edited the /etc/proxmox-backup/datastore.cfg file by adding the existing datastore with the following commands:

datastore: pb-firewall
        path /mnt/pb-firewall

I reconnected the disk in proxmox ve as proxmox backup server drive.
I gave chown -R backup:backup /mnt/pb-firewall but no other backups can be created.

I attach here the errors:

PVE Log

INFO: starting new backup job: vzdump 101 --mode snapshot --storage pb-firewall --remove 0 --node pve
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2020-12-17 15:38:42
INFO: status = running
INFO: VM Name: OpnSense
  /dev/sdc: open failed: No medium found
  /dev/sdd: open failed: No medium found
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/101/2020-12-17T14:38:42Z'
INFO: started backup task 'e1cc45d1-0dad-4712-8a5f-f83020ba5cd6'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: OK (352.0 MiB of 32.0 GiB dirty)
INFO: using fast incremental mode (dirty-bitmap), 352.0 MiB dirty of 32.0 GiB total
INFO:   1% (4.0 MiB of 352.0 MiB) in  1s, read: 4.0 MiB/s, write: 4.0 MiB/s
ERROR: backup write data failed: command error: write_data upload error: pipelined request failed: No such file or directory (os error 2)
INFO: aborting backup job
ERROR: Backup of VM 101 failed - backup write data failed: command error: write_data upload error: pipelined request failed: No such file or directory (os error 2)
INFO: Failed at 2020-12-17 15:38:44
INFO: Backup job finished with errors
TASK ERROR: job errors

PBS Log

2020-12-17T15:38:43+01:00: starting new backup on datastore 'pb-firewall': "vm/101/2020-12-17T14:38:42Z"
2020-12-17T15:38:43+01:00: download 'index.json.blob' from previous backup.
2020-12-17T15:38:43+01:00: register chunks in 'drive-scsi0.img.fidx' from previous backup.
2020-12-17T15:38:43+01:00: download 'drive-scsi0.img.fidx' from previous backup.
2020-12-17T15:38:43+01:00: created new fixed index 1 ("vm/101/2020-12-17T14:38:42Z/drive-scsi0.img.fidx")
2020-12-17T15:38:43+01:00: add blob "/mnt/pb-firewall/vm/101/2020-12-17T14:38:42Z/qemu-server.conf.blob" (389 bytes, comp: 389)
2020-12-17T15:38:43+01:00: POST /fixed_chunk: 400 Bad Request: No such file or directory (os error 2)
2020-12-17T15:38:43+01:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: protocol error: stream no longer needed
2020-12-17T15:38:43+01:00: backup ended and finish failed: backup ended but finished flag is not set.
2020-12-17T15:38:43+01:00: removing unfinished backup
2020-12-17T15:38:43+01:00: TASK ERROR: removing backup snapshot "/mnt/pb-firewall/vm/101/2020-12-17T14:38:42Z" failed - Directory not empty (os error 39)