Problem backing up using backup client

Dunuin

Dunuin

Distinguished Member
Jun 30, 2020
14,355
4,210
243
Germany
Is this some kind of bug or am I just too stupid to backup my unmounted virtio SCSI disks using the proxmox-backup-client?

Because this works:
Code: 
root@TestPVE2Backup:~# cat /root/scripts/backup.sh.t1
#! /bin/bash

# define repo and password:
export PBS_REPOSITORY="root@pam@192.168.43.81:8007:TestDS1"
export PBS_PASSWORD="<redacted>"

echo "Backup scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part1-3:"
proxmox-backup-client backup disk1_part1.img:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part1 disk1_part2.img:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part2 disk1_part3.img:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part3 --ns TestNS --backup-type host --crypt-mode none --backup-id TestPVE2B

echo "Backup scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part1-3:"
proxmox-backup-client backup disk2_part1.img:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part1 disk2_part2.img:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part2 disk2_part3.img:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part3 --ns TestNS --backup-type host --crypt-mode none --backup-id TestPVE2C
Code: 
root@TestPVE2Backup:~# /root/scripts/backup.sh.t1
Backup scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part1-3:
Starting backup: [TestNS]:host/TestPVE2B/2023-06-22T16:58:58Z
Client name: TestPVE2Backup
Starting backup protocol: Thu Jun 22 18:58:58 2023
No previous manifest available.
Upload image '/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part1' to 'root@pam@192.168.43.81:8007:TestDS1' as disk1_part1.img.fidx
disk1_part1.img: had to backup 1007 KiB of 1007 KiB (compressed 59 B) in 0.02s
disk1_part1.img: average backup speed: 43.859 MiB/s
Upload image '/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part2' to 'root@pam@192.168.43.81:8007:TestDS1' as disk1_part2.img.fidx
disk1_part2.img: had to backup 512 MiB of 512 MiB (compressed 509.836 MiB) in 10.42s
disk1_part2.img: average backup speed: 49.128 MiB/s
Upload image '/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part3' to 'root@pam@192.168.43.81:8007:TestDS1' as disk1_part3.img.fidx
disk1_part3.img: had to backup 3.624 GiB of 15.499 GiB (compressed 2.358 GiB) in 203.78s
disk1_part3.img: average backup speed: 18.211 MiB/s
disk1_part3.img: backup was done incrementally, reused 11.875 GiB (76.6%)
Duration: 214.35s
End Time: Thu Jun 22 19:02:32 2023
Backup scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part1-3:
Starting backup: [TestNS]:host/TestPVE2C/2023-06-22T17:02:32Z
Client name: TestPVE2Backup
Starting backup protocol: Thu Jun 22 19:02:32 2023
No previous manifest available.
Upload image '/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part1' to 'root@pam@192.168.43.81:8007:TestDS1' as disk2_part1.img.fidx
disk2_part1.img: had to backup 1007 KiB of 1007 KiB (compressed 59 B) in 0.03s
disk2_part1.img: average backup speed: 28.349 MiB/s
Upload image '/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part2' to 'root@pam@192.168.43.81:8007:TestDS1' as disk2_part2.img.fidx
disk2_part2.img: had to backup 512 MiB of 512 MiB (compressed 509.836 MiB) in 9.90s
disk2_part2.img: average backup speed: 51.739 MiB/s
Upload image '/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part3' to 'root@pam@192.168.43.81:8007:TestDS1' as disk2_part3.img.fidx
disk2_part3.img: had to backup 3.624 GiB of 15.499 GiB (compressed 2.358 GiB) in 215.20s
disk2_part3.img: average backup speed: 17.245 MiB/s
disk2_part3.img: backup was done incrementally, reused 11.875 GiB (76.6%)
Duration: 225.26s
End Time: Thu Jun 22 19:06:17 2023

But this always fails:
Code: 
root@TestPVE2Backup:~# cat /root/scripts/backup.sh
#! /bin/bash

# define repo and password:
export PBS_REPOSITORY="root@pam@192.168.43.81:8007:TestDS1"
export PBS_PASSWORD="<redacted>"

# create new temp folder for GPT backups
mkdir /tmp/GPT_backup

# create backup of partition tables (GPT)
sgdisk -b=/tmp/GPT_backup/GPT_backup_disk1.bin /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1
sgdisk -b=/tmp/GPT_backup/GPT_backup_disk2.bin /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2

# backup GPT backups as well as partitions 1 to 3 of both disks to PBS
proxmox-backup-client backup gpt_backup.pxar:/tmp/GPT_backup/ disk1_part1.img:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part1 disk1_part2.img:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part2 disk1_part3.img:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part3 disk2_part1.img:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part1 disk2_part2.img:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part2 disk2_part3.img:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part3 --ns TestNS --backup-type host --crypt-mode none --backup-id TestPVE2

# clean up
rm -r /tmp/GPT_backup
Code: 
root@TestPVE2Backup:~# /root/scripts/backup.sh
The operation has completed successfully.
The operation has completed successfully.
Error: unable to access '/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part1' - No such file or directory (os error 2)


"/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part2" and "/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part3" fail with the same error.
When removing all 3 partitions of scsi-0QEMU_QEMU_HARDDISK_drive-scsi2 it is working:
Code: 
cat /root/scripts/backup.sh.t2
#! /bin/bash

# define repo and password:
export PBS_REPOSITORY="root@pam@192.168.43.81:8007:TestDS1"
export PBS_PASSWORD="<redacted>"

# create new temp folder for GPT backups
mkdir /tmp/GPT_backup

# create backup of partition tables (GPT)
sgdisk -b=/tmp/GPT_backup/GPT_backup_disk1.bin /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1
sgdisk -b=/tmp/GPT_backup/GPT_backup_disk2.bin /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2

# backup GPT backups as well as partitions 1 to 3 of both disks to PBS
proxmox-backup-client backup gpt_backup.pxar:/tmp/GPT_backup/ disk1_part1.img:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part1 disk1_part2.img:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part2 disk1_part3.img:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part3 --ns TestNS --backup-type host --crypt-mode none --backup-id TestPVE2

# clean up
rm -r /tmp/GPT_backup
Code: 
root@TestPVE2Backup:~# /root/scripts/backup.sh.t2
The operation has completed successfully.
The operation has completed successfully.
Starting backup: [TestNS]:host/TestPVE2/2023-06-22T17:23:38Z
Client name: TestPVE2Backup
Starting backup protocol: Thu Jun 22 19:23:38 2023
Downloading previous manifest (Thu Jun 22 18:27:12 2023)
Upload directory '/tmp/GPT_backup/' to 'root@pam@192.168.43.81:8007:TestDS1' as gpt_backup.pxar.didx
gpt_backup.pxar: had to backup 35.354 KiB of 35.354 KiB (compressed 656 B) in 0.00s
gpt_backup.pxar: average backup speed: 8.466 MiB/s
Upload image '/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part1' to 'root@pam@192.168.43.81:8007:TestDS1' as disk1_part1.img.fidx
disk1_part1.img: had to backup 0 B of 1007 KiB (compressed 0 B) in 0.01s
disk1_part1.img: average backup speed: 0 B/s
Upload image '/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part2' to 'root@pam@192.168.43.81:8007:TestDS1' as disk1_part2.img.fidx
disk1_part2.img: had to backup 0 B of 512 MiB (compressed 0 B) in 7.01s
disk1_part2.img: average backup speed: 0 B/s
disk1_part2.img: backup was done incrementally, reused 512 MiB (100.0%)
Upload image '/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part3' to 'root@pam@192.168.43.81:8007:TestDS1' as disk1_part3.img.fidx
disk1_part3.img: had to backup 0 B of 15.499 GiB (compressed 0 B) in 184.09s
disk1_part3.img: average backup speed: 0 B/s
disk1_part3.img: backup was done incrementally, reused 15.499 GiB (100.0%)
Uploaded backup catalog (103 B)
Duration: 191.84s
End Time: Thu Jun 22 19:33:08 2023

"scsi-0QEMU_QEMU_HARDDISK_drive-scsi2" maps to sdc and backing up sdc1 to sdc3 works fine too, so I'm totally clueless:
Code: 
root@TestPVE2Backup:~# lsblk
NAME                          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                             8:0    0   16G  0 disk
├─sda1                          8:1    0  512M  0 part /boot/efi
├─sda2                          8:2    0  488M  0 part /boot
└─sda3                          8:3    0   15G  0 part
  ├─TestPVE2Backup--vg-root   254:0    0   14G  0 lvm  /
  └─TestPVE2Backup--vg-swap_1 254:1    0  976M  0 lvm  [SWAP]
sdb                             8:16   0   32G  0 disk
├─sdb1                          8:17   0 1007K  0 part
├─sdb2                          8:18   0  512M  0 part
├─sdb3                          8:19   0 15.5G  0 part
├─sdb4                          8:20   0    2G  0 part
└─sdb5                          8:21   0   14G  0 part
sdc                             8:32   0   32G  0 disk
├─sdc1                          8:33   0 1007K  0 part
├─sdc2                          8:34   0  512M  0 part
├─sdc3                          8:35   0 15.5G  0 part
├─sdc4                          8:36   0    2G  0 part
└─sdc5                          8:37   0   14G  0 part
sr0                            11:0    1 1024M  0 rom

root@TestPVE2Backup:~# ls -la /dev/disk/by-id
total 0
drwxr-xr-x 2 root root 480 Jun 22 17:25 .
drwxr-xr-x 8 root root 160 Jun 22 17:05 ..
lrwxrwxrwx 1 root root   9 Jun 22 17:05 ata-QEMU_DVD-ROM_QM00003 -> ../../sr0
lrwxrwxrwx 1 root root  10 Jun 22 17:05 dm-name-TestPVE2Backup--vg-root -> ../../dm-0
lrwxrwxrwx 1 root root  10 Jun 22 17:05 dm-name-TestPVE2Backup--vg-swap_1 -> ../../dm-1
lrwxrwxrwx 1 root root  10 Jun 22 17:05 dm-uuid-LVM-Mqh50AD3fsOb5gdG8DaZp2YnGt8vik67dJid8rzbwbkuf4eZ552ILr7hzlflq0y9 -> ../../dm-0
lrwxrwxrwx 1 root root  10 Jun 22 17:05 dm-uuid-LVM-Mqh50AD3fsOb5gdG8DaZp2YnGt8vik67NcnhA5SmPYbUl4IPfG9YNLHZpXAYrvCV -> ../../dm-1
lrwxrwxrwx 1 root root  10 Jun 22 17:05 lvm-pv-uuid-IGgZ5k-Z9PI-C8KD-jiY6-Rcbq-FRos-a5ySFz -> ../../sda3
lrwxrwxrwx 1 root root   9 Jun 22 17:05 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0 -> ../../sda
lrwxrwxrwx 1 root root  10 Jun 22 17:05 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part1 -> ../../sda1
lrwxrwxrwx 1 root root  10 Jun 22 17:05 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part2 -> ../../sda2
lrwxrwxrwx 1 root root  10 Jun 22 17:05 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part3 -> ../../sda3
lrwxrwxrwx 1 root root   9 Jun 22 17:05 scsi-0QEMU_QEMU_HARDDISK_drive-scsi1 -> ../../sdb
lrwxrwxrwx 1 root root  10 Jun 22 17:05 scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part1 -> ../../sdb1
lrwxrwxrwx 1 root root  10 Jun 22 17:05 scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part2 -> ../../sdb2
lrwxrwxrwx 1 root root  10 Jun 22 17:05 scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part3 -> ../../sdb3
lrwxrwxrwx 1 root root  10 Jun 22 17:05 scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part4 -> ../../sdb4
lrwxrwxrwx 1 root root  10 Jun 22 17:05 scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part5 -> ../../sdb5
lrwxrwxrwx 1 root root   9 Jun 22 17:25 scsi-0QEMU_QEMU_HARDDISK_drive-scsi2 -> ../../sdc
lrwxrwxrwx 1 root root  10 Jun 22 17:25 scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part1 -> ../../sdc1
lrwxrwxrwx 1 root root  10 Jun 22 17:25 scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part2 -> ../../sdc2
lrwxrwxrwx 1 root root  10 Jun 22 17:25 scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part3 -> ../../sdc3
lrwxrwxrwx 1 root root  10 Jun 22 17:25 scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part4 -> ../../sdc4
lrwxrwxrwx 1 root root  10 Jun 22 17:25 scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part5 -> ../../sdc5

Code: 
root@TestPVE2Backup:~# cat /root/scripts/backup.sh.t3
#! /bin/bash

# define repo and password:
export PBS_REPOSITORY="root@pam@192.168.43.81:8007:TestDS1"
export PBS_PASSWORD="<redacted>"

# create new temp folder for GPT backups
mkdir /tmp/GPT_backup

# create backup of partition tables (GPT)
sgdisk -b=/tmp/GPT_backup/GPT_backup_disk1.bin /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1
sgdisk -b=/tmp/GPT_backup/GPT_backup_disk2.bin /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2

# backup GPT backups as well as partitions 1 to 3 of both disks to PBS
proxmox-backup-client backup gpt_backup.pxar:/tmp/GPT_backup/ disk1_part1.img:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part1 disk1_part2.img:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part2 disk1_part3.img:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part3 disk2_part1.img:/dev/sdc1 disk2_part2.img:/dev/sdc2 disk2_part3.img:/dev/sdc3 --ns TestNS --backup-type host --crypt-mode none --backup-id TestPVE2

# clean up
rm -r /tmp/GPT_backup

Code: 
root@TestPVE2Backup:~# /root/scripts/backup.sh.t3
The operation has completed successfully.
The operation has completed successfully.
Starting backup: [TestNS]:host/TestPVE2/2023-06-22T17:40:42Z
Client name: TestPVE2Backup
Starting backup protocol: Thu Jun 22 19:40:42 2023
Downloading previous manifest (Thu Jun 22 19:29:56 2023)
Upload directory '/tmp/GPT_backup/' to 'root@pam@192.168.43.81:8007:TestDS1' as gpt_backup.pxar.didx
gpt_backup.pxar: had to backup 35.354 KiB of 35.354 KiB (compressed 656 B) in 0.01s
gpt_backup.pxar: average backup speed: 6.101 MiB/s
Upload image '/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part1' to 'root@pam@192.168.43.81:8007:TestDS1' as disk1_part1.img.fidx
disk1_part1.img: had to backup 0 B of 1007 KiB (compressed 0 B) in 0.01s
disk1_part1.img: average backup speed: 0 B/s
Upload image '/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part2' to 'root@pam@192.168.43.81:8007:TestDS1' as disk1_part2.img.fidx
disk1_part2.img: had to backup 0 B of 512 MiB (compressed 0 B) in 6.91s
disk1_part2.img: average backup speed: 0 B/s
disk1_part2.img: backup was done incrementally, reused 512 MiB (100.0%)
Upload image '/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part3' to 'root@pam@192.168.43.81:8007:TestDS1' as disk1_part3.img.fidx
disk1_part3.img: had to backup 0 B of 15.499 GiB (compressed 0 B) in 179.21s
disk1_part3.img: average backup speed: 0 B/s
disk1_part3.img: backup was done incrementally, reused 15.499 GiB (100.0%)
Upload image '/dev/sdc1' to 'root@pam@192.168.43.81:8007:TestDS1' as disk2_part1.img.fidx
Error downloading .fidx from previous manifest: Unable to open fixed index "/mnt/datastore/TestDS1/ns/TestNS/host/TestPVE2/2023-06-22T17:29:56Z/disk2_part1.img.fidx" - No such file or directory (os error 2)
disk2_part1.img: had to backup 1007 KiB of 1007 KiB (compressed 59 B) in 0.02s
disk2_part1.img: average backup speed: 41.841 MiB/s
Upload image '/dev/sdc2' to 'root@pam@192.168.43.81:8007:TestDS1' as disk2_part2.img.fidx
Error downloading .fidx from previous manifest: Unable to open fixed index "/mnt/datastore/TestDS1/ns/TestNS/host/TestPVE2/2023-06-22T17:29:56Z/disk2_part2.img.fidx" - No such file or directory (os error 2)
disk2_part2.img: had to backup 512 MiB of 512 MiB (compressed 509.836 MiB) in 9.73s
disk2_part2.img: average backup speed: 52.64 MiB/s
Upload image '/dev/sdc3' to 'root@pam@192.168.43.81:8007:TestDS1' as disk2_part3.img.fidx
Error downloading .fidx from previous manifest: Unable to open fixed index "/mnt/datastore/TestDS1/ns/TestNS/host/TestPVE2/2023-06-22T17:29:56Z/disk2_part3.img.fidx" - No such file or directory (os error 2)
disk2_part3.img: had to backup 3.624 GiB of 15.499 GiB (compressed 2.358 GiB) in 207.43s
disk2_part3.img: average backup speed: 17.89 MiB/s
disk2_part3.img: backup was done incrementally, reused 11.875 GiB (76.6%)
Uploaded backup catalog (103 B)
Duration: 403.47s
End Time: Thu Jun 22 19:47:25 2023

Would be glad if someone could explain me what is going wrong here.
 
you could add something like stat $dev; stat -L $dev to your script to check the symlink and its target right before the PBS client call, maybe that sheds some light?
 
  • Like
Reactions: Dunuin
fabian said:
you could add something like stat $dev; stat -L $dev to your script to check the symlink and its target right before the PBS client call, maybe that sheds some light?
Click to expand...
Code: 
root@TestPVE2Backup:~# stat /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part1; stat -L /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part1
stat /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part2; stat -L /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part2
stat /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part3; stat -L /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part3
  File: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part1 -> ../../sdc1
  Size: 10              Blocks: 0          IO Block: 4096   symbolic link
Device: 5h/5d   Inode: 318         Links: 1
Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-06-23 19:09:18.360000000 +0200
Modify: 2023-06-23 19:09:18.204000000 +0200
Change: 2023-06-23 19:09:18.204000000 +0200
 Birth: -
  File: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part1
  Size: 0               Blocks: 0          IO Block: 4096   block special file
Device: 5h/5d   Inode: 239         Links: 1     Device type: 8,21
Access: (0660/brw-rw----)  Uid: (    0/    root)   Gid: (    6/    disk)
Access: 2023-06-23 19:09:18.204000000 +0200
Modify: 2023-06-23 19:09:18.204000000 +0200
Change: 2023-06-23 19:09:18.204000000 +0200
 Birth: -
  File: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part2 -> ../../sdc2
  Size: 10              Blocks: 0          IO Block: 4096   symbolic link
Device: 5h/5d   Inode: 325         Links: 1
Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-06-23 19:09:18.376000000 +0200
Modify: 2023-06-23 19:09:18.232000000 +0200
Change: 2023-06-23 19:09:18.232000000 +0200
 Birth: -
  File: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part2
  Size: 0               Blocks: 0          IO Block: 4096   block special file
Device: 5h/5d   Inode: 240         Links: 1     Device type: 8,22
Access: (0660/brw-rw----)  Uid: (    0/    root)   Gid: (    6/    disk)
Access: 2023-06-23 19:09:18.232000000 +0200
Modify: 2023-06-23 19:09:18.232000000 +0200
Change: 2023-06-23 19:09:18.232000000 +0200
 Birth: -
  File: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part3 -> ../../sdc3
  Size: 10              Blocks: 0          IO Block: 4096   symbolic link
Device: 5h/5d   Inode: 295         Links: 1
Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-06-23 19:09:18.600000000 +0200
Modify: 2023-06-23 19:09:18.348000000 +0200
Change: 2023-06-23 19:09:18.348000000 +0200
 Birth: -
  File: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part3
  Size: 0               Blocks: 0          IO Block: 4096   block special file
Device: 5h/5d   Inode: 242         Links: 1     Device type: 8,23
Access: (0660/brw-rw----)  Uid: (    0/    root)   Gid: (    6/    disk)
Access: 2023-06-23 19:09:18.344000000 +0200
Modify: 2023-06-23 19:09:18.344000000 +0200
Change: 2023-06-23 19:09:18.344000000 +0200
 Birth: -

I'm still confused why this works...:
Code: 
proxmox-backup-client backup \
   disk2_part1.img:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part1 \
   disk2_part2.img:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part2 \
   disk2_part3.img:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part3 \
   --ns TestNS --backup-type host --crypt-mode none --backup-id TestPVE2C



Upload image '/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part1' to 'root@pam@192.168.43.81:8007:TestDS1' as disk2_part1.img.fidx
disk2_part1.img: had to backup 1007 KiB of 1007 KiB (compressed 59 B) in 0.03s
disk2_part1.img: average backup speed: 28.349 MiB/s
Upload image '/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part2' to 'root@pam@192.168.43.81:8007:TestDS1' as disk2_part2.img.fidx
disk2_part2.img: had to backup 512 MiB of 512 MiB (compressed 509.836 MiB) in 9.90s
disk2_part2.img: average backup speed: 51.739 MiB/s
Upload image '/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part3' to 'root@pam@192.168.43.81:8007:TestDS1' as disk2_part3.img.fidx
disk2_part3.img: had to backup 3.624 GiB of 15.499 GiB (compressed 2.358 GiB) in 215.20s
disk2_part3.img: average backup speed: 17.245 MiB/s
disk2_part3.img: backup was done incrementally, reused 11.875 GiB (76.6%)

...but not this:
Code: 
proxmox-backup-client backup \
   gpt_backup.pxar:/tmp/GPT_backup/ \
   disk1_part1.img:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part1 \
   disk1_part2.img:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part2 \
   disk1_part3.img:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1-part3 \
   disk2_part1.img:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part1 \
   disk2_part2.img:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part2 \
   disk2_part3.img:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part3 \
   --ns TestNS --backup-type host --crypt-mode none --backup-id TestPVE2



Error: unable to access '/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi2-part1' - No such file or directory (os error 2)
Both are using the exact same paths.

Edit: Meanwhile I detached both virtual disks and readded them and now the same scripts that failed run fine...not sure what the problem was, because restarting the VMs wasn't fixing it...
 
Last edited:
But I now got another problem. I used "proxmox-backup-client map ..." to map a vitual disk to loop0 so I can dd it to the empty partition. But there I get dd: error reading '/dev/loop0': Input/output error errors:

restore script:
Code: 
#! /bin/bash

# define repo and password:
export PBS_REPOSITORY="root@pam@192.168.43.81:8007:TestDS1"
export PBS_PASSWORD="<redacted>"

# create new mountpoint folder for GPT backups
mkdir /mnt/GPT_backup

# mount archive with the partition table backups, restore partition tables and unmount it
proxmox-backup-client mount "host/TestPVE2/2023-06-22T19:28:21Z" "gpt_backup.pxar" "/mnt/GPT_backup" --ns "TestNS"
sgdisk -l=/mnt/GPT_backup/GPT_backup_disk1.bin /dev/sdb
sgdisk -l=/mnt/GPT_backup/GPT_backup_disk2.bin /dev/sdc
umount /mnt/GPT_backup

# map and restore disk 1 partition 1
proxmox-backup-client map "host/TestPVE2/2023-06-22T19:28:21Z" "disk1_part1.img" --ns "TestNS"
dd if=/dev/loop0 of=/dev/sdb1 bs=512 conv=noerror,sync status=progress
proxmox-backup-client unmap /dev/loop0

Output:
Code: 
root@TestPVE2Backup:~# /root/scripts/restore.sh
mkdir: cannot create directory ‘/mnt/GPT_backup’: File exists
FUSE library version: 3.10.3
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
The operation has completed successfully.
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
The operation has completed successfully.
Image 'root@pam@192.168.43.81:8007:TestDS1:host/TestPVE2/2023-06-22T19:28:21Z/disk1_part1.img' mapped on /dev/loop0
dd: error reading '/dev/loop0': Input/output error
2008+0 records in
2008+0 records out
1028096 bytes (1.0 MB, 1004 KiB) copied, 0.135013 s, 7.6 MB/s
dd: error reading '/dev/loop0': Input/output error
2008+1 records in
2009+0 records out
1028608 bytes (1.0 MB, 1004 KiB) copied, 0.137282 s, 7.5 MB/s
dd: error reading '/dev/loop0': Input/output error
2008+2 records in
2010+0 records out
1029120 bytes (1.0 MB, 1005 KiB) copied, 0.138416 s, 7.4 MB/s
dd: error reading '/dev/loop0': Input/output error
2008+3 records in
2011+0 records out
1029632 bytes (1.0 MB, 1006 KiB) copied, 0.139396 s, 7.4 MB/s
dd: error reading '/dev/loop0': Input/output error
2008+4 records in
2012+0 records out
1030144 bytes (1.0 MB, 1006 KiB) copied, 0.140603 s, 7.3 MB/s
dd: error reading '/dev/loop0': Input/output error
2008+5 records in
2013+0 records out
1030656 bytes (1.0 MB, 1006 KiB) copied, 0.14134 s, 7.3 MB/s
2008+6 records in
2014+0 records out
1031168 bytes (1.0 MB, 1007 KiB) copied, 0.158156 s, 6.5 MB/s

The corresponding PBS task log shows this:
Code: 
2023-06-24T01:21:02+02:00: starting new backup reader datastore 'TestDS1': "/mnt/datastore/TestDS1"
2023-06-24T01:21:02+02:00: protocol upgrade done
2023-06-24T01:21:02+02:00: GET /download
2023-06-24T01:21:02+02:00: download "/mnt/datastore/TestDS1/ns/TestNS/host/TestPVE2/2023-06-22T19:28:21Z/index.json.blob"
2023-06-24T01:21:02+02:00: GET /download
2023-06-24T01:21:02+02:00: download "/mnt/datastore/TestDS1/ns/TestNS/host/TestPVE2/2023-06-22T19:28:21Z/disk1_part1.img.fidx"
2023-06-24T01:21:02+02:00: register chunks in 'disk1_part1.img.fidx' as downloadable.
2023-06-24T01:21:02+02:00: GET /chunk
2023-06-24T01:21:02+02:00: download chunk "/mnt/datastore/TestDS1/.chunks/2c48/2c4881bc6abcae1bfe2c92d042a3ae2903e3245732a86762c84be3f7e5b50705"
2023-06-24T01:21:02+02:00: reader finished successfully
2023-06-24T01:21:02+02:00: TASK OK

Can I somehow debug/fix these IO errors?
 
Last edited:
how is the connection between the two hosts? does restoring to STDOUT and piping that into the block device work instead of going via map? does a verification of the backup come back OK?
 
fabian said:
how is the connection between the two hosts?
Click to expand...
Both are VMs on the same PVE host (7.4-15), connected to the same bridge, using virtio NICs. Never had any issues with that PVE host. So I would guess the connection should be good.
fabian said:
does restoring to STDOUT and piping that into the block device work instead of going via map?
Click to expand...
Will try a proxmox-backup-client restore "host/TestPVE2/2023-06-22T19:28:21Z" "disk1_part1.img" - --ns "TestNS" > /dev/sdb1 and report back later.
fabian said:
does a verification of the backup come back OK?
Click to expand...
Yes, I verified them and "All OK".
 
Not sure how to see if there are any IO errors.

When running this...
Code: 
#! /bin/bash

# define repo and password:
export PBS_REPOSITORY="root@pam@192.168.43.81:8007:TestDS1"
export PBS_PASSWORD="<redacted>"

# create new mountpoint folder for GPT backups
mkdir /mnt/GPT_backup

# mount archive with the partition table backups, restore partition tables and unmount it
proxmox-backup-client mount "host/TestPVE2/2023-06-22T19:28:21Z" "gpt_backup.pxar" "/mnt/GPT_backup" --ns "TestNS"
sgdisk -l=/mnt/GPT_backup/GPT_backup_disk1.bin /dev/sdb
sgdisk -l=/mnt/GPT_backup/GPT_backup_disk2.bin /dev/sdc
umount /mnt/GPT_backup

proxmox-backup-client restore "host/TestPVE2/2023-06-22T19:28:21Z" "disk1_part1.img" - --ns "TestNS" > /dev/sdb1

# clean up
rmdir /mnt/GPT_backup
... I only get:
Code: 
root@TestPVE2Backup:~# /root/scripts/restore2.sh
FUSE library version: 3.10.3
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
The operation has completed successfully.
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
The operation has completed successfully.
restore image complete (bytes=1031168, duration=0.01s, speed=122.62MB/s)

And here the PBS log:
Code: 
2023-06-28T02:27:29+02:00: starting new backup reader datastore 'TestDS1': "/mnt/datastore/TestDS1"
2023-06-28T02:27:29+02:00: protocol upgrade done
2023-06-28T02:27:29+02:00: GET /download
2023-06-28T02:27:29+02:00: download "/mnt/datastore/TestDS1/ns/TestNS/host/TestPVE2/2023-06-22T19:28:21Z/index.json.blob"
2023-06-28T02:27:29+02:00: GET /download
2023-06-28T02:27:29+02:00: download "/mnt/datastore/TestDS1/ns/TestNS/host/TestPVE2/2023-06-22T19:28:21Z/disk1_part1.img.fidx"
2023-06-28T02:27:29+02:00: register chunks in 'disk1_part1.img.fidx' as downloadable.
2023-06-28T02:27:29+02:00: GET /chunk
2023-06-28T02:27:29+02:00: download chunk "/mnt/datastore/TestDS1/.chunks/2c48/2c4881bc6abcae1bfe2c92d042a3ae2903e3245732a86762c84be3f7e5b50705"
2023-06-28T02:27:29+02:00: reader finished successfully
2023-06-28T02:27:29+02:00: TASK OK

When testing the connection it mostly looks fine despite some lost packets?:
Code: 
root@TestPVE2Backup:~# iperf3 -c 192.168.43.81 -t 120 -i 10
Connecting to host 192.168.43.81, port 5201
[  5] local 192.168.43.182 port 51944 connected to 192.168.43.81 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-10.00  sec  13.9 GBytes  12.0 Gbits/sec    0   3.04 MBytes
[  5]  10.00-20.00  sec  14.3 GBytes  12.3 Gbits/sec    0   3.04 MBytes
[  5]  20.00-30.00  sec  14.6 GBytes  12.6 Gbits/sec    0   3.04 MBytes
[  5]  30.00-40.00  sec  13.8 GBytes  11.9 Gbits/sec    0   3.04 MBytes
[  5]  40.00-50.00  sec  13.8 GBytes  11.8 Gbits/sec    0   3.04 MBytes
[  5]  50.00-60.00  sec  13.6 GBytes  11.7 Gbits/sec   45   3.04 MBytes
[  5]  60.00-70.00  sec  13.2 GBytes  11.3 Gbits/sec   45   3.04 MBytes
[  5]  70.00-80.00  sec  14.2 GBytes  12.2 Gbits/sec    0   3.04 MBytes
[  5]  80.00-90.00  sec  14.5 GBytes  12.5 Gbits/sec    0   3.04 MBytes
[  5]  90.00-100.00 sec  14.0 GBytes  12.0 Gbits/sec   90   3.04 MBytes
[  5] 100.00-110.00 sec  14.2 GBytes  12.2 Gbits/sec    0   3.04 MBytes
[  5] 110.00-120.00 sec  13.7 GBytes  11.8 Gbits/sec   90   3.04 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-120.00 sec   168 GBytes  12.0 Gbits/sec  270             sender
[  5]   0.00-120.04 sec   168 GBytes  12.0 Gbits/sec                  receiver

iperf Done.

Edit:
Looks like dding from loop only complains about the restore of the first and third PVE partitions (the 1MB legacy and ZFS ones):
Code: 
#! /bin/bash

# define repo and password:
export PBS_REPOSITORY="root@pam@192.168.43.81:8007:TestDS1"
export PBS_PASSWORD="<redacted>"

# create new mountpoint folder for GPT backups
mkdir /mnt/GPT_backup

# mount archive with the partition table backups, restore partition tables and unmount it
proxmox-backup-client mount "host/TestPVE2/2023-06-22T19:28:21Z" "gpt_backup.pxar" "/mnt/GPT_backup" --ns "TestNS"
sgdisk -l=/mnt/GPT_backup/GPT_backup_disk1.bin /dev/sdb
sgdisk -l=/mnt/GPT_backup/GPT_backup_disk2.bin /dev/sdc
umount /mnt/GPT_backup

# map and restore disk 1 partition 1
proxmox-backup-client map "host/TestPVE2/2023-06-22T19:28:21Z" "disk1_part1.img" --ns "TestNS"
dd if=/dev/loop0 of=/dev/sdb1 bs=4K conv=noerror,sync status=progress
proxmox-backup-client unmap /dev/loop0

# map and restore disk 1 partition 2
proxmox-backup-client map "host/TestPVE2/2023-06-22T19:28:21Z" "disk1_part2.img" --ns "TestNS"
dd if=/dev/loop0 of=/dev/sdb2 bs=4K conv=noerror,sync status=progress
proxmox-backup-client unmap /dev/loop0

# map and restore disk 1 partition 3
proxmox-backup-client map "host/TestPVE2/2023-06-22T19:28:21Z" "disk1_part3.img" --ns "TestNS"
dd if=/dev/loop0 of=/dev/sdb3 bs=4K conv=noerror,sync status=progress
proxmox-backup-client unmap /dev/loop0

# map and restore disk 2 partition 1
proxmox-backup-client map "host/TestPVE2/2023-06-22T19:28:21Z" "disk2_part1.img" --ns "TestNS"
dd if=/dev/loop0 of=/dev/sdc1 bs=4K conv=noerror,sync status=progress
proxmox-backup-client unmap /dev/loop0

# map and restore disk 2 partition 2
proxmox-backup-client map "host/TestPVE2/2023-06-22T19:28:21Z" "disk2_part2.img" --ns "TestNS"
dd if=/dev/loop0 of=/dev/sdc2 bs=4K conv=noerror,sync status=progress
proxmox-backup-client unmap /dev/loop0

# map and restore disk 2 partition 3
proxmox-backup-client map "host/TestPVE2/2023-06-22T19:28:21Z" "disk2_part3.img" --ns "TestNS"
dd if=/dev/loop0 of=/dev/sdc3 bs=4K conv=noerror,sync status=progress
proxmox-backup-client unmap /dev/loop0

# creating encrypted swap on partition 4 is done later when booted into restored PVE

# create new ZFS pool on partition 5
#zpool create dpool -o ashift=12 mirror /dev/sdb5 /dev/sdc5
#zpool export dpool
# creating encrypted datasets, adding PVE storages and restoring guests + data is done later when booted into restored PVE

# clean up
rmdir /mnt/GPT_backup

Code: 
root@TestPVE2Backup:~# /root/scripts/restore.sh
mkdir: cannot create directory ‘/mnt/GPT_backup’: File exists
FUSE library version: 3.10.3
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk1.bin
The operation has completed successfully.
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
Error 38 when determining sector size! Setting sector size to 512
Disk device is /mnt/GPT_backup/GPT_backup_disk2.bin
The operation has completed successfully.
Image 'root@pam@192.168.43.81:8007:TestDS1:host/TestPVE2/2023-06-22T19:28:21Z/disk1_part1.img' mapped on /dev/loop0
dd: error reading '/dev/loop0': Input/output error
251+0 records in
251+0 records out
1028096 bytes (1.0 MB, 1004 KiB) copied, 0.00854993 s, 120 MB/s
dd: /dev/loop0: cannot seek: Invalid argument
dd: error writing '/dev/sdb1': No space left on device
251+1 records in
251+0 records out
1031168 bytes (1.0 MB, 1007 KiB) copied, 0.0235841 s, 43.7 MB/s
Image 'root@pam@192.168.43.81:8007:TestDS1:host/TestPVE2/2023-06-22T19:28:21Z/disk1_part2.img' mapped on /dev/loop0
532537344 bytes (533 MB, 508 MiB) copied, 11 s, 48.4 MB/s
131072+0 records in
131072+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 11.2634 s, 47.7 MB/s
Image 'root@pam@192.168.43.81:8007:TestDS1:host/TestPVE2/2023-06-22T19:28:21Z/disk1_part3.img' mapped on /dev/loop0
16600440832 bytes (17 GB, 15 GiB) copied, 154 s, 108 MB/s
dd: error reading '/dev/loop0': Input/output error
4062976+0 records in
4062976+0 records out
16641949696 bytes (17 GB, 15 GiB) copied, 154.408 s, 108 MB/s
dd: /dev/loop0: cannot seek: Invalid argument
dd: error writing '/dev/sdb3': No space left on device
4062976+1 records in
4062976+0 records out
16641950208 bytes (17 GB, 15 GiB) copied, 155.374 s, 107 MB/s
Image 'root@pam@192.168.43.81:8007:TestDS1:host/TestPVE2/2023-06-22T19:28:21Z/disk2_part1.img' mapped on /dev/loop0
dd: error reading '/dev/loop0': Input/output error
251+0 records in
251+0 records out
1028096 bytes (1.0 MB, 1004 KiB) copied, 0.0139386 s, 73.8 MB/s
dd: /dev/loop0: cannot seek: Invalid argument
dd: error writing '/dev/sdc1': No space left on device
251+1 records in
251+0 records out
1031168 bytes (1.0 MB, 1007 KiB) copied, 0.0319611 s, 32.3 MB/s
Image 'root@pam@192.168.43.81:8007:TestDS1:host/TestPVE2/2023-06-22T19:28:21Z/disk2_part2.img' mapped on /dev/loop0
503177216 bytes (503 MB, 480 MiB) copied, 9 s, 55.8 MB/s
131072+0 records in
131072+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 9.70948 s, 55.3 MB/s
Image 'root@pam@192.168.43.81:8007:TestDS1:host/TestPVE2/2023-06-22T19:28:21Z/disk2_part3.img' mapped on /dev/loop0
16603017216 bytes (17 GB, 15 GiB) copied, 154 s, 108 MB/s
dd: error reading '/dev/loop0': Input/output error
4062976+0 records in
4062976+0 records out
16641949696 bytes (17 GB, 15 GiB) copied, 154.231 s, 108 MB/s
dd: /dev/loop0: cannot seek: Invalid argument
dd: error writing '/dev/sdc3': No space left on device
4062976+1 records in
4062976+0 records out
16641950208 bytes (17 GB, 15 GiB) copied, 155.374 s, 107 MB/s

No PBS logs show any errors despite I/O errors when reading from loop0:
Code: 
2023-06-28T02:44:55+02:00: starting new backup reader datastore 'TestDS1': "/mnt/datastore/TestDS1"
2023-06-28T02:44:55+02:00: protocol upgrade done
2023-06-28T02:44:55+02:00: GET /download
2023-06-28T02:44:55+02:00: download "/mnt/datastore/TestDS1/ns/TestNS/host/TestPVE2/2023-06-22T19:28:21Z/index.json.blob"
2023-06-28T02:44:55+02:00: GET /download
2023-06-28T02:44:55+02:00: download "/mnt/datastore/TestDS1/ns/TestNS/host/TestPVE2/2023-06-22T19:28:21Z/disk1_part1.img.fidx"
2023-06-28T02:44:55+02:00: register chunks in 'disk1_part1.img.fidx' as downloadable.
2023-06-28T02:44:55+02:00: GET /chunk
2023-06-28T02:44:55+02:00: download chunk "/mnt/datastore/TestDS1/.chunks/2c48/2c4881bc6abcae1bfe2c92d042a3ae2903e3245732a86762c84be3f7e5b50705"
2023-06-28T02:44:55+02:00: reader finished successfully
2023-06-28T02:44:55+02:00: TASK OK
 
Last edited:
mhm. could you post the exact sizes in bytes of the three partitions? I wonder if something in the mapping code assumes aligned/padded chunks..

you could also compare the "restore"d partitions and the "map+dd"-ed ones (e.g., by calling sha256sum on the results) to see if they are identical despite the errors. I will try to reproduce the issue!
 
Code: 
root@TestPVE2Backup:~# fdisk -l /dev/sdb
Disk /dev/sdb: 32 GiB, 34359738368 bytes, 67108864 sectors
Disk model: QEMU HARDDISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 427047FD-4D28-408B-B424-0DC3843F222D

Device        Start      End  Sectors  Size Type
/dev/sdb1        34     2047     2014 1007K BIOS boot
/dev/sdb2      2048  1050623  1048576  512M EFI System
/dev/sdb3   1050624 33554432 32503809 15.5G Solaris /usr & Apple ZFS
/dev/sdb4  33556480 37750783  4194304    2G Linux filesystem
/dev/sdb5  37750784 67108830 29358047   14G Linux filesystem

Sizes of all 3 partitions and the mapped images do match:
sdb1: 1,031,168 B
loop0 of sdb1 backup: 1,031,168 B
sdb2: 536,870,912 B
loop0 of sdb2 backup: 536,870,912 B
sdb3: 16,641,950,208 B
loop0 of sdb3 backup: 16,641,950,208 B
Tried to checksum loop and partitions and for sdb2 and sdc2 the sha256sum of the restored partition and loop0 match. But checksumming of /dev/loop0 fails for the backup of sdb1, sdb3, sdc1 and sdc3 because if I/O errors:
Code: 
#! /bin/bash

# define repo and password:
export PBS_REPOSITORY="root@pam@192.168.43.81:8007:TestDS1"
export PBS_PASSWORD="<redacted>"

# map and restore disk 1 partition 1
proxmox-backup-client map "host/TestPVE2/2023-06-22T19:28:21Z" "disk1_part1.img" --ns "TestNS"
dd if=/dev/loop0 of=/dev/sdb1 bs=512 conv=noerror,sync status=progress
sha256sum /dev/loop0
sha256sum /dev/sdb1
fdisk -l /dev/loop0
proxmox-backup-client unmap /dev/loop0

# map and restore disk 1 partition 2
proxmox-backup-client map "host/TestPVE2/2023-06-22T19:28:21Z" "disk1_part2.img" --ns "TestNS"
dd if=/dev/loop0 of=/dev/sdb2 bs=512 conv=noerror,sync status=progress
sha256sum /dev/loop0
sha256sum /dev/sdb2
fdisk -l /dev/loop0
proxmox-backup-client unmap /dev/loop0

# map and restore disk 1 partition 3
proxmox-backup-client map "host/TestPVE2/2023-06-22T19:28:21Z" "disk1_part3.img" --ns "TestNS"
dd if=/dev/loop0 of=/dev/sdb3 bs=512 conv=noerror,sync status=progress
sha256sum /dev/loop0
sha256sum /dev/sdb3
fdisk -l /dev/loop0
proxmox-backup-client unmap /dev/loop0

# map and restore disk 2 partition 1
proxmox-backup-client map "host/TestPVE2/2023-06-22T19:28:21Z" "disk2_part1.img" --ns "TestNS"
dd if=/dev/loop0 of=/dev/sdc1 bs=512K conv=noerror,sync status=progress
sha256sum /dev/loop0
sha256sum /dev/sdc1
fdisk -l /dev/loop0
proxmox-backup-client unmap /dev/loop0

# map and restore disk 2 partition 2
proxmox-backup-client map "host/TestPVE2/2023-06-22T19:28:21Z" "disk2_part2.img" --ns "TestNS"
dd if=/dev/loop0 of=/dev/sdc2 bs=512 conv=noerror,sync status=progress
sha256sum /dev/loop0
sha256sum /dev/sdc2
fdisk -l /dev/loop0
proxmox-backup-client unmap /dev/loop0

# map and restore disk 2 partition 3
proxmox-backup-client map "host/TestPVE2/2023-06-22T19:28:21Z" "disk2_part3.img" --ns "TestNS"
dd if=/dev/loop0 of=/dev/sdc3 bs=512 conv=noerror,sync status=progress
sha256sum /dev/loop0
sha256sum /dev/sdc3
fdisk -l /dev/loop0
proxmox-backup-client unmap /dev/loop0

Code: 
root@TestPVE2Backup:~# /root/scripts/restore3.sh
Image 'root@pam@192.168.43.81:8007:TestDS1:host/TestPVE2/2023-06-22T19:28:21Z/disk1_part1.img' mapped on /dev/loop0
dd: error reading '/dev/loop0': Input/output error
2008+0 records in
2008+0 records out
1028096 bytes (1.0 MB, 1004 KiB) copied, 0.126067 s, 8.2 MB/s
dd: error reading '/dev/loop0': Input/output error
2008+1 records in
2009+0 records out
1028608 bytes (1.0 MB, 1004 KiB) copied, 0.137656 s, 7.5 MB/s
dd: error reading '/dev/loop0': Input/output error
2008+2 records in
2010+0 records out
1029120 bytes (1.0 MB, 1005 KiB) copied, 0.148645 s, 6.9 MB/s
dd: error reading '/dev/loop0': Input/output error
2008+3 records in
2011+0 records out
1029632 bytes (1.0 MB, 1006 KiB) copied, 0.151589 s, 6.8 MB/s
dd: error reading '/dev/loop0': Input/output error
2008+4 records in
2012+0 records out
1030144 bytes (1.0 MB, 1006 KiB) copied, 0.152125 s, 6.8 MB/s
dd: error reading '/dev/loop0': Input/output error
2008+5 records in
2013+0 records out
1030656 bytes (1.0 MB, 1006 KiB) copied, 0.152942 s, 6.7 MB/s
2008+6 records in
2014+0 records out
1031168 bytes (1.0 MB, 1007 KiB) copied, 0.170364 s, 6.1 MB/s
sha256sum: /dev/loop0: Input/output error
2c4881bc6abcae1bfe2c92d042a3ae2903e3245732a86762c84be3f7e5b50705  /dev/sdb1
Disk /dev/loop0: 1007 KiB, 1031168 bytes, 2014 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Image 'root@pam@192.168.43.81:8007:TestDS1:host/TestPVE2/2023-06-22T19:28:21Z/disk1_part2.img' mapped on /dev/loop0
529752576 bytes (530 MB, 505 MiB) copied, 30 s, 17.7 MB/s
1048576+0 records in
1048576+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 30.5492 s, 17.6 MB/s
f1d5d064e5777ed1f52d18e2714097d681e3932b368255ff4bf7b6bfff9b6dc7  /dev/loop0
f1d5d064e5777ed1f52d18e2714097d681e3932b368255ff4bf7b6bfff9b6dc7  /dev/sdb2
Disk /dev/loop0: 512 MiB, 536870912 bytes, 1048576 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000
Image 'root@pam@192.168.43.81:8007:TestDS1:host/TestPVE2/2023-06-22T19:28:21Z/disk1_part3.img' mapped on /dev/loop0
16606116864 bytes (17 GB, 15 GiB) copied, 223 s, 74.5 MB/s
dd: error reading '/dev/loop0': Input/output error
32503808+0 records in
32503808+0 records out
16641949696 bytes (17 GB, 15 GiB) copied, 223.463 s, 74.5 MB/s
32503808+1 records in
32503809+0 records out
16641950208 bytes (17 GB, 15 GiB) copied, 224.391 s, 74.2 MB/s
sha256sum: /dev/loop0: Input/output error
9bf29279e817e172d339d44e1265f5adc4d689ca6feb157b4a99579b71a00790  /dev/sdb3
Disk /dev/loop0: 15.5 GiB, 16641950208 bytes, 32503809 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Image 'root@pam@192.168.43.81:8007:TestDS1:host/TestPVE2/2023-06-22T19:28:21Z/disk2_part1.img' mapped on /dev/loop0
dd: error writing '/dev/sdc1': No space left on device
1+1 records in
1+0 records out
1031168 bytes (1.0 MB, 1007 KiB) copied, 0.0556188 s, 18.5 MB/s
sha256sum: /dev/loop0: Input/output error
2c4881bc6abcae1bfe2c92d042a3ae2903e3245732a86762c84be3f7e5b50705  /dev/sdc1
Disk /dev/loop0: 1007 KiB, 1031168 bytes, 2014 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Image 'root@pam@192.168.43.81:8007:TestDS1:host/TestPVE2/2023-06-22T19:28:21Z/disk2_part2.img' mapped on /dev/loop0
528351744 bytes (528 MB, 504 MiB) copied, 27 s, 19.5 MB/s
1048576+0 records in
1048576+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 27.7199 s, 19.4 MB/s
7c8b7077a45942daa8e754b2f4aef3e20c747735b8b1a8480f881da82d99d0bd  /dev/loop0
7c8b7077a45942daa8e754b2f4aef3e20c747735b8b1a8480f881da82d99d0bd  /dev/sdc2
Disk /dev/loop0: 512 MiB, 536870912 bytes, 1048576 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000
Image 'root@pam@192.168.43.81:8007:TestDS1:host/TestPVE2/2023-06-22T19:28:21Z/disk2_part3.img' mapped on /dev/loop0
16628412928 bytes (17 GB, 15 GiB) copied, 220 s, 75.6 MB/s
dd: error reading '/dev/loop0': Input/output error
32503808+0 records in
32503808+0 records out
16641949696 bytes (17 GB, 15 GiB) copied, 220.149 s, 75.6 MB/s
32503808+1 records in
32503809+0 records out
16641950208 bytes (17 GB, 15 GiB) copied, 221.15 s, 75.3 MB/s
sha256sum: /dev/loop0: Input/output error
30cf794ce24e4281fbdbb577e9756fa9ecfcd6dd48dadc04271a81bfa4cce1a2  /dev/sdc3
Disk /dev/loop0: 15.5 GiB, 16641950208 bytes, 32503809 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

And when piping restore to a file and hashing it:
Code: 
#! /bin/bash

# define repo and password:
export PBS_REPOSITORY="root@pam@192.168.43.81:8007:TestDS1"
export PBS_PASSWORD="<redacted>"

proxmox-backup-client restore "host/TestPVE2/2023-06-22T19:28:21Z" "disk1_part1.img" - --ns "TestNS" > /mnt/sdd/sdb1
sha256sum /mnt/sdd/sdb1
proxmox-backup-client restore "host/TestPVE2/2023-06-22T19:28:21Z" "disk1_part2.img" - --ns "TestNS" > /mnt/sdd/sdb2
sha256sum /mnt/sdd/sdb2
proxmox-backup-client restore "host/TestPVE2/2023-06-22T19:28:21Z" "disk1_part3.img" - --ns "TestNS" > /mnt/sdd/sdb3
sha256sum /mnt/sdd/sdb3
Code: 
root@TestPVE2Backup:~# /root/scripts/test.sh
restore image complete (bytes=1031168, duration=0.01s, speed=103.05MB/s)
2c4881bc6abcae1bfe2c92d042a3ae2903e3245732a86762c84be3f7e5b50705  /mnt/sdd/sdb1
restore image complete (bytes=536870912, duration=8.74s, speed=58.61MB/s)
f1d5d064e5777ed1f52d18e2714097d681e3932b368255ff4bf7b6bfff9b6dc7  /mnt/sdd/sdb2
restore image complete (bytes=16641950208, duration=109.88s, speed=144.44MB/s)
9bf29279e817e172d339d44e1265f5adc4d689ca6feb157b4a99579b71a00790  /mnt/sdd/sdb3
So here the checksums match the content of sdb1 and sdb3 restored via dding from loop. For the next test I should checksum the partitions before the backup and after the restore to see if it still matches.

VM configs in case that matter:
All VM disks are stored on the same healthy ZFS mirror on the same host. No ECC RAM errors. SSDs look healthy and are connected to the mainboards chipset (Intel C612).
Debian/PVE VM:
Code: 
root@Voyager:~# cat /etc/pve/qemu-server/151.conf
agent: 1
bios: ovmf
boot: order=ide2;scsi0;net0
cores: 2
cpu: host,flags=+aes
efidisk0: dpool_vlt_VM8K:vm-151-disk-0,efitype=4m,size=1M
ide2: none,media=cdrom
machine: q35
memory: 2048
meta: creation-qemu=7.2.0,ctime=1687340848
name: TestPVE2
net0: virtio=56:E7:8C:87:66:6E,bridge=vmbr43,firewall=1
net1: virtio=9E:DD:8D:98:7A:B4,bridge=vmbr4,firewall=1
numa: 0
ostype: l26
scsi0: dpool_vlt_VM8K:vm-151-disk-1,discard=on,iothread=1,size=16G,ssd=1
scsi1: dpool_vlt_VM8K:vm-151-disk-5,discard=on,iothread=1,size=32G,ssd=1
scsi2: dpool_vlt_VM8K:vm-151-disk-6,discard=on,iothread=1,size=32G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=d17dba85-3107-4348-aa3f-90478efddb1e
sockets: 1
tpmstate0: dpool_vlt_VM8K:vm-151-disk-4,size=4M,version=v2.0
unused0: dpool_vlt_VM8K:vm-151-disk-2
unused1: dpool_vlt_VM8K:vm-151-disk-3
vmgenid: 1b1c0ae7-5842-4a6d-a03f-510ef418c8cb
PBS VM:
Code: 
root@Voyager:~# cat /etc/pve/qemu-server/154.conf
agent: 1
bios: ovmf
boot: order=scsi0;ide2;net0
cores: 2
cpu: host,flags=+aes
efidisk0: dpool_vlt_VM8K:vm-154-disk-0,efitype=4m,size=1M
ide2: data:iso/proxmox-backup-server_2.4-1.iso,media=cdrom,size=841792K
machine: q35
memory: 2048
meta: creation-qemu=7.2.0,ctime=1687341285
name: TestPBS
net0: virtio=52:58:1D:57:72:09,bridge=vmbr43,firewall=1
numa: 0
ostype: l26
scsi0: dpool_vlt_VM8K:vm-154-disk-1,discard=on,iothread=1,size=16G,ssd=1
scsi1: dpool_vlt_VM8K:vm-154-disk-2,discard=on,iothread=1,size=100G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=7dee4420-b1eb-4ba9-83ce-65852c179ce3
sockets: 1
tpmstate0: dpool_vlt_VM8K:vm-154-disk-3,size=4M,version=v2.0
vmgenid: c1c78a28-294a-496e-a78c-45d16ec1c983
 
Last edited:
that seems to confirm my hunch that `map` doesn't like the fact that the first and third partition don't end with a "full" chunk, or are very small. I can reproduce the problem and will try to find the root cause :)
 
  • Like
Reactions: Dunuin
https://lists.proxmox.com/pipermail/pbs-devel/2023-June/006277.html

my hunch was indeed correct - if the end of the image/archive was not aligned with the incoming read request, it would fail. depending on the size of the image and the way of accessing it the alignment might have been 4k or any multiple of it (up to 128k in my tests, but those might not have covered all variants ;))
 
  • Like
Reactions: Dunuin and fiona
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back