No previous backup found, cannot do incremental backup

[mracar]

Hello,

Im having same issue with one of my nodes out of three. New datastore didnt fixed the issue, its still happening. All disks including backup directory mounted from iSCSI. FYI The server was installed with Proxmox 7 first then upgraded to 8 but the VM is created while server is running 8. There is no EFI disk. All VM's are booting with old plain BIOS.

pveversion -v
proxmox-ve: 8.2.0 (running kernel: 6.8.8-1-pve)
pve-manager: 8.2.4 (running version: 8.2.4/faa83925c9641325)
proxmox-kernel-helper: 8.1.0
pve-kernel-5.15: 7.4-14
proxmox-kernel-6.8: 6.8.8-1
proxmox-kernel-6.8.8-1-pve-signed: 6.8.8-1
pve-kernel-5.15.158-1-pve: 5.15.158-1
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.7
libpve-cluster-perl: 8.0.7
libpve-common-perl: 8.2.1
libpve-guest-common-perl: 5.1.3
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.9
libpve-storage-perl: 8.2.2
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.2.4-1
proxmox-backup-file-restore: 3.2.4-1
proxmox-firewall: 0.4.2
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.6
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.7
pve-container: 5.1.12
pve-docs: 8.2.2
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.1
pve-firewall: 5.0.7
pve-firmware: 3.12-1
pve-ha-manager: 4.0.5
pve-i18n: 3.2.2
pve-qemu-kvm: 8.1.5-6
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.4-pve1

Output of the backup:
INFO: Starting Backup of VM 115 (qemu)
INFO: Backup started at 2024-07-08 10:31:43
INFO: status = running
INFO: VM Name: XXXXXXX
INFO: include disk 'scsi0' 'CoreSsdBootPool:vm-115-disk-0' 20G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/115/2024-07-08T07:31:43Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 115 qmp command 'backup' failed - backup register image failed: command error: no previous backup found, cannot do incremental backup
INFO: aborting backup job

Qm config:
agent: 1
boot: order=scsi0
cores: 2
cpu: host
memory: 4086
meta: creation-qemu=8.1.5,ctime=1719245391
name: XXXXX
net0: virtio=XXXX,bridge=XXXX,tag=113
numa: 0
onboot: 1
ostype: l26
scsi0: CoreSsdBootPool:vm-115-disk-0,iothread=1,size=20G
scsihw: virtio-scsi-single
smbios1: uuid=XXXX
sockets: 2
tags: XXX
vmgenid: XXXX


PBS Log:
2024-07-08T10:31:43+03:00: starting new backup on datastore 'PrxmxBckNew02' from ::ffff:127.0.0.1: "vm/115/2024-07-08T07:31:43Z"
2024-07-08T10:31:43+03:00: download 'index.json.blob' from previous backup.
2024-07-08T10:31:43+03:00: GET /previous: 400 Bad Request: Unable to open fixed index "/mnt/prxmxbcknew/XXX/vm/115/2024-07-06T01:21:53Z/drive-scsi0.img.fidx" - got unknown magic number
2024-07-08T10:31:43+03:00: backup failed: connection error: not connected
2024-07-08T10:31:43+03:00: removing failed backup
2024-07-08T10:31:43+03:00: TASK ERROR: connection error: not connected

 

[Chris]

Hello,

Im having same issue with one of my nodes out of three. New datastore didnt fixed the issue, its still happening. All disks including backup directory mounted from iSCSI. FYI The server was installed with Proxmox 7 first then upgraded to 8 but the VM is created while server is running 8. There is no EFI disk. All VM's are booting with old plain BIOS.

pveversion -v
proxmox-ve: 8.2.0 (running kernel: 6.8.8-1-pve)
pve-manager: 8.2.4 (running version: 8.2.4/faa83925c9641325)
proxmox-kernel-helper: 8.1.0
pve-kernel-5.15: 7.4-14
proxmox-kernel-6.8: 6.8.8-1
proxmox-kernel-6.8.8-1-pve-signed: 6.8.8-1
pve-kernel-5.15.158-1-pve: 5.15.158-1
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.7
libpve-cluster-perl: 8.0.7
libpve-common-perl: 8.2.1
libpve-guest-common-perl: 5.1.3
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.9
libpve-storage-perl: 8.2.2
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.2.4-1
proxmox-backup-file-restore: 3.2.4-1
proxmox-firewall: 0.4.2
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.6
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.7
pve-container: 5.1.12
pve-docs: 8.2.2
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.1
pve-firewall: 5.0.7
pve-firmware: 3.12-1
pve-ha-manager: 4.0.5
pve-i18n: 3.2.2
pve-qemu-kvm: 8.1.5-6
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.4-pve1

Output of the backup:
INFO: Starting Backup of VM 115 (qemu)
INFO: Backup started at 2024-07-08 10:31:43
INFO: status = running
INFO: VM Name: XXXXXXX
INFO: include disk 'scsi0' 'CoreSsdBootPool:vm-115-disk-0' 20G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/115/2024-07-08T07:31:43Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 115 qmp command 'backup' failed - backup register image failed: command error: no previous backup found, cannot do incremental backup
INFO: aborting backup job

Qm config:
agent: 1
boot: order=scsi0
cores: 2
cpu: host
memory: 4086
meta: creation-qemu=8.1.5,ctime=1719245391
name: XXXXX
net0: virtio=XXXX,bridge=XXXX,tag=113
numa: 0
onboot: 1
ostype: l26
scsi0: CoreSsdBootPool:vm-115-disk-0,iothread=1,size=20G
scsihw: virtio-scsi-single
smbios1: uuid=XXXX
sockets: 2
tags: XXX
vmgenid: XXXX


PBS Log:
2024-07-08T10:31:43+03:00: starting new backup on datastore 'PrxmxBckNew02' from ::ffff:127.0.0.1: "vm/115/2024-07-08T07:31:43Z"
2024-07-08T10:31:43+03:00: download 'index.json.blob' from previous backup.
2024-07-08T10:31:43+03:00: GET /previous: 400 Bad Request: Unable to open fixed index "/mnt/prxmxbcknew/XXX/vm/115/2024-07-06T01:21:53Z/drive-scsi0.img.fidx" - got unknown magic number
2024-07-08T10:31:43+03:00: backup failed: connection error: not connected
2024-07-08T10:31:43+03:00: removing failed backup
2024-07-08T10:31:43+03:00: TASK ERROR: connection error: not connected

Hi,
so if you say that the issue is present on 1 of 3 nodes do you mean that the backups work without any issues when performed if the same VM is migrated to an other node?
Can you also verify that the fixed index file is corrupted in the same manner, e.g. by running

head -c4096 /mnt/prxmxbcknew/XXX/vm/115/2024-07-06T01:21:53Z/drive-scsi0.img.fidx | hexdump

.
Also, since your datastore is not backed by CephFS, this would point to the issue being unrelated to that.

 

[Chris]

Hi Chris,

the issue is still there.

Best regards
Simon

May I still ask you to verify at what point the actual corruption occurs, e.g. are the first 16 bytes already corrupted after starting the backup or does the corruption happen at the end? You might do something like described here to try to monitor this https://forum.proxmox.com/threads/backup-suceeds-but-ends-up-failing-verification.149249/post-676207.

 

[mracar]

Hi,
so if you say that the issue is present on 1 of 3 nodes do you mean that the backups work without any issues when performed if the same VM is migrated to an other node?
Can you also verify that the fixed index file is corrupted in the same manner, e.g. by running

head -c4096 /mnt/prxmxbcknew/XXX/vm/115/2024-07-06T01:21:53Z/drive-scsi0.img.fidx | hexdump

.
Also, since your datastore is not backed by CephFS, this would point to the issue being unrelated to that.

Hey,

Index file seems like not corrupted like SiJux posted earlier. Migrating VM to another node fixes the issue. But when you migrate back to original node, issue comes back. I can create another thread if you want to since yes my datastore is not backed by CephFS.

 

[SiJux]

May I still ask you to verify at what point the actual corruption occurs, e.g. are the first 16 bytes already corrupted after starting the backup or does the corruption happen at the end? You might do something like described here to try to monitor this https://forum.proxmox.com/threads/backup-suceeds-but-ends-up-failing-verification.149249/post-676207.

Hi Chris,

the file is corrupted from the beginning.

I can't confirm that the issue is gone on another node. I'Ve migrated a affected vm to another node - same issue.

Best Regards
Simon

 

[fiona]

Hi,

Hey,

Index file seems like not corrupted like SiJux posted earlier. Migrating VM to another node fixes the issue. But when you migrate back to original node, issue comes back. I can create another thread if you want to since yes my datastore is not backed by CephFS.

can you please post the output from the head command for completeness, as well as the output of pveversion -v from the working node and the node with the issue. Just to be sure, are the nodes in the same cluster, i.e. using the very same PBS storage as a target? Is there anything in the journal of the PBS around the time the issue happens.

 

[Chris]

Hi Chris,

the file is corrupted from the beginning.

I can't confirm that the issue is gone on another node. I'Ve migrated a affected vm to another node - same issue.

Best Regards
Simon

Is this true also for the first run, so the one creating the reference for the second run?

 

[mracar]

Hi,

can you please post the output from the head command for completeness, as well as the output of pveversion -v from the working node and the node with the issue. Just to be sure, are the nodes in the same cluster, i.e. using the very same PBS storage as a target? Is there anything in the journal of the PBS around the time the issue happens.

Hello Fiona,

Yes both nodes in the same cluster. They are using same iSCSI disk with different subdirectory inside. one of is running in /mnt/prxmxbck/XXX01 other one is /mnt/prxmxbck/XXX02. Every node have own their PBS installed on PVE and every PVE uses own PBS server connected with 127.0.0.1. I also tried creating an another iSCSI share with fresh disk, the issue is still persisted.


Working node:
proxmox-ve: 8.2.0 (running kernel: 6.8.4-3-pve)
pve-manager: 8.2.2 (running version: 8.2.2/9355359cd7afbae4)
proxmox-kernel-helper: 8.1.0
pve-kernel-5.15: 7.4-13
proxmox-kernel-6.8: 6.8.4-3
proxmox-kernel-6.8.4-3-pve-signed: 6.8.4-3
pve-kernel-5.15.152-1-pve: 5.15.152-1
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.6
libpve-cluster-perl: 8.0.6
libpve-common-perl: 8.2.1
libpve-guest-common-perl: 5.1.2
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.2.1
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.2.3-1
proxmox-backup-file-restore: 3.2.3-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.6
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.6
pve-container: 5.1.10
pve-docs: 8.2.2
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.0
pve-firewall: 5.0.7
pve-firmware: 3.11-1
pve-ha-manager: 4.0.4
pve-i18n: 3.2.2
pve-qemu-kvm: 8.1.5-6
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.3-pve2


Broken node:
proxmox-ve: 8.2.0 (running kernel: 6.8.8-1-pve)
pve-manager: 8.2.4 (running version: 8.2.4/faa83925c9641325)
proxmox-kernel-helper: 8.1.0
pve-kernel-5.15: 7.4-14
proxmox-kernel-6.8: 6.8.8-1
proxmox-kernel-6.8.8-1-pve-signed: 6.8.8-1
pve-kernel-5.15.158-1-pve: 5.15.158-1
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.7
libpve-cluster-perl: 8.0.7
libpve-common-perl: 8.2.1
libpve-guest-common-perl: 5.1.3
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.9
libpve-storage-perl: 8.2.2
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.2.4-1
proxmox-backup-file-restore: 3.2.4-1
proxmox-firewall: 0.4.2
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.6
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.7
pve-container: 5.1.12
pve-docs: 8.2.2
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.1
pve-firewall: 5.0.7
pve-firmware: 3.12-1
pve-ha-manager: 4.0.5
pve-i18n: 3.2.2
pve-qemu-kvm: 8.1.5-6
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.4-pve1

HEAD command:

0000000 930d 0e44 3d18 991a a501 313b cf21 8e12
0000010 3f26 501a 55a8 63a9 24a5 c623 4649 abbd
0000020 ab43 edc6 e1dd d5f5 f159 2ecc 400f 2282
0000030 82b6 9541 864a 11d5 07cd 8da7 16b8 5081
0000040 03ba f07b 9100 220d 74a8 259a 927a efe3
0000050 cfca 448c b24e 8423 2fa2 bc05 0a2c b8aa
0000060 1a3d aad4 bc0d 262b d460 1c21 2150 006e
0000070 9edd 231d 9723 6cca 62c2 0446 af14 dac2
0000080 a9a1 8395 6c1f 1ccf 5254 7c2c b3e1 2504
0000090 0396 1acf d547 41b3 8577 448a a585 9138
00000a0 40ae 59ed 32a9 75aa 4966 cadc 022c c172
00000b0 9f00 3a29 591f 577d 6c4a 78ed b4d4 30d2
00000c0 0383 d051 2448 8088 d917 9e1d 959c f74a
00000d0 8064 7722 cd3a 1182 2168 4a2f 0310 bc4c
00000e0 94cf bfff 0cd6 432e 915e 7b16 9a27 b00c
00000f0 e668 6bff a4cb 0064 fffc b2da 0741 1df1
0000100 9023 f87d 5238 c0c0 0c42 fff8 6db5 dc2e
0000110 ac9c d9c8 c925 95a9 9481 8113 a810 aa1c
0000120 5432 7373 2735 a792 50a6 59d0 45b1 2ade
0000130 844a 936b 4c2a 31f8 bb6c f84c e8f3 91cb
0000140 0508 6184 721c 505c d313 5024 a1c8 e8a0
0000150 27b2 22f2 d4f3 fd0a ba51 137d bada 8b26
0000160 9c4c 5234 32da 2839 1dd2 302a 6bba a141
0000170 e852 5ab2 6591 f451 79cc 8c6f be6c 48de
0000180 fb6a 5330 9b52 62f3 fbc8 daff c232 9746
0000190 6298 58e4 6294 8445 a2e1 1a2a 4295 91a5
00001a0 0687 133e 2498 4d0f aff4 8335 0581 fff8
00001b0 65b5 2a4e 96b3 283e b226 96d2 7661 fcbc
00001c0 f2f3 6bfa a4cb 2f50 6fb3 9038 92d9 a4f6
00001d0 5120 b119 23ae 2130 cdd5 6318 a3a1 339d
00001e0 ff57 8b5f a066 6aab d482 d74c 10c6 0d22
00001f0 3341 fff1 96d7 0139 3bcc 84b3 0a6d 0000
0000200 ca48 5234 f342 bffe ccb6 a3c9 c64b ffc6
0000210 3e95 45fe 0a6a 9805 a047 8e02 d317 d7ff
0000220 0996 997e 7ffe 996d 1914 a8d9 b0ae 262a
0000230 0c44 04c2 b4c1 a24e 11a6 20b2 1af3 3925
0000240 21d2 d646 502a afe0 732d 22d2 fff0 cb6b
0000250 ac9c ed61 60f0 93f0 6674 4936 1170 ef33
0000260 8942 7920 678e 28cd 8a2e 9cd4 6064 7011
0000270 1360 4178 cbb0 081b 48d3 0d07 f092 4516
0000280 aac9 73ec 04b5 0808 83a2 fc88 b5ff 8265
0000290 ed86 d7ff 19a2 bc10 1f99 3411 5f6b ed34
00002a0 c57a b15e ac57 4510 886e 6694 1230 65c4
00002b0 8f84 30e2 4352 47d1 5803 0d26 0156 a4cb
00002c0 8158 12db a0d5 b1bd e803 44ce ab18 bfc0
00002d0 ccb6 8909 fff7 cb6b f49a 24a1 e433 9687
00002e0 3904 58a2 d289 f607 26e3 54ba 7322 4ab1
00002f0 c18f 85ff b312 597f 2659 ac5e 5821 da2c
0000300 2d74 1d10 c889 1cae fe16 b6bf 504c ec4b
0000310 6bff 64db c898 907e 0092 1b8c 871d 5cc0
0000320 7ffe 9b6d 138c 2b79 e4ce 958f 111c b2a4
0000330 daae bee0 a322 1840 81c0 260f 805a 1a40
0000340 26a1 6bde 6016 60a4 56ac 15a6 742c c47f
0000350 6d7f 2099 2616 139f 7265 f8c4 6258 bd29
0000360 1fc9 fca0 6d7f 9399 ef18 1c13 0bdc 0935
0000370 ae4a 7ffc 996d c694 ffc8 6daf 018b 46e9
0000380 9cde d0a8 d268 41e9 f02d 6bff e0db b7dc
0000390 9409 6529 0f1a 2034 e04e fed4 b6bf c9ac
00003a0 371d a04c 8d17 88c3 fcbe daff a9b6 0c27
00003b0 3508 4ef1 bfff ccb6 a389 7fbc a280 0a9f
00003c0 effe ffe0 9c54 195a e561 d309 9f34 a748
00003d0 cc4f f962 2355 6c64 c19a 7851 a0a3 6b50
00003e0 4383 e97b fb4c 3ba2 cb02 7508 cde4 eec9
00003f0 9ab8 12e9 4e4d 69e7 345f ed9b 7583 6dc4
0000400 1c31 72c1 7da6 3998 65b2 6584 2757 acce
0000410 e364 0158 7475 389d 510b 0846 d9cf 6e99
0000420 129e e182 034c bcdf 1fac 429f e4ae 87b4
0000430 0f9a 3037 c3eb c3ed 8b6c bf31 3a91 c162
0000440 911e 5394 3e3f 3b82 caae f708 ded4 794c
0000450 7a28 8542 8b6d a861 232d bb94 5374 e826
0000460 3af7 72c9 9875 10c5 1574 5174 64da 26e1
0000470 7228 1103 6146 7593 86c2 ccc3 062c ec88
0000480 a888 8c59 3a3c a2ca 7640 b10e afff 132d
0000490 21c4 a973 1195 788a d055 7ffd 19ad a19c
00004a0 27d3 ff23 73ff ca24 14b6 021c 744d f919
00004b0 db6b dc64 d3a0 9cbb 4c2d 7cb3 7b6e 9874
00004c0 d380 59bd 874f cd9c f29b 55b9 2c00 69a2
00004d0 4e8f 65ef 9a52 44c0 3aa6 80ac b243 e77a
00004e0 68a1 5e46 e990 5b5f d846 1336 4de6 2825
00004f0 fffa 33da 4c08 68cc 65a8 b46f aad6 902c
0000500 1e69 9961 0914 8fee 9aa9 38aa ffd3 9ad7
0000510 d0c1 fec2 6d7f e099 f68d 6bff a4cb be08
0000520 2a31 a52f d188 c841 f222 6bff a4cb d9c0
0000530 afff fb6d a030 eab5 fe5d 8054 153f fff6
0000540 d780 fffd 0106 2e4d cd75 9496 5054 0bff
0000550 cdf5 324d fe61 6db5 e21f bad4 6154 3311
0000560 a234 0212 7416 2d1e 49d4 742b 8322 8181
0000570 e869 2f02 8db6 e40d 8d25 f285 16cc f4ff
0000580 b62d b259 1c36 cd65 1476 712f a449 8741
0000590 1180 583e a858 7e99 d869 21a0 8703 c0c5
00005a0 b944 0bcd 04ec 0514 a7ee 54ca 318b 05da
00005b0 0c5e 32dc f315 4632 c0bb 818b a65b a262
00005c0 44cf ce19 0592 07c4 81c9 2d14 18bc a44a
00005d0 6691 08cf 251f 8e8d 3219 9860 d06e 8190
00005e0 96c3 16a9 45ca 3e6c ac86 b1f5 b0d2 0941
00005f0 69c3 92c9 1335 9281 4aee 1857 4491 661a
0000600 18a4 73e1 5024 b11a 0d30 c21c 6434 80e0
0000610 0f75 3965 2d3e 1061 5a0b aa25 5571 97d8
0000620 22a2 3bac 2518 cce4 2323 94c5 f0e0 0fcd
0000630 0c8a 0dd3 5861 d0f7 21b5 5a53 89ea 6300
0000640 0745 80d5 2fb0 2cc5 85d6 2622 80f7 08c0
0000650 d1aa aaa1 0f96 285d 4662 dcc4 471c e150
0000660 6478 4041 92c0 9108 1642 80d0 d061 2620
0000670 f616 01b5 5a71 eb18 3340 cbe7 e3c8 b01e
0000680 530d 5f0d d388 2ac2 9a1a 16ae 2b4d d341
0000690 916c 805c bd38 2156 af4e cf55 b0b8 466a
00006a0 4ba1 3baf 8656 6594 3041 4795 5024 3398
00006b0 c01c c4c4 bec2 5c36 b0ab 57b4 9b48 f806
00006c0 524b 00c1 86d8 86a9 6baf 15d5 31c0 f85e
00006d0 2a08 1773 1fc0 45ad ed81 4015 6228 e897
00006e0 74a8 b491 4845 8b27 4381 1111 8942 ccc2
00006f0 8c34 0703 0db0 5f61 b01b 5604 7c21 4652
0000700 9d87 7c36 f456 36c0 354c 617c 8409 9802
0000710 0a83 9284 470b 8762 1560 2842 5162 e85e
0000720 88b0 3a5d 436c 465d 8cba e9c0 e29b 0658
0000730 430c 0698 d12b d829 1d10 e924 f934 c258
0000740 b053 c450 010c 5db1 43fa 21cd 5107 9bd3
0000750 9129 28bc 8d7b c01c 6434 d300 b5f0 ca81
0000760 a115 9399 8416 5a5f 5840 7a14 3c67 12c0
0000770 2568 e54a b08a 50d1 9611 3840 512b 2a38
0000780 7057 8186 424f c1b3 1428 b7b1 c045 cc36
0000790 102f c090 8a1e f988 1210 be64 6834 6698
00007a0 6c20 741c 5303 c535 9652 620c ed6e 0516
00007b0 2284 1746 b9ef 2169 2906 5113 a313 9120
00007c0 1fbc c67b 7184 00c0 e1a6 836b 19a0 4409
00007d0 6c71 b72a c112 c8d8 1620 81e6 99c4 5c1a
00007e0 6fde 4579 85a9 9add b042 530d 5f0d 9307
00007f0 07b4 5938 ca00 0c44 453a ffec 370f 2918
0000800 8908 8901 8b36 1420 399a 0c89 1322 7220
0000810 2b08 3f45 7071 b46d 0703 7091 2365 c346
0000820 c394 a986 a986 6098 6063 ecd7 4f63 098c
0000830 383a 8a1c e071 8180 3344 d421 5811 d755
0000840 3006 2757 4b12 d1e6 8270 cf8d 3b2e aa76
0000850 f609 5194 c2d3 3084 2c88 5f06 5162 2b00
0000860 32e1 52ac 0025 07a1 9c2c 3830 3429 5a4e
0000870 6091 5085 0d18 361c 986c 986a c18a c129
0000880 65fe 4918 11a0 e1c7 a4d0 ea0d 110e 752d
0000890 8887 0c76 8603 3131 1c0e ac30 cbd9 f36b
00008a0 f6d0 43d5 e1db 6bed d787 dc03 0f57 87b0
00008b0 afb7 de1e 710f 3f5f 97c4 c96e b66f d1c9
00008c0 0fd7 5f72 c83f e61f 7ebe 8f98 f635 3f37
00008d0 e8d8 07eb afb7 7a1f e808 a2a0 9d4d 27a5
00008e0 da4f 3ebe 0569 bedf be0a e04f 3ebe 9f81
00008f0 7dbc f27d 9142 0afb 8645 f732 ca15 250c
0000900 2bee 1894 db8a 2857 5fae 9854 651c 645e
0000910 b061 bef3 79da b71d 75af 3b5c 5f6f bced
0000920 dc76 dabe ef71 7dc8 23bd 89df 7afb 3e27
0000930 9fd6 277b 1614 7afa ee07 1deb 4ef8 d7db
0000940 2d3b 320f 3546 644a 8141 fb74 e90a 81ae
0000950 bafb 8606 fb8c 190a c243 857d db84 afbc
0000960 ce6d ee46 1b2b cd99 57dc f336 78b1 77c8
0000970 1c72 57d0 6f36 d95f 7dbc 0602 acd9 2c78
0000980 819e f6f3 ce15 192b 55f7 8632 f723 8e15
0000990 270c 2bee 589c db4e 3957 9c6d afb9 6671
00009a0 ef39 9cab ab1f 7e4f 83f3 5f58 c0e3 8d7d
00009b0 ef03 8a98 c842 0925 4b73 f790 c835 b881
00009c0 057d 06e2 fb79 e41a e20d 35f6 2b88 ee44
00009d0 10ab 0cf9 c0fc ba7f ec5a 10ab afb8 6042
00009e0 7121 855f 42c4 beda 690a 4875 40f1 21c1
00009f0 2291 493d bede 7926 c491 457d 2712 fb91
0000a00 449a 124e 35f7 ac49 ee64 92ab c979 bafb
0000a10 4be4 beda 692e c095 557d e702 1208 86c6
0000a20 0d94 b715 54af 29dc 5f70 c0a7 e254 52be
0000a30 a989 7dc0 02a5 9157 5afb ae45 f732 cab5
0000a40 e56c b57d 2af2 5f6f bcab 6838 5a3c 4401
0000a50 4bd1 57db 6d2f b797 5daf 2f5e 5f70 c0bd
0000a60 de5e 7abe b379 5f71 71b3 ee56 566b cece
0000a70 cd7d 59ce afb7 9759 ca2f 6cec c6ca 4203
0000a80 b819 65af 46e0 afb9 728d b999 65af 32e6
0000a90 5f72 c8cb de67 33d7 d33f f5f6 7b4c edc6
0000aa0 19eb 4cd7 57dc 7b33 5a64 7516 596d 70f2
0000ab0 cafb edc3 fb83 07da 88f3 cafb e623 f701
0000ac0 0795 27ec d5f7 f64f fb9b 37ea 78f7 7b5f
0000ad0 efdc abed abdf 7527 01c0 0bda 9312 c09a
0000ae0 6d7d 0702 7dca 940d cc9b 6d7d 3732 fb91
0000af0 44da 6c16 2afb 3ed8 abea dba0 4157 58f7
0000b00 fe3e 0e57 abe9 fbe0 782a 7127 9d5f d6c4
0000b10 ee13 4bb3 1af1 f5f7 171a 7dce 9c2d df0d
0000b20 c357 e75f 5ebe c3e7 afb7 ef87 59a5 a3ee
0000b30 aa32 f5f4 edd0 a1eb ac7b f21e 876f 5f43
0000b40 db0f 1ebe 86b6 5f72 e50d 601b f02a fa15
0000b50 6061 db8b 2dd7 226e d7dc 7844 dc4b 2dd7
0000b60 16f1 afb8 605b b945 8aaf e2f2 bafb 1e78
0000b70 7d95 9c55 aafb 1f38 cfcb 2bfd fafd f8aa
0000b80 55f6 cdf1 fa8b 6912 ceea 25de f6fc 7e15
0000b90 e2ff febe b0c5 57dc 8658 bee3 f1c2 b98f
0000ba0 3faf 2066 2bef a390 d038 2057 5f6d b481
0000bb0 e2c7 bf01 7dad fd7d f701 07f5 278c c7d5
0000bc0 4765 2065 3449 bede 78d2 72a9 a95f e434
0000bd0 d2be 6990 7dc4 11a5 cdcf 9d7d 8f9b 07d5
0000be0 6fff f4c6 4e75 ebee ec9c f8e4 d2be afbd
0000bf0 3f74 3e83 9fee 2cc0 3afb f6f7 ee75 83bd
0000c00 b21a 2150 6f6e 1ec8 afb8 f47b 7dc8 b3d5
0000c10 ee67 9eeb 723d f75f 79f4 fb5f dafc eda7
0000c20 9f6b 515e afa4 dc7d f6be 2f71 e16d 153f
0000c30 f5e6 4f95 57dc 763e 2499 8184 d0ea 4957
0000c40 5f70 c125 8d7d f724 9235 57dc 7349 255f
0000c50 7dcd 2495 55f7 dc92 01d7 d7de de01 6b69
0000c60 3afa db40 01d7 4bda f85b 694e f524 0075
0000c70 afb7 b803 7697 ef90 0284 57da e201 0abe
0000c80 6310 f161 71b7 4265 02ab fec6 2ac0 740e
0000c90 4689 bc95 5b98 5c09 697c efd6 0694 2b48
0000ca0 2179 12b7 f7b8 33d2 fdf7 4947 b194 6719
0000cb0 80d1 8ce8 50a5 f012 0522 1c52 7dc6 0014
0000cc0 8951 d057 526c f875 6812 91ca 6e2f f91f
0000cd0 dee2 0bcb 0f7e d16a c047 7dbc cc04 847b
0000ce0 ee11 8c23 b5f0 b454 41ef fa28 1648 23ee
0000cf0 f859 5e5e 7bed 8850 023e 7de6 cc04 dd2f
0000d00 ff72 0604 c775 8cd2 3848 aa89 c4ea 47a6
0000d10 1832 131a 626e fdcd 97d3 0e25 28d1 52d4
0000d20 3595 6d3d fcad 06b6 5b6f b7fb 6db5 ed5a
0000d30 b4db 3fe2 7c10 677f af83 f230 2dbe fdeb
0000d40 c103 5bbf 6f37 7ac3 86df ff35 9381 3ff8
0000d50 5582 07a4 4a2b ebc0 7f9e 6913 a3ed 5c48
0000d60 6dfd 985c 94f0 91b9 0fc8 3001 aaaa 6ea2
0000d70 2e14 e6c3 9bb5 6994 05a5 01ce d1e2 170d
0000d80 6f68 475c 8307 ded5 9405 4ff9 0d3c 3060
0000d90 8c71 d35b 55e9 bddc bb8a 24d7 6d5f 7c53
0000da0 4db5 b7f1 e2b4 10bf ab97 6a65 96af 3da9
0000db0 d7d8 26fc e059 4055 abe5 ca80 72c7 ff82
0000dc0 c336 0859 9f10 0085 b1f1 ff9a 3009 8aba
0000dd0 7ce7 cf15 5bf9 ed67 e85f 2af0 bf9e e78a
0000de0 65bf ff26 144f 2167 7b80 0216 97b8 b966
0000df0 977f 0189 dc6f 8d4d 27e2 069e 7250 98a0
0000e00 714b 7e08 438a 8a70 f243 1c53 5392 fa1c
0000e10 1c53 8a7a ee43 1c53 5372 7a18 c29f 9bd0
0000e20 50e2 14fc e287 8714 a7de f438 30a6 3f34
0000e30 a185 5a39 e4d9 654f 445e 918b 0ecc 4fb9
0000e40 e43b cb1e 7fc6 f24d 0ed3 4fb5 d43b 55a6
0000e50 4fbe 7cab cd2f 7f82 8c27 55a7 4fca 94ab
0000e60 80ef fc52 2b3f 569f 3f09 12ad 03fe ee28
0000e70 6cef 34f0 f143 0cd3 d7c5 8c32 b1ff 69f9
0000e80 dc89 25a7 af72 4895 c77e f0f1 4634 d3f1
0000e90 c518 1adf df01 4ce9 aa18 2bca 1b7f e617
0000ea0 1a00 6da0 b4d8 724a dc5f b782 16e3 e874
0000eb0 e938 71dc f9d2 795c 3bdc e99d b8ea b9ab
0000ec0 aee3 7be6 f82d 1a9f 8ee5 f04b 5c76 f782
0000ed0 0b5e e6fe 84d8 3b8e 7730 81dc 5cf9 dc79
0000ee0 353f 3b66 ab0e 38ed 082c 47cf 8159 17fe
0000ef0 55d0 5555 dff2 85c6 4949 feda 5c6d 7198
0000f00 3c36 cf0d a7fb 58c6 3443 e50a 67c2 2182
0000f10 3fb8 e1b9 64fe 1f38 c373 98f9 6ae1 0db8
0000f20 c357 16fd 1787 87ae 70dc 643d 1ab8 c371
0000f30 88d5 c5bf 70e1 727e f9c3 70c9 6f7e bf38
0000f40 f1c3 1b70 878e 857b 82a3 799a 4b58 7031
0000f50 de43 1b86 c2f2 8109 4e1b 0c08 0417 b86e
0000f60 3020 101c e1b8 c080 d570 5c37 dff5 f0e2
0000f70 fdc2 1b80 07ee 570c 1b7d faae 8b7e ffc3
0000f80 6703 0e1a 1b86 860e f007 2815 a38c 2c81
0000f90 0aa9 b06d 9d70 ecdb 66da b7d7 f5d9 de3e
0000fa0 3e66 cf1e 166d 3ecf 36de f1f3 9962 0cdb
0000fb0 9b4b 6961 0cdb 9b4b 795d ecdb 2c4a 2137
0000fc0 a521 43a4 368f c2c3 866d 336d 712d 699b
0000fd0 b369 712c 659b 3369 b8c4 10cd 0cdb dc08
0000fe0 4066 6fe0 b989 d53f 9b53 c61d 766d 66d8
0000ff0 e640 0336 7f32 c36b 1ffc d338 8666 9b71
0001000

 

[fiona]

Hello Fiona,

Yes both nodes in the same cluster. They are using same iSCSI disk with different subdirectory inside. one of is running in /mnt/prxmxbck/XXX01 other one is /mnt/prxmxbck/XXX02. Every node have own their PBS installed on PVE and every PVE uses own PBS server connected with 127.0.0.1. I also tried creating an another iSCSI share with fresh disk, the issue is still persisted.

Oh, okay. Then please also share the output of proxmox-backup-manager versions --verbose from the working and broken node. Because this is much more likely a server-side issue rather than a Proxmox VE-side issue.

HEAD command:

0000000 930d 0e44 3d18 991a a501 313b cf21 8e12
0000010 3f26 501a 55a8 63a9 24a5 c623 4649 abbd
0000020 ab43 edc6 e1dd d5f5 f159 2ecc 400f 2282
0000030 82b6 9541 864a 11d5 07cd 8da7 16b8 5081
0000040 03ba f07b 9100 220d 74a8 259a 927a efe3
0000050 cfca 448c b24e 8423 2fa2 bc05 0a2c b8aa
0000060 1a3d aad4 bc0d 262b d460 1c21 2150 006e
0000070 9edd 231d 9723 6cca 62c2 0446 af14 dac2
0000080 a9a1 8395 6c1f 1ccf 5254 7c2c b3e1 2504
0000090 0396 1acf d547 41b3 8577 448a a585 9138
00000a0 40ae 59ed 32a9 75aa 4966 cadc 022c c172
00000b0 9f00 3a29 591f 577d 6c4a 78ed b4d4 30d2
00000c0 0383 d051 2448 8088 d917 9e1d 959c f74a
00000d0 8064 7722 cd3a 1182 2168 4a2f 0310 bc4c
00000e0 94cf bfff 0cd6 432e 915e 7b16 9a27 b00c
00000f0 e668 6bff a4cb 0064 fffc b2da 0741 1df1
0000100 9023 f87d 5238 c0c0 0c42 fff8 6db5 dc2e
0000110 ac9c d9c8 c925 95a9 9481 8113 a810 aa1c
0000120 5432 7373 2735 a792 50a6 59d0 45b1 2ade
0000130 844a 936b 4c2a 31f8 bb6c f84c e8f3 91cb
0000140 0508 6184 721c 505c d313 5024 a1c8 e8a0
0000150 27b2 22f2 d4f3 fd0a ba51 137d bada 8b26
0000160 9c4c 5234 32da 2839 1dd2 302a 6bba a141
0000170 e852 5ab2 6591 f451 79cc 8c6f be6c 48de
0000180 fb6a 5330 9b52 62f3 fbc8 daff c232 9746
0000190 6298 58e4 6294 8445 a2e1 1a2a 4295 91a5
00001a0 0687 133e 2498 4d0f aff4 8335 0581 fff8
00001b0 65b5 2a4e 96b3 283e b226 96d2 7661 fcbc
00001c0 f2f3 6bfa a4cb 2f50 6fb3 9038 92d9 a4f6
00001d0 5120 b119 23ae 2130 cdd5 6318 a3a1 339d
00001e0 ff57 8b5f a066 6aab d482 d74c 10c6 0d22
00001f0 3341 fff1 96d7 0139 3bcc 84b3 0a6d 0000
0000200 ca48 5234 f342 bffe ccb6 a3c9 c64b ffc6
0000210 3e95 45fe 0a6a 9805 a047 8e02 d317 d7ff
0000220 0996 997e 7ffe 996d 1914 a8d9 b0ae 262a
0000230 0c44 04c2 b4c1 a24e 11a6 20b2 1af3 3925
0000240 21d2 d646 502a afe0 732d 22d2 fff0 cb6b
0000250 ac9c ed61 60f0 93f0 6674 4936 1170 ef33
0000260 8942 7920 678e 28cd 8a2e 9cd4 6064 7011
0000270 1360 4178 cbb0 081b 48d3 0d07 f092 4516
0000280 aac9 73ec 04b5 0808 83a2 fc88 b5ff 8265
0000290 ed86 d7ff 19a2 bc10 1f99 3411 5f6b ed34
00002a0 c57a b15e ac57 4510 886e 6694 1230 65c4
00002b0 8f84 30e2 4352 47d1 5803 0d26 0156 a4cb
00002c0 8158 12db a0d5 b1bd e803 44ce ab18 bfc0
00002d0 ccb6 8909 fff7 cb6b f49a 24a1 e433 9687
00002e0 3904 58a2 d289 f607 26e3 54ba 7322 4ab1
00002f0 c18f 85ff b312 597f 2659 ac5e 5821 da2c
0000300 2d74 1d10 c889 1cae fe16 b6bf 504c ec4b
0000310 6bff 64db c898 907e 0092 1b8c 871d 5cc0
0000320 7ffe 9b6d 138c 2b79 e4ce 958f 111c b2a4
0000330 daae bee0 a322 1840 81c0 260f 805a 1a40
0000340 26a1 6bde 6016 60a4 56ac 15a6 742c c47f
0000350 6d7f 2099 2616 139f 7265 f8c4 6258 bd29
0000360 1fc9 fca0 6d7f 9399 ef18 1c13 0bdc 0935
0000370 ae4a 7ffc 996d c694 ffc8 6daf 018b 46e9
0000380 9cde d0a8 d268 41e9 f02d 6bff e0db b7dc
0000390 9409 6529 0f1a 2034 e04e fed4 b6bf c9ac
00003a0 371d a04c 8d17 88c3 fcbe daff a9b6 0c27
00003b0 3508 4ef1 bfff ccb6 a389 7fbc a280 0a9f
00003c0 effe ffe0 9c54 195a e561 d309 9f34 a748
00003d0 cc4f f962 2355 6c64 c19a 7851 a0a3 6b50
00003e0 4383 e97b fb4c 3ba2 cb02 7508 cde4 eec9
00003f0 9ab8 12e9 4e4d 69e7 345f ed9b 7583 6dc4
0000400 1c31 72c1 7da6 3998 65b2 6584 2757 acce
0000410 e364 0158 7475 389d 510b 0846 d9cf 6e99
0000420 129e e182 034c bcdf 1fac 429f e4ae 87b4
0000430 0f9a 3037 c3eb c3ed 8b6c bf31 3a91 c162
0000440 911e 5394 3e3f 3b82 caae f708 ded4 794c
0000450 7a28 8542 8b6d a861 232d bb94 5374 e826
0000460 3af7 72c9 9875 10c5 1574 5174 64da 26e1
0000470 7228 1103 6146 7593 86c2 ccc3 062c ec88
0000480 a888 8c59 3a3c a2ca 7640 b10e afff 132d
0000490 21c4 a973 1195 788a d055 7ffd 19ad a19c
00004a0 27d3 ff23 73ff ca24 14b6 021c 744d f919
00004b0 db6b dc64 d3a0 9cbb 4c2d 7cb3 7b6e 9874
00004c0 d380 59bd 874f cd9c f29b 55b9 2c00 69a2
00004d0 4e8f 65ef 9a52 44c0 3aa6 80ac b243 e77a
00004e0 68a1 5e46 e990 5b5f d846 1336 4de6 2825
00004f0 fffa 33da 4c08 68cc 65a8 b46f aad6 902c
0000500 1e69 9961 0914 8fee 9aa9 38aa ffd3 9ad7
0000510 d0c1 fec2 6d7f e099 f68d 6bff a4cb be08
0000520 2a31 a52f d188 c841 f222 6bff a4cb d9c0
0000530 afff fb6d a030 eab5 fe5d 8054 153f fff6
0000540 d780 fffd 0106 2e4d cd75 9496 5054 0bff
0000550 cdf5 324d fe61 6db5 e21f bad4 6154 3311
0000560 a234 0212 7416 2d1e 49d4 742b 8322 8181
0000570 e869 2f02 8db6 e40d 8d25 f285 16cc f4ff
0000580 b62d b259 1c36 cd65 1476 712f a449 8741
0000590 1180 583e a858 7e99 d869 21a0 8703 c0c5
00005a0 b944 0bcd 04ec 0514 a7ee 54ca 318b 05da
00005b0 0c5e 32dc f315 4632 c0bb 818b a65b a262
00005c0 44cf ce19 0592 07c4 81c9 2d14 18bc a44a
00005d0 6691 08cf 251f 8e8d 3219 9860 d06e 8190
00005e0 96c3 16a9 45ca 3e6c ac86 b1f5 b0d2 0941
00005f0 69c3 92c9 1335 9281 4aee 1857 4491 661a
0000600 18a4 73e1 5024 b11a 0d30 c21c 6434 80e0
0000610 0f75 3965 2d3e 1061 5a0b aa25 5571 97d8
0000620 22a2 3bac 2518 cce4 2323 94c5 f0e0 0fcd
0000630 0c8a 0dd3 5861 d0f7 21b5 5a53 89ea 6300
0000640 0745 80d5 2fb0 2cc5 85d6 2622 80f7 08c0
0000650 d1aa aaa1 0f96 285d 4662 dcc4 471c e150
0000660 6478 4041 92c0 9108 1642 80d0 d061 2620
0000670 f616 01b5 5a71 eb18 3340 cbe7 e3c8 b01e
0000680 530d 5f0d d388 2ac2 9a1a 16ae 2b4d d341
0000690 916c 805c bd38 2156 af4e cf55 b0b8 466a
00006a0 4ba1 3baf 8656 6594 3041 4795 5024 3398
00006b0 c01c c4c4 bec2 5c36 b0ab 57b4 9b48 f806
00006c0 524b 00c1 86d8 86a9 6baf 15d5 31c0 f85e
00006d0 2a08 1773 1fc0 45ad ed81 4015 6228 e897
00006e0 74a8 b491 4845 8b27 4381 1111 8942 ccc2
00006f0 8c34 0703 0db0 5f61 b01b 5604 7c21 4652
0000700 9d87 7c36 f456 36c0 354c 617c 8409 9802
0000710 0a83 9284 470b 8762 1560 2842 5162 e85e
0000720 88b0 3a5d 436c 465d 8cba e9c0 e29b 0658
0000730 430c 0698 d12b d829 1d10 e924 f934 c258
0000740 b053 c450 010c 5db1 43fa 21cd 5107 9bd3
0000750 9129 28bc 8d7b c01c 6434 d300 b5f0 ca81
0000760 a115 9399 8416 5a5f 5840 7a14 3c67 12c0
0000770 2568 e54a b08a 50d1 9611 3840 512b 2a38
0000780 7057 8186 424f c1b3 1428 b7b1 c045 cc36
0000790 102f c090 8a1e f988 1210 be64 6834 6698
00007a0 6c20 741c 5303 c535 9652 620c ed6e 0516
00007b0 2284 1746 b9ef 2169 2906 5113 a313 9120
00007c0 1fbc c67b 7184 00c0 e1a6 836b 19a0 4409
00007d0 6c71 b72a c112 c8d8 1620 81e6 99c4 5c1a
00007e0 6fde 4579 85a9 9add b042 530d 5f0d 9307
00007f0 07b4 5938 ca00 0c44 453a ffec 370f 2918
0000800 8908 8901 8b36 1420 399a 0c89 1322 7220
0000810 2b08 3f45 7071 b46d 0703 7091 2365 c346
0000820 c394 a986 a986 6098 6063 ecd7 4f63 098c
0000830 383a 8a1c e071 8180 3344 d421 5811 d755
0000840 3006 2757 4b12 d1e6 8270 cf8d 3b2e aa76
0000850 f609 5194 c2d3 3084 2c88 5f06 5162 2b00
0000860 32e1 52ac 0025 07a1 9c2c 3830 3429 5a4e
0000870 6091 5085 0d18 361c 986c 986a c18a c129
0000880 65fe 4918 11a0 e1c7 a4d0 ea0d 110e 752d
0000890 8887 0c76 8603 3131 1c0e ac30 cbd9 f36b
00008a0 f6d0 43d5 e1db 6bed d787 dc03 0f57 87b0
00008b0 afb7 de1e 710f 3f5f 97c4 c96e b66f d1c9
00008c0 0fd7 5f72 c83f e61f 7ebe 8f98 f635 3f37
00008d0 e8d8 07eb afb7 7a1f e808 a2a0 9d4d 27a5
00008e0 da4f 3ebe 0569 bedf be0a e04f 3ebe 9f81
00008f0 7dbc f27d 9142 0afb 8645 f732 ca15 250c
0000900 2bee 1894 db8a 2857 5fae 9854 651c 645e
0000910 b061 bef3 79da b71d 75af 3b5c 5f6f bced
0000920 dc76 dabe ef71 7dc8 23bd 89df 7afb 3e27
0000930 9fd6 277b 1614 7afa ee07 1deb 4ef8 d7db
0000940 2d3b 320f 3546 644a 8141 fb74 e90a 81ae
0000950 bafb 8606 fb8c 190a c243 857d db84 afbc
0000960 ce6d ee46 1b2b cd99 57dc f336 78b1 77c8
0000970 1c72 57d0 6f36 d95f 7dbc 0602 acd9 2c78
0000980 819e f6f3 ce15 192b 55f7 8632 f723 8e15
0000990 270c 2bee 589c db4e 3957 9c6d afb9 6671
00009a0 ef39 9cab ab1f 7e4f 83f3 5f58 c0e3 8d7d
00009b0 ef03 8a98 c842 0925 4b73 f790 c835 b881
00009c0 057d 06e2 fb79 e41a e20d 35f6 2b88 ee44
00009d0 10ab 0cf9 c0fc ba7f ec5a 10ab afb8 6042
00009e0 7121 855f 42c4 beda 690a 4875 40f1 21c1
00009f0 2291 493d bede 7926 c491 457d 2712 fb91
0000a00 449a 124e 35f7 ac49 ee64 92ab c979 bafb
0000a10 4be4 beda 692e c095 557d e702 1208 86c6
0000a20 0d94 b715 54af 29dc 5f70 c0a7 e254 52be
0000a30 a989 7dc0 02a5 9157 5afb ae45 f732 cab5
0000a40 e56c b57d 2af2 5f6f bcab 6838 5a3c 4401
0000a50 4bd1 57db 6d2f b797 5daf 2f5e 5f70 c0bd
0000a60 de5e 7abe b379 5f71 71b3 ee56 566b cece
0000a70 cd7d 59ce afb7 9759 ca2f 6cec c6ca 4203
0000a80 b819 65af 46e0 afb9 728d b999 65af 32e6
0000a90 5f72 c8cb de67 33d7 d33f f5f6 7b4c edc6
0000aa0 19eb 4cd7 57dc 7b33 5a64 7516 596d 70f2
0000ab0 cafb edc3 fb83 07da 88f3 cafb e623 f701
0000ac0 0795 27ec d5f7 f64f fb9b 37ea 78f7 7b5f
0000ad0 efdc abed abdf 7527 01c0 0bda 9312 c09a
0000ae0 6d7d 0702 7dca 940d cc9b 6d7d 3732 fb91
0000af0 44da 6c16 2afb 3ed8 abea dba0 4157 58f7
0000b00 fe3e 0e57 abe9 fbe0 782a 7127 9d5f d6c4
0000b10 ee13 4bb3 1af1 f5f7 171a 7dce 9c2d df0d
0000b20 c357 e75f 5ebe c3e7 afb7 ef87 59a5 a3ee
0000b30 aa32 f5f4 edd0 a1eb ac7b f21e 876f 5f43
0000b40 db0f 1ebe 86b6 5f72 e50d 601b f02a fa15
0000b50 6061 db8b 2dd7 226e d7dc 7844 dc4b 2dd7
0000b60 16f1 afb8 605b b945 8aaf e2f2 bafb 1e78
0000b70 7d95 9c55 aafb 1f38 cfcb 2bfd fafd f8aa
0000b80 55f6 cdf1 fa8b 6912 ceea 25de f6fc 7e15
0000b90 e2ff febe b0c5 57dc 8658 bee3 f1c2 b98f
0000ba0 3faf 2066 2bef a390 d038 2057 5f6d b481
0000bb0 e2c7 bf01 7dad fd7d f701 07f5 278c c7d5
0000bc0 4765 2065 3449 bede 78d2 72a9 a95f e434
0000bd0 d2be 6990 7dc4 11a5 cdcf 9d7d 8f9b 07d5
0000be0 6fff f4c6 4e75 ebee ec9c f8e4 d2be afbd
0000bf0 3f74 3e83 9fee 2cc0 3afb f6f7 ee75 83bd
0000c00 b21a 2150 6f6e 1ec8 afb8 f47b 7dc8 b3d5
0000c10 ee67 9eeb 723d f75f 79f4 fb5f dafc eda7
0000c20 9f6b 515e afa4 dc7d f6be 2f71 e16d 153f
0000c30 f5e6 4f95 57dc 763e 2499 8184 d0ea 4957
0000c40 5f70 c125 8d7d f724 9235 57dc 7349 255f
0000c50 7dcd 2495 55f7 dc92 01d7 d7de de01 6b69
0000c60 3afa db40 01d7 4bda f85b 694e f524 0075
0000c70 afb7 b803 7697 ef90 0284 57da e201 0abe
0000c80 6310 f161 71b7 4265 02ab fec6 2ac0 740e
0000c90 4689 bc95 5b98 5c09 697c efd6 0694 2b48
0000ca0 2179 12b7 f7b8 33d2 fdf7 4947 b194 6719
0000cb0 80d1 8ce8 50a5 f012 0522 1c52 7dc6 0014
0000cc0 8951 d057 526c f875 6812 91ca 6e2f f91f
0000cd0 dee2 0bcb 0f7e d16a c047 7dbc cc04 847b
0000ce0 ee11 8c23 b5f0 b454 41ef fa28 1648 23ee
0000cf0 f859 5e5e 7bed 8850 023e 7de6 cc04 dd2f
0000d00 ff72 0604 c775 8cd2 3848 aa89 c4ea 47a6
0000d10 1832 131a 626e fdcd 97d3 0e25 28d1 52d4
0000d20 3595 6d3d fcad 06b6 5b6f b7fb 6db5 ed5a
0000d30 b4db 3fe2 7c10 677f af83 f230 2dbe fdeb
0000d40 c103 5bbf 6f37 7ac3 86df ff35 9381 3ff8
0000d50 5582 07a4 4a2b ebc0 7f9e 6913 a3ed 5c48
0000d60 6dfd 985c 94f0 91b9 0fc8 3001 aaaa 6ea2
0000d70 2e14 e6c3 9bb5 6994 05a5 01ce d1e2 170d
0000d80 6f68 475c 8307 ded5 9405 4ff9 0d3c 3060
0000d90 8c71 d35b 55e9 bddc bb8a 24d7 6d5f 7c53
0000da0 4db5 b7f1 e2b4 10bf ab97 6a65 96af 3da9
0000db0 d7d8 26fc e059 4055 abe5 ca80 72c7 ff82
0000dc0 c336 0859 9f10 0085 b1f1 ff9a 3009 8aba
0000dd0 7ce7 cf15 5bf9 ed67 e85f 2af0 bf9e e78a
0000de0 65bf ff26 144f 2167 7b80 0216 97b8 b966
0000df0 977f 0189 dc6f 8d4d 27e2 069e 7250 98a0
0000e00 714b 7e08 438a 8a70 f243 1c53 5392 fa1c
0000e10 1c53 8a7a ee43 1c53 5372 7a18 c29f 9bd0
0000e20 50e2 14fc e287 8714 a7de f438 30a6 3f34
0000e30 a185 5a39 e4d9 654f 445e 918b 0ecc 4fb9
0000e40 e43b cb1e 7fc6 f24d 0ed3 4fb5 d43b 55a6
0000e50 4fbe 7cab cd2f 7f82 8c27 55a7 4fca 94ab
0000e60 80ef fc52 2b3f 569f 3f09 12ad 03fe ee28
0000e70 6cef 34f0 f143 0cd3 d7c5 8c32 b1ff 69f9
0000e80 dc89 25a7 af72 4895 c77e f0f1 4634 d3f1
0000e90 c518 1adf df01 4ce9 aa18 2bca 1b7f e617
0000ea0 1a00 6da0 b4d8 724a dc5f b782 16e3 e874
0000eb0 e938 71dc f9d2 795c 3bdc e99d b8ea b9ab
0000ec0 aee3 7be6 f82d 1a9f 8ee5 f04b 5c76 f782
0000ed0 0b5e e6fe 84d8 3b8e 7730 81dc 5cf9 dc79
0000ee0 353f 3b66 ab0e 38ed 082c 47cf 8159 17fe
0000ef0 55d0 5555 dff2 85c6 4949 feda 5c6d 7198
0000f00 3c36 cf0d a7fb 58c6 3443 e50a 67c2 2182
0000f10 3fb8 e1b9 64fe 1f38 c373 98f9 6ae1 0db8
0000f20 c357 16fd 1787 87ae 70dc 643d 1ab8 c371
0000f30 88d5 c5bf 70e1 727e f9c3 70c9 6f7e bf38
0000f40 f1c3 1b70 878e 857b 82a3 799a 4b58 7031
0000f50 de43 1b86 c2f2 8109 4e1b 0c08 0417 b86e
0000f60 3020 101c e1b8 c080 d570 5c37 dff5 f0e2
0000f70 fdc2 1b80 07ee 570c 1b7d faae 8b7e ffc3
0000f80 6703 0e1a 1b86 860e f007 2815 a38c 2c81
0000f90 0aa9 b06d 9d70 ecdb 66da b7d7 f5d9 de3e
0000fa0 3e66 cf1e 166d 3ecf 36de f1f3 9962 0cdb
0000fb0 9b4b 6961 0cdb 9b4b 795d ecdb 2c4a 2137
0000fc0 a521 43a4 368f c2c3 866d 336d 712d 699b
0000fd0 b369 712c 659b 3369 b8c4 10cd 0cdb dc08
0000fe0 4066 6fe0 b989 d53f 9b53 c61d 766d 66d8
0000ff0 e640 0336 7f32 c36b 1ffc d338 8666 9b71
0001000

It is corrupted, it starts with the wrong bytes and actual data should be much shorter (rest padded with zeroes).

 

[mracar]

Oh, okay. Then please also share the output of proxmox-backup-manager versions --verbose from the working and broken node. Because this is much more likely a server-side issue rather than a Proxmox VE-side issue.

It is corrupted, it starts with the wrong bytes and actual data should be much shorter (rest padded with zeroes).

Working node:
proxmox-backup unknown running kernel: 6.8.4-3-pve
proxmox-backup-server 3.2.7-1 running version: 3.2.3
proxmox-kernel-helper 8.1.0
pve-kernel-5.15 7.4-13
proxmox-kernel-6.8 6.8.8-2
proxmox-kernel-6.8.4-3-pve-signed 6.8.4-3
pve-kernel-5.15.152-1-pve 5.15.152-1
pve-kernel-5.15.102-1-pve 5.15.102-1
ifupdown2 3.2.0-1+pmx8
libjs-extjs 7.0.0-4
proxmox-backup-docs 3.2.7-1
proxmox-backup-client 3.2.7-1
proxmox-mail-forward 0.2.3
proxmox-mini-journalreader 1.4.0
proxmox-offline-mirror-helper 0.6.6
proxmox-widget-toolkit 4.2.3
pve-xtermjs 5.3.0-3
smartmontools 7.3-pve1
zfsutils-linux 2.2.4-pve1

Broken node:

proxmox-backup unknown running kernel: 6.8.8-1-pve
proxmox-backup-server 3.2.7-1 running version: 3.2.4
proxmox-kernel-helper 8.1.0
pve-kernel-5.15 7.4-14
proxmox-kernel-6.8 6.8.8-2
proxmox-kernel-6.8.8-1-pve-signed 6.8.8-1
pve-kernel-5.15.158-1-pve 5.15.158-1
pve-kernel-5.15.102-1-pve 5.15.102-1
ifupdown2 3.2.0-1+pmx8
libjs-extjs 7.0.0-4
proxmox-backup-docs 3.2.7-1
proxmox-backup-client 3.2.7-1
proxmox-mail-forward 0.2.3
proxmox-mini-journalreader 1.4.0
proxmox-offline-mirror-helper 0.6.6
proxmox-widget-toolkit 4.2.3
pve-xtermjs 5.3.0-3
smartmontools 7.3-pve1
zfsutils-linux 2.2.4-pve1

 

[Chris]

Hello Fiona,

Yes both nodes in the same cluster. They are using same iSCSI disk with different subdirectory inside. one of is running in /mnt/prxmxbck/XXX01 other one is /mnt/prxmxbck/XXX02. Every node have own their PBS installed on PVE and every PVE uses own PBS server connected with 127.0.0.1. I also tried creating an another iSCSI share with fresh disk, the issue is still persisted.


Working node:
proxmox-ve: 8.2.0 (running kernel: 6.8.4-3-pve)
pve-manager: 8.2.2 (running version: 8.2.2/9355359cd7afbae4)
proxmox-kernel-helper: 8.1.0
pve-kernel-5.15: 7.4-13
proxmox-kernel-6.8: 6.8.4-3
proxmox-kernel-6.8.4-3-pve-signed: 6.8.4-3
pve-kernel-5.15.152-1-pve: 5.15.152-1
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.6
libpve-cluster-perl: 8.0.6
libpve-common-perl: 8.2.1
libpve-guest-common-perl: 5.1.2
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.2.1
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.2.3-1
proxmox-backup-file-restore: 3.2.3-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.6
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.6
pve-container: 5.1.10
pve-docs: 8.2.2
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.0
pve-firewall: 5.0.7
pve-firmware: 3.11-1
pve-ha-manager: 4.0.4
pve-i18n: 3.2.2
pve-qemu-kvm: 8.1.5-6
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.3-pve2


Broken node:
proxmox-ve: 8.2.0 (running kernel: 6.8.8-1-pve)
pve-manager: 8.2.4 (running version: 8.2.4/faa83925c9641325)
proxmox-kernel-helper: 8.1.0
pve-kernel-5.15: 7.4-14
proxmox-kernel-6.8: 6.8.8-1
proxmox-kernel-6.8.8-1-pve-signed: 6.8.8-1
pve-kernel-5.15.158-1-pve: 5.15.158-1
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.7
libpve-cluster-perl: 8.0.7
libpve-common-perl: 8.2.1
libpve-guest-common-perl: 5.1.3
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.9
libpve-storage-perl: 8.2.2
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.2.4-1
proxmox-backup-file-restore: 3.2.4-1
proxmox-firewall: 0.4.2
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.6
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.7
pve-container: 5.1.12
pve-docs: 8.2.2
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.1
pve-firewall: 5.0.7
pve-firmware: 3.12-1
pve-ha-manager: 4.0.5
pve-i18n: 3.2.2
pve-qemu-kvm: 8.1.5-6
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.4-pve1

HEAD command:

0000000 930d 0e44 3d18 991a a501 313b cf21 8e12
0000010 3f26 501a 55a8 63a9 24a5 c623 4649 abbd
0000020 ab43 edc6 e1dd d5f5 f159 2ecc 400f 2282
0000030 82b6 9541 864a 11d5 07cd 8da7 16b8 5081
0000040 03ba f07b 9100 220d 74a8 259a 927a efe3
0000050 cfca 448c b24e 8423 2fa2 bc05 0a2c b8aa
0000060 1a3d aad4 bc0d 262b d460 1c21 2150 006e
0000070 9edd 231d 9723 6cca 62c2 0446 af14 dac2
0000080 a9a1 8395 6c1f 1ccf 5254 7c2c b3e1 2504
0000090 0396 1acf d547 41b3 8577 448a a585 9138
00000a0 40ae 59ed 32a9 75aa 4966 cadc 022c c172
00000b0 9f00 3a29 591f 577d 6c4a 78ed b4d4 30d2
00000c0 0383 d051 2448 8088 d917 9e1d 959c f74a
00000d0 8064 7722 cd3a 1182 2168 4a2f 0310 bc4c
00000e0 94cf bfff 0cd6 432e 915e 7b16 9a27 b00c
00000f0 e668 6bff a4cb 0064 fffc b2da 0741 1df1
0000100 9023 f87d 5238 c0c0 0c42 fff8 6db5 dc2e
0000110 ac9c d9c8 c925 95a9 9481 8113 a810 aa1c
0000120 5432 7373 2735 a792 50a6 59d0 45b1 2ade
0000130 844a 936b 4c2a 31f8 bb6c f84c e8f3 91cb
0000140 0508 6184 721c 505c d313 5024 a1c8 e8a0
0000150 27b2 22f2 d4f3 fd0a ba51 137d bada 8b26
0000160 9c4c 5234 32da 2839 1dd2 302a 6bba a141
0000170 e852 5ab2 6591 f451 79cc 8c6f be6c 48de
0000180 fb6a 5330 9b52 62f3 fbc8 daff c232 9746
0000190 6298 58e4 6294 8445 a2e1 1a2a 4295 91a5
00001a0 0687 133e 2498 4d0f aff4 8335 0581 fff8
00001b0 65b5 2a4e 96b3 283e b226 96d2 7661 fcbc
00001c0 f2f3 6bfa a4cb 2f50 6fb3 9038 92d9 a4f6
00001d0 5120 b119 23ae 2130 cdd5 6318 a3a1 339d
00001e0 ff57 8b5f a066 6aab d482 d74c 10c6 0d22
00001f0 3341 fff1 96d7 0139 3bcc 84b3 0a6d 0000
0000200 ca48 5234 f342 bffe ccb6 a3c9 c64b ffc6
0000210 3e95 45fe 0a6a 9805 a047 8e02 d317 d7ff
0000220 0996 997e 7ffe 996d 1914 a8d9 b0ae 262a
0000230 0c44 04c2 b4c1 a24e 11a6 20b2 1af3 3925
0000240 21d2 d646 502a afe0 732d 22d2 fff0 cb6b
0000250 ac9c ed61 60f0 93f0 6674 4936 1170 ef33
0000260 8942 7920 678e 28cd 8a2e 9cd4 6064 7011
0000270 1360 4178 cbb0 081b 48d3 0d07 f092 4516
0000280 aac9 73ec 04b5 0808 83a2 fc88 b5ff 8265
0000290 ed86 d7ff 19a2 bc10 1f99 3411 5f6b ed34
00002a0 c57a b15e ac57 4510 886e 6694 1230 65c4
00002b0 8f84 30e2 4352 47d1 5803 0d26 0156 a4cb
00002c0 8158 12db a0d5 b1bd e803 44ce ab18 bfc0
00002d0 ccb6 8909 fff7 cb6b f49a 24a1 e433 9687
00002e0 3904 58a2 d289 f607 26e3 54ba 7322 4ab1
00002f0 c18f 85ff b312 597f 2659 ac5e 5821 da2c
0000300 2d74 1d10 c889 1cae fe16 b6bf 504c ec4b
0000310 6bff 64db c898 907e 0092 1b8c 871d 5cc0
0000320 7ffe 9b6d 138c 2b79 e4ce 958f 111c b2a4
0000330 daae bee0 a322 1840 81c0 260f 805a 1a40
0000340 26a1 6bde 6016 60a4 56ac 15a6 742c c47f
0000350 6d7f 2099 2616 139f 7265 f8c4 6258 bd29
0000360 1fc9 fca0 6d7f 9399 ef18 1c13 0bdc 0935
0000370 ae4a 7ffc 996d c694 ffc8 6daf 018b 46e9
0000380 9cde d0a8 d268 41e9 f02d 6bff e0db b7dc
0000390 9409 6529 0f1a 2034 e04e fed4 b6bf c9ac
00003a0 371d a04c 8d17 88c3 fcbe daff a9b6 0c27
00003b0 3508 4ef1 bfff ccb6 a389 7fbc a280 0a9f
00003c0 effe ffe0 9c54 195a e561 d309 9f34 a748
00003d0 cc4f f962 2355 6c64 c19a 7851 a0a3 6b50
00003e0 4383 e97b fb4c 3ba2 cb02 7508 cde4 eec9
00003f0 9ab8 12e9 4e4d 69e7 345f ed9b 7583 6dc4
0000400 1c31 72c1 7da6 3998 65b2 6584 2757 acce
0000410 e364 0158 7475 389d 510b 0846 d9cf 6e99
0000420 129e e182 034c bcdf 1fac 429f e4ae 87b4
0000430 0f9a 3037 c3eb c3ed 8b6c bf31 3a91 c162
0000440 911e 5394 3e3f 3b82 caae f708 ded4 794c
0000450 7a28 8542 8b6d a861 232d bb94 5374 e826
0000460 3af7 72c9 9875 10c5 1574 5174 64da 26e1
0000470 7228 1103 6146 7593 86c2 ccc3 062c ec88
0000480 a888 8c59 3a3c a2ca 7640 b10e afff 132d
0000490 21c4 a973 1195 788a d055 7ffd 19ad a19c
00004a0 27d3 ff23 73ff ca24 14b6 021c 744d f919
00004b0 db6b dc64 d3a0 9cbb 4c2d 7cb3 7b6e 9874
00004c0 d380 59bd 874f cd9c f29b 55b9 2c00 69a2
00004d0 4e8f 65ef 9a52 44c0 3aa6 80ac b243 e77a
00004e0 68a1 5e46 e990 5b5f d846 1336 4de6 2825
00004f0 fffa 33da 4c08 68cc 65a8 b46f aad6 902c
0000500 1e69 9961 0914 8fee 9aa9 38aa ffd3 9ad7
0000510 d0c1 fec2 6d7f e099 f68d 6bff a4cb be08
0000520 2a31 a52f d188 c841 f222 6bff a4cb d9c0
0000530 afff fb6d a030 eab5 fe5d 8054 153f fff6
0000540 d780 fffd 0106 2e4d cd75 9496 5054 0bff
0000550 cdf5 324d fe61 6db5 e21f bad4 6154 3311
0000560 a234 0212 7416 2d1e 49d4 742b 8322 8181
0000570 e869 2f02 8db6 e40d 8d25 f285 16cc f4ff
0000580 b62d b259 1c36 cd65 1476 712f a449 8741
0000590 1180 583e a858 7e99 d869 21a0 8703 c0c5
00005a0 b944 0bcd 04ec 0514 a7ee 54ca 318b 05da
00005b0 0c5e 32dc f315 4632 c0bb 818b a65b a262
00005c0 44cf ce19 0592 07c4 81c9 2d14 18bc a44a
00005d0 6691 08cf 251f 8e8d 3219 9860 d06e 8190
00005e0 96c3 16a9 45ca 3e6c ac86 b1f5 b0d2 0941
00005f0 69c3 92c9 1335 9281 4aee 1857 4491 661a
0000600 18a4 73e1 5024 b11a 0d30 c21c 6434 80e0
0000610 0f75 3965 2d3e 1061 5a0b aa25 5571 97d8
0000620 22a2 3bac 2518 cce4 2323 94c5 f0e0 0fcd
0000630 0c8a 0dd3 5861 d0f7 21b5 5a53 89ea 6300
0000640 0745 80d5 2fb0 2cc5 85d6 2622 80f7 08c0
0000650 d1aa aaa1 0f96 285d 4662 dcc4 471c e150
0000660 6478 4041 92c0 9108 1642 80d0 d061 2620
0000670 f616 01b5 5a71 eb18 3340 cbe7 e3c8 b01e
0000680 530d 5f0d d388 2ac2 9a1a 16ae 2b4d d341
0000690 916c 805c bd38 2156 af4e cf55 b0b8 466a
00006a0 4ba1 3baf 8656 6594 3041 4795 5024 3398
00006b0 c01c c4c4 bec2 5c36 b0ab 57b4 9b48 f806
00006c0 524b 00c1 86d8 86a9 6baf 15d5 31c0 f85e
00006d0 2a08 1773 1fc0 45ad ed81 4015 6228 e897
00006e0 74a8 b491 4845 8b27 4381 1111 8942 ccc2
00006f0 8c34 0703 0db0 5f61 b01b 5604 7c21 4652
0000700 9d87 7c36 f456 36c0 354c 617c 8409 9802
0000710 0a83 9284 470b 8762 1560 2842 5162 e85e
0000720 88b0 3a5d 436c 465d 8cba e9c0 e29b 0658
0000730 430c 0698 d12b d829 1d10 e924 f934 c258
0000740 b053 c450 010c 5db1 43fa 21cd 5107 9bd3
0000750 9129 28bc 8d7b c01c 6434 d300 b5f0 ca81
0000760 a115 9399 8416 5a5f 5840 7a14 3c67 12c0
0000770 2568 e54a b08a 50d1 9611 3840 512b 2a38
0000780 7057 8186 424f c1b3 1428 b7b1 c045 cc36
0000790 102f c090 8a1e f988 1210 be64 6834 6698
00007a0 6c20 741c 5303 c535 9652 620c ed6e 0516
00007b0 2284 1746 b9ef 2169 2906 5113 a313 9120
00007c0 1fbc c67b 7184 00c0 e1a6 836b 19a0 4409
00007d0 6c71 b72a c112 c8d8 1620 81e6 99c4 5c1a
00007e0 6fde 4579 85a9 9add b042 530d 5f0d 9307
00007f0 07b4 5938 ca00 0c44 453a ffec 370f 2918
0000800 8908 8901 8b36 1420 399a 0c89 1322 7220
0000810 2b08 3f45 7071 b46d 0703 7091 2365 c346
0000820 c394 a986 a986 6098 6063 ecd7 4f63 098c
0000830 383a 8a1c e071 8180 3344 d421 5811 d755
0000840 3006 2757 4b12 d1e6 8270 cf8d 3b2e aa76
0000850 f609 5194 c2d3 3084 2c88 5f06 5162 2b00
0000860 32e1 52ac 0025 07a1 9c2c 3830 3429 5a4e
0000870 6091 5085 0d18 361c 986c 986a c18a c129
0000880 65fe 4918 11a0 e1c7 a4d0 ea0d 110e 752d
0000890 8887 0c76 8603 3131 1c0e ac30 cbd9 f36b
00008a0 f6d0 43d5 e1db 6bed d787 dc03 0f57 87b0
00008b0 afb7 de1e 710f 3f5f 97c4 c96e b66f d1c9
00008c0 0fd7 5f72 c83f e61f 7ebe 8f98 f635 3f37
00008d0 e8d8 07eb afb7 7a1f e808 a2a0 9d4d 27a5
00008e0 da4f 3ebe 0569 bedf be0a e04f 3ebe 9f81
00008f0 7dbc f27d 9142 0afb 8645 f732 ca15 250c
0000900 2bee 1894 db8a 2857 5fae 9854 651c 645e
0000910 b061 bef3 79da b71d 75af 3b5c 5f6f bced
0000920 dc76 dabe ef71 7dc8 23bd 89df 7afb 3e27
0000930 9fd6 277b 1614 7afa ee07 1deb 4ef8 d7db
0000940 2d3b 320f 3546 644a 8141 fb74 e90a 81ae
0000950 bafb 8606 fb8c 190a c243 857d db84 afbc
0000960 ce6d ee46 1b2b cd99 57dc f336 78b1 77c8
0000970 1c72 57d0 6f36 d95f 7dbc 0602 acd9 2c78
0000980 819e f6f3 ce15 192b 55f7 8632 f723 8e15
0000990 270c 2bee 589c db4e 3957 9c6d afb9 6671
00009a0 ef39 9cab ab1f 7e4f 83f3 5f58 c0e3 8d7d
00009b0 ef03 8a98 c842 0925 4b73 f790 c835 b881
00009c0 057d 06e2 fb79 e41a e20d 35f6 2b88 ee44
00009d0 10ab 0cf9 c0fc ba7f ec5a 10ab afb8 6042
00009e0 7121 855f 42c4 beda 690a 4875 40f1 21c1
00009f0 2291 493d bede 7926 c491 457d 2712 fb91
0000a00 449a 124e 35f7 ac49 ee64 92ab c979 bafb
0000a10 4be4 beda 692e c095 557d e702 1208 86c6
0000a20 0d94 b715 54af 29dc 5f70 c0a7 e254 52be
0000a30 a989 7dc0 02a5 9157 5afb ae45 f732 cab5
0000a40 e56c b57d 2af2 5f6f bcab 6838 5a3c 4401
0000a50 4bd1 57db 6d2f b797 5daf 2f5e 5f70 c0bd
0000a60 de5e 7abe b379 5f71 71b3 ee56 566b cece
0000a70 cd7d 59ce afb7 9759 ca2f 6cec c6ca 4203
0000a80 b819 65af 46e0 afb9 728d b999 65af 32e6
0000a90 5f72 c8cb de67 33d7 d33f f5f6 7b4c edc6
0000aa0 19eb 4cd7 57dc 7b33 5a64 7516 596d 70f2
0000ab0 cafb edc3 fb83 07da 88f3 cafb e623 f701
0000ac0 0795 27ec d5f7 f64f fb9b 37ea 78f7 7b5f
0000ad0 efdc abed abdf 7527 01c0 0bda 9312 c09a
0000ae0 6d7d 0702 7dca 940d cc9b 6d7d 3732 fb91
0000af0 44da 6c16 2afb 3ed8 abea dba0 4157 58f7
0000b00 fe3e 0e57 abe9 fbe0 782a 7127 9d5f d6c4
0000b10 ee13 4bb3 1af1 f5f7 171a 7dce 9c2d df0d
0000b20 c357 e75f 5ebe c3e7 afb7 ef87 59a5 a3ee
0000b30 aa32 f5f4 edd0 a1eb ac7b f21e 876f 5f43
0000b40 db0f 1ebe 86b6 5f72 e50d 601b f02a fa15
0000b50 6061 db8b 2dd7 226e d7dc 7844 dc4b 2dd7
0000b60 16f1 afb8 605b b945 8aaf e2f2 bafb 1e78
0000b70 7d95 9c55 aafb 1f38 cfcb 2bfd fafd f8aa
0000b80 55f6 cdf1 fa8b 6912 ceea 25de f6fc 7e15
0000b90 e2ff febe b0c5 57dc 8658 bee3 f1c2 b98f
0000ba0 3faf 2066 2bef a390 d038 2057 5f6d b481
0000bb0 e2c7 bf01 7dad fd7d f701 07f5 278c c7d5
0000bc0 4765 2065 3449 bede 78d2 72a9 a95f e434
0000bd0 d2be 6990 7dc4 11a5 cdcf 9d7d 8f9b 07d5
0000be0 6fff f4c6 4e75 ebee ec9c f8e4 d2be afbd
0000bf0 3f74 3e83 9fee 2cc0 3afb f6f7 ee75 83bd
0000c00 b21a 2150 6f6e 1ec8 afb8 f47b 7dc8 b3d5
0000c10 ee67 9eeb 723d f75f 79f4 fb5f dafc eda7
0000c20 9f6b 515e afa4 dc7d f6be 2f71 e16d 153f
0000c30 f5e6 4f95 57dc 763e 2499 8184 d0ea 4957
0000c40 5f70 c125 8d7d f724 9235 57dc 7349 255f
0000c50 7dcd 2495 55f7 dc92 01d7 d7de de01 6b69
0000c60 3afa db40 01d7 4bda f85b 694e f524 0075
0000c70 afb7 b803 7697 ef90 0284 57da e201 0abe
0000c80 6310 f161 71b7 4265 02ab fec6 2ac0 740e
0000c90 4689 bc95 5b98 5c09 697c efd6 0694 2b48
0000ca0 2179 12b7 f7b8 33d2 fdf7 4947 b194 6719
0000cb0 80d1 8ce8 50a5 f012 0522 1c52 7dc6 0014
0000cc0 8951 d057 526c f875 6812 91ca 6e2f f91f
0000cd0 dee2 0bcb 0f7e d16a c047 7dbc cc04 847b
0000ce0 ee11 8c23 b5f0 b454 41ef fa28 1648 23ee
0000cf0 f859 5e5e 7bed 8850 023e 7de6 cc04 dd2f
0000d00 ff72 0604 c775 8cd2 3848 aa89 c4ea 47a6
0000d10 1832 131a 626e fdcd 97d3 0e25 28d1 52d4
0000d20 3595 6d3d fcad 06b6 5b6f b7fb 6db5 ed5a
0000d30 b4db 3fe2 7c10 677f af83 f230 2dbe fdeb
0000d40 c103 5bbf 6f37 7ac3 86df ff35 9381 3ff8
0000d50 5582 07a4 4a2b ebc0 7f9e 6913 a3ed 5c48
0000d60 6dfd 985c 94f0 91b9 0fc8 3001 aaaa 6ea2
0000d70 2e14 e6c3 9bb5 6994 05a5 01ce d1e2 170d
0000d80 6f68 475c 8307 ded5 9405 4ff9 0d3c 3060
0000d90 8c71 d35b 55e9 bddc bb8a 24d7 6d5f 7c53
0000da0 4db5 b7f1 e2b4 10bf ab97 6a65 96af 3da9
0000db0 d7d8 26fc e059 4055 abe5 ca80 72c7 ff82
0000dc0 c336 0859 9f10 0085 b1f1 ff9a 3009 8aba
0000dd0 7ce7 cf15 5bf9 ed67 e85f 2af0 bf9e e78a
0000de0 65bf ff26 144f 2167 7b80 0216 97b8 b966
0000df0 977f 0189 dc6f 8d4d 27e2 069e 7250 98a0
0000e00 714b 7e08 438a 8a70 f243 1c53 5392 fa1c
0000e10 1c53 8a7a ee43 1c53 5372 7a18 c29f 9bd0
0000e20 50e2 14fc e287 8714 a7de f438 30a6 3f34
0000e30 a185 5a39 e4d9 654f 445e 918b 0ecc 4fb9
0000e40 e43b cb1e 7fc6 f24d 0ed3 4fb5 d43b 55a6
0000e50 4fbe 7cab cd2f 7f82 8c27 55a7 4fca 94ab
0000e60 80ef fc52 2b3f 569f 3f09 12ad 03fe ee28
0000e70 6cef 34f0 f143 0cd3 d7c5 8c32 b1ff 69f9
0000e80 dc89 25a7 af72 4895 c77e f0f1 4634 d3f1
0000e90 c518 1adf df01 4ce9 aa18 2bca 1b7f e617
0000ea0 1a00 6da0 b4d8 724a dc5f b782 16e3 e874
0000eb0 e938 71dc f9d2 795c 3bdc e99d b8ea b9ab
0000ec0 aee3 7be6 f82d 1a9f 8ee5 f04b 5c76 f782
0000ed0 0b5e e6fe 84d8 3b8e 7730 81dc 5cf9 dc79
0000ee0 353f 3b66 ab0e 38ed 082c 47cf 8159 17fe
0000ef0 55d0 5555 dff2 85c6 4949 feda 5c6d 7198
0000f00 3c36 cf0d a7fb 58c6 3443 e50a 67c2 2182
0000f10 3fb8 e1b9 64fe 1f38 c373 98f9 6ae1 0db8
0000f20 c357 16fd 1787 87ae 70dc 643d 1ab8 c371
0000f30 88d5 c5bf 70e1 727e f9c3 70c9 6f7e bf38
0000f40 f1c3 1b70 878e 857b 82a3 799a 4b58 7031
0000f50 de43 1b86 c2f2 8109 4e1b 0c08 0417 b86e
0000f60 3020 101c e1b8 c080 d570 5c37 dff5 f0e2
0000f70 fdc2 1b80 07ee 570c 1b7d faae 8b7e ffc3
0000f80 6703 0e1a 1b86 860e f007 2815 a38c 2c81
0000f90 0aa9 b06d 9d70 ecdb 66da b7d7 f5d9 de3e
0000fa0 3e66 cf1e 166d 3ecf 36de f1f3 9962 0cdb
0000fb0 9b4b 6961 0cdb 9b4b 795d ecdb 2c4a 2137
0000fc0 a521 43a4 368f c2c3 866d 336d 712d 699b
0000fd0 b369 712c 659b 3369 b8c4 10cd 0cdb dc08
0000fe0 4066 6fe0 b989 d53f 9b53 c61d 766d 66d8
0000ff0 e640 0336 7f32 c36b 1ffc d338 8666 9b71
0001000

This does not seem to be the same issue, so I would suggest to open a dedicated thread for this, and also making sure that both the VM disks truly have exclusive access to their iscsi target. From above output, it seems like something is overwriting your data.

Edit: Also please do reboot both PBS instances, as both of them are some binaries which have been replaced by another version. Then double check if the issue persists (note that you will of course have to remove the corrupt snapshot first).

 

[SiJux]

Hi Chris,

today I have one more information:
I have a complete new created VM which owns 3 Disks: 1x EFI-Disk, 1x 60GB for OS etc. and 1x 100GB for data
If I backup the VM the backup is broken.
If I detach the 100GB disk, the backup works.
If I detach the 60GB disk (with 100GB attached), the backup works.
After that I attached both disks the Backup works fine.
Tested with running/powered off VM

For all VMs which are affected: They have more than one disk.

Best Regards
Simon

 

[Chris]

Hi Chris,

today I have one more information:
I have a complete new created VM which owns 3 Disks: 1x EFI-Disk, 1x 60GB for OS etc. and 1x 100GB for data
If I backup the VM the backup is broken.
If I detach the 100GB disk, the backup works.
If I detach the 60GB disk (with 100GB attached), the backup works.
After that I attached both disks the Backup works fine.
Tested with running/powered off VM

For all VMs which are affected: They have more than one disk.

Best Regards
Simon

Thanks for sharing this additional information, the described behavior is very odd, but I do hope it helps to narrow down the issue. Please share the exact qm config <VMID> --current for this particular machine and (once again, for completeness) the pveversion -v of the host the VM is running on. Do you stop/shutdown the VM before detaching the disks or are you doing this on the running VM?

Also, please try and see if the issue persists with the latest version of qemu [0] if possible.

[0] https://forum.proxmox.com/threads/qemu-9-0-available-on-pve-no-subscription-as-of-now.149772/

 

[SiJux]

Hi Chris,

please see following configs of two vms:

root@pve:~# qm config 167 --current
agent: 1
balloon: 6144
bios: ovmf
boot: order=scsi0;ide2
cores: 2
cpu: EPYC
efidisk0: PVE_SATA_01:167/vm-167-disk-0.qcow2,efitype=4m,pre-enrolled-keys=1,size=528K
hotplug: disk,network,usb,memory,cpu
ide2: none,media=cdrom
lock: backup
machine: q35
memory: 8192
meta: creation-qemu=7.2.0,ctime=1686807027
name: elastic
net0: virtio=00:1a:4a:5e:XX:XX,bridge=vmbrXXX,firewall=1
numa: 1
onboot: 1
ostype: l26
rng0: source=/dev/urandom
scsi0: PVE_SSD_01:167/vm-167-disk-0.qcow2,iothread=1,size=20G
scsi1: PVE_SSD_01:167/vm-167-disk-1.qcow2,iothread=1,size=100G
scsihw: virtio-scsi-single
smbios1: uuid=168b7337-ef9d-4636-9951-f30da3b9de36
sockets: 1
vmgenid: 3500f9c7-f916-4e49-a8fd-a06401dc8c95



root@pve:~# qm config 123 --current
agent: 1
bios: ovmf
boot: order=scsi0;ide2;net0
cores: 2
cpu: EPYC
efidisk0: PVE_SATA_01:123/vm-123-disk-0.qcow2,efitype=4m,pre-enrolled-keys=1,size=528K
ide2: none,media=cdrom
memory: 8192
meta: creation-qemu=8.1.5,ctime=1720527743
name: mail
net0: virtio=BC:24:11:2D:XX:XX,bridge=vmbr1029,firewall=1
numa: 0
ostype: l26
scsi0: PVE_SATA_01:123/vm-123-disk-1.qcow2,iothread=1,size=60G
scsi1: PVE_SATA_01:123/vm-123-disk-2.qcow2,iothread=1,size=100G
scsihw: virtio-scsi-single
smbios1: uuid=381f9603-4c1a-42d7-b4bb-9ed80570cf1d
sockets: 1
vmgenid: da549816-d2cf-4f9d-aae2-6efa28d4eb05

and the Version-Information:

root@pve:~# pveversion -v
proxmox-ve: 8.2.0 (running kernel: 6.8.8-2-pve)
pve-manager: 8.2.4 (running version: 8.2.4/faa83925c9641325)
proxmox-kernel-helper: 8.1.0
pve-kernel-5.15: 7.4-13
proxmox-kernel-6.8: 6.8.8-2
proxmox-kernel-6.8.8-2-pve-signed: 6.8.8-2
proxmox-kernel-6.8.8-1-pve-signed: 6.8.8-1
proxmox-kernel-6.8.4-3-pve-signed: 6.8.4-3
pve-kernel-5.15.152-1-pve: 5.15.152-1
pve-kernel-5.15.149-1-pve: 5.15.149-1
pve-kernel-5.15.131-2-pve: 5.15.131-3
pve-kernel-5.15.131-1-pve: 5.15.131-2
pve-kernel-5.15.111-1-pve: 5.15.111-1
pve-kernel-5.15.102-1-pve: 5.15.102-1
amd64-microcode: 3.20230808.1.1~deb12u1
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.7
libpve-cluster-perl: 8.0.7
libpve-common-perl: 8.2.1
libpve-guest-common-perl: 5.1.3
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.9
libpve-storage-perl: 8.2.3
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.2.7-1
proxmox-backup-file-restore: 3.2.7-1
proxmox-firewall: 0.4.2
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.7
pve-container: 5.1.12
pve-docs: 8.2.2
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.1
pve-firewall: 5.0.7
pve-firmware: 3.12-1
pve-ha-manager: 4.0.5
pve-i18n: 3.2.2
pve-qemu-kvm: 9.0.0-6
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.4-pve1

Best regards
Simon

 

[SiJux]

Hi Chris,

one more information: After attaching both disks again, start the vm and take a Backup: the backup does not work anymore
The backups created with the powered off vm are verified now and I could restore files from the backup.

That's really strange...

Best regards
Simon

 

[Chris]

Hi Chris,

please see following configs of two vms:

root@pve:~# qm config 167 --current
agent: 1
balloon: 6144
bios: ovmf
boot: order=scsi0;ide2
cores: 2
cpu: EPYC
efidisk0: PVE_SATA_01:167/vm-167-disk-0.qcow2,efitype=4m,pre-enrolled-keys=1,size=528K
hotplug: disk,network,usb,memory,cpu
ide2: none,media=cdrom
lock: backup
machine: q35
memory: 8192
meta: creation-qemu=7.2.0,ctime=1686807027
name: elastic
net0: virtio=00:1a:4a:5e:XX:XX,bridge=vmbrXXX,firewall=1
numa: 1
onboot: 1
ostype: l26
rng0: source=/dev/urandom
scsi0: PVE_SSD_01:167/vm-167-disk-0.qcow2,iothread=1,size=20G
scsi1: PVE_SSD_01:167/vm-167-disk-1.qcow2,iothread=1,size=100G
scsihw: virtio-scsi-single
smbios1: uuid=168b7337-ef9d-4636-9951-f30da3b9de36
sockets: 1
vmgenid: 3500f9c7-f916-4e49-a8fd-a06401dc8c95



root@pve:~# qm config 123 --current
agent: 1
bios: ovmf
boot: order=scsi0;ide2;net0
cores: 2
cpu: EPYC
efidisk0: PVE_SATA_01:123/vm-123-disk-0.qcow2,efitype=4m,pre-enrolled-keys=1,size=528K
ide2: none,media=cdrom
memory: 8192
meta: creation-qemu=8.1.5,ctime=1720527743
name: mail
net0: virtio=BC:24:11:2D:XX:XX,bridge=vmbr1029,firewall=1
numa: 0
ostype: l26
scsi0: PVE_SATA_01:123/vm-123-disk-1.qcow2,iothread=1,size=60G
scsi1: PVE_SATA_01:123/vm-123-disk-2.qcow2,iothread=1,size=100G
scsihw: virtio-scsi-single
smbios1: uuid=381f9603-4c1a-42d7-b4bb-9ed80570cf1d
sockets: 1
vmgenid: da549816-d2cf-4f9d-aae2-6efa28d4eb05

and the Version-Information:

root@pve:~# pveversion -v
proxmox-ve: 8.2.0 (running kernel: 6.8.8-2-pve)
pve-manager: 8.2.4 (running version: 8.2.4/faa83925c9641325)
proxmox-kernel-helper: 8.1.0
pve-kernel-5.15: 7.4-13
proxmox-kernel-6.8: 6.8.8-2
proxmox-kernel-6.8.8-2-pve-signed: 6.8.8-2
proxmox-kernel-6.8.8-1-pve-signed: 6.8.8-1
proxmox-kernel-6.8.4-3-pve-signed: 6.8.4-3
pve-kernel-5.15.152-1-pve: 5.15.152-1
pve-kernel-5.15.149-1-pve: 5.15.149-1
pve-kernel-5.15.131-2-pve: 5.15.131-3
pve-kernel-5.15.131-1-pve: 5.15.131-2
pve-kernel-5.15.111-1-pve: 5.15.111-1
pve-kernel-5.15.102-1-pve: 5.15.102-1
amd64-microcode: 3.20230808.1.1~deb12u1
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.7
libpve-cluster-perl: 8.0.7
libpve-common-perl: 8.2.1
libpve-guest-common-perl: 5.1.3
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.9
libpve-storage-perl: 8.2.3
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.2.7-1
proxmox-backup-file-restore: 3.2.7-1
proxmox-firewall: 0.4.2
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.7
pve-container: 5.1.12
pve-docs: 8.2.2
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.1
pve-firewall: 5.0.7
pve-firmware: 3.12-1
pve-ha-manager: 4.0.5
pve-i18n: 3.2.2
pve-qemu-kvm: 9.0.0-6
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.4-pve1

Best regards
Simon

Thanks for the additional information, but unfortunately still no luck to reproduce this. Please share the proxmox-backup-manager version --verbose once again, since you did perform some upgrades in between.

Also, please do make sure that the latest kernel and version of proxmox-backup-server are running, best perform a full shutdown/start cycle of your PBS VM. While looking at your last backup server version output it seems that your installed and running version did not match. Make sure that is the case, so one is not chasing some strange edge case because of version mismatches.

Last but not least you could still try to create a new CephFS and create a completely fresh datastore on it to check if that is also affected by this.

 

[SiJux]

Hi Chris,

I've tested with a completly new created cephfs-pool and a new datastore in PBS: Same Issue.

After Upgrading to the latest PBS-Release including reboot it seems the issue has gone.
but rejoiced too soon... For some VMs the issue is still there.

Here the requested data:

root@pbs:~# proxmox-backup-manager version --verbose
proxmox-backup                    3.2.0        running kernel: 6.8.8-2-pve
proxmox-backup-server             3.2.7-1      running version: 3.2.7     
proxmox-kernel-helper             8.1.0                                   
proxmox-kernel-6.8                6.8.8-2                                 
proxmox-kernel-6.8.8-2-pve-signed 6.8.8-2                                 
proxmox-kernel-6.8.8-1-pve-signed 6.8.8-1                                 
proxmox-kernel-6.8.4-3-pve-signed 6.8.4-3                                 
proxmox-kernel-6.8.4-2-pve-signed 6.8.4-2                                 
ifupdown2                         3.2.0-1+pmx8                           
libjs-extjs                       7.0.0-4                                 
proxmox-backup-docs               3.2.7-1                                 
proxmox-backup-client             3.2.7-1                                 
proxmox-mail-forward              0.2.3                                   
proxmox-mini-journalreader        1.4.0                                   
proxmox-offline-mirror-helper     0.6.6                                   
proxmox-widget-toolkit            4.2.3                                   
pve-xtermjs                       5.3.0-3                                 
smartmontools                     7.3-pve1                               
zfsutils-linux                    2.2.4-pve1

One more interesting:
VM-Backup failes. Now I've deleted all Backups for the VM, rebooted the PBS. After that the backup for this VM works fine...

Best regards
Simon

 

[Chris]

Hi Chris,

I've tested with a completly new created cephfs-pool and a new datastore in PBS: Same Issue.

After Upgrading to the latest PBS-Release including reboot it seems the issue has gone.
but rejoiced too soon... For some VMs the issue is still there.

Here the requested data:

root@pbs:~# proxmox-backup-manager version --verbose
proxmox-backup                    3.2.0        running kernel: 6.8.8-2-pve
proxmox-backup-server             3.2.7-1      running version: 3.2.7   
proxmox-kernel-helper             8.1.0                                 
proxmox-kernel-6.8                6.8.8-2                               
proxmox-kernel-6.8.8-2-pve-signed 6.8.8-2                               
proxmox-kernel-6.8.8-1-pve-signed 6.8.8-1                               
proxmox-kernel-6.8.4-3-pve-signed 6.8.4-3                               
proxmox-kernel-6.8.4-2-pve-signed 6.8.4-2                               
ifupdown2                         3.2.0-1+pmx8                         
libjs-extjs                       7.0.0-4                               
proxmox-backup-docs               3.2.7-1                               
proxmox-backup-client             3.2.7-1                               
proxmox-mail-forward              0.2.3                                 
proxmox-mini-journalreader        1.4.0                                 
proxmox-offline-mirror-helper     0.6.6                                 
proxmox-widget-toolkit            4.2.3                                 
pve-xtermjs                       5.3.0-3                               
smartmontools                     7.3-pve1                             
zfsutils-linux                    2.2.4-pve1

One more interesting:
VM-Backup failes. Now I've deleted all Backups for the VM, rebooted the PBS. After that the backup for this VM works fine...

Best regards
Simon

Thanks again for the feedback!

Another option you could try is to mount CephFS via the ceph-fuse client instead of the in-kernel one. Please see https://docs.ceph.com/en/latest/cephfs/mount-using-fuse/

 

[SiJux]

Hi Chris,

just a quick update:

If I remove a failed backup in PBS, reboot the PBS, then create the backup.
If I use this workaround, all backups will be created and work fine.
For me it smells like a ugly bug.

Best regards
Simon

 

[Chris]

Hi Chris,

just a quick update:

If I remove a failed backup in PBS, reboot the PBS, then create the backup.
If I use this workaround, all backups will be created and work fine.
For me it smells like a ugly bug.

Best regards
Simon

Thanks for the continuous feedback!
DId you also try to use the ceph-fuse client? This would help nailing this down further.