HTTP Error 404 Not Found: Path '/fixed_index' not found.

[franciscopaniskaseker]

I am using Proxmox CE 6.2-14 and last PBS beta version. When I try to manually backup one VM, I get this:

()
INFO: starting new backup job: vzdump 110 --remove 0 --storage backup --node pve3 --mode snapshot
INFO: Starting Backup of VM 110 (qemu)
INFO: Backup started at 2020-11-01 21:01:33
INFO: status = running
INFO: VM Name: xxx.yyy.com
INFO: include disk 'scsi0' 'cephrdb:vm-110-disk-0' 20G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating pbs archive on storage 'backup'
ERROR: VM 110 qmp command 'backup' failed - backup register image failed: command error: HTTP Error 404 Not Found: Path '/fixed_index' not found.
ERROR: Backup of VM 110 failed - VM 110 qmp command 'backup' failed - backup register image failed: command error: HTTP Error 404 Not Found: Path '/fixed_index' not found.
INFO: Failed at 2020-11-01 21:01:33
INFO: Backup job finished with errors
TASK ERROR: job errors

 

[dcsapak]

can you post the output of 'pveversion -v' please ?

 

[franciscopaniskaseker]

proxmox-ve: 6.2-2 (running kernel: 5.4.34-1-pve)
pve-manager: 6.2-14 (running version: 6.2-14/2001b502)
pve-kernel-5.4: 6.2-7
pve-kernel-helper: 6.2-7
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph: 14.2.11-pve1
ceph-fuse: 14.2.11-pve1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-9
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 0.9.4-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.3-6
pve-cluster: 6.2-1
pve-container: 3.2-2
pve-docs: 6.2-6
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-3
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-17
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve2

 

[dcsapak]

ok thanks, whats the version of your proxmox-backup-server ?
(proxmox-backup-client version --repository <yourserverIp>:<anydatastore>)

 

[franciscopaniskaseker]

root@pve1:~# proxmox-backup-client version --repository x.y.z.w:backup
Password for "root@pam": ********************
client version: 0.9.4
server version: 0.9.4
root@pve1:~#

 

[dcsapak]

ok maybe your qemu process still runs with an old (incompatbile) version of the libproxmox-backup-qemu0 library, can you restart your qemu process (by e.g., powering down and starting the vm again)

 

[franciscopaniskaseker]

We did more one update on all packages and then restarted the entire host. Then I tried the backup again:

INFO: starting new backup job: vzdump 101 --node pve2 --storage backup --mode snapshot --remove 0
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2020-11-06 18:08:33
INFO: status = running
INFO: VM Name: Main
INFO: include disk 'scsi0' 'cephrdb:vm-101-disk-0' 251G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating pbs archive on storage 'backup'
ERROR: VM 101 qmp command 'backup' failed - backup register image failed: command error: HTTP Error 404 Not Found: Path '/fixed_index' not found.
ERROR: Backup of VM 101 failed - VM 101 qmp command 'backup' failed - backup register image failed: command error: HTTP Error 404 Not Found: Path '/fixed_index' not found.
INFO: Failed at 2020-11-06 18:08:48
INFO: Backup job finished with errors
TASK ERROR: job errors

 

[dcsapak]

mhmm.. thats weird and should not happen...

did you positively shutdown the vm and start it new? this is necessary to load the new library,
also it seems that you started the backup from a different node this time, did you upgrade on all nodes to the latest version?

please try to upgrade all nodes (also pbs), shutdown the vm and start it again (restart from inside the vm is not enough!) and try again
if this also fails, please post 'pveversion -v' from all pve nodes as well as the output from 'dpkg -l | grep -E "proxmox|pve|pbs"' from both pve and pbs

 

[franciscopaniskaseker]

Actually I updated all servers (proxmox and pbs) and restarted all the servers (which will restart virtual servers and libvirt.

I executed from different node, but all of them are using same packages version. I attached the commands results.

Is there any log that I can read to try understand what

HTTP Error 404 Not Found: Path '/fixed_index' not found.

means?

 

[franciscopaniskaseker]

Is there any other ISO version to try or a md5sum to check the ISO?
https://www.proxmox.com/en/downloads/category/iso-images-pbs

 

[dcsapak]

you have to update pbs manually with 'apt update && apt dist-upgrade'

just saw that you did...


the error message should only happen when connecting with an old client or to an old server.... i really do not understand where it comes form in your case...

 

[dcsapak]

also i just saw in your dpkg output that you have libproxmox-backup-qemu0 in an old version on some hosts... please check why that is (should be 0.7.1-1)

 

[franciscopaniskaseker]

Ok, I will check.