[SOLVED] S3 datastore error

R

rahman

Renowned Member
Nov 1, 2010
92
4
73
Last sync job from local datastore to s3 backed datastore failed with a strange error:

Code: 
detected wrong types in mmaped files: incorrect magic number for request counters detected

I see the same error in all PBS GUI sections about this datastore. How can I check what is the problem?

Code: 
root@pbs1:~# proxmox-backup-manager version --verbose
proxmox-backup                      4.0.0         running kernel: 6.17.13-3-pve
proxmox-backup-server               4.1.7-2       running version: 4.1.7       
proxmox-kernel-helper               9.0.4                                     
proxmox-kernel-6.17                 6.17.13-3                                 
proxmox-kernel-6.17.13-3-pve-signed 6.17.13-3                                 
proxmox-kernel-6.17.13-2-pve-signed 6.17.13-2                                 
proxmox-kernel-6.17.2-1-pve-signed  6.17.2-1                                   
ifupdown2                           3.3.0-1+pmx12                             
libjs-extjs                         7.0.0-5                                   
proxmox-backup-docs                 4.1.7-2                                   
proxmox-backup-client               4.1.7-2                                   
proxmox-mail-forward                1.0.2                                     
proxmox-mini-journalreader          1.6                                       
proxmox-offline-mirror-helper       0.7.3                                     
proxmox-widget-toolkit              5.1.9                                     
pve-xtermjs                         5.5.0-3                                   
smartmontools                       7.4-pve1                                   
zfsutils-linux                      2.4.1-pve1
 
Please post the output of hexdump /var/lib/proxmox-backup/s3-statistics/<you-s3-counter>.shmem | head -n 1, where <your-s3-counter> is composed by endpoint id, bucket name and datastore name.
 
Chris said:
Please post the output of hexdump /var/lib/proxmox-backup/s3-statistics/<you-s3-counter>.shmem | head -n 1, where <your-s3-counter> is composed by endpoint id, bucket name and datastore name.
Click to expand...

Here it is:

Code: 
root@pbs1:~# hexdump /var/lib/proxmox-backup/s3-statistics/ulakfkm-default-bucket-ulakfkm.shmem | head -n 1
0000000 0000 0000 0000 0000 0000 0000 0000 0000
 
Okay, so for some reason this does not contain the correct header, the expected output would be 0000000 6ee0 fc58 4d1a 05b4 0000 0000 0000 0000. Please set the maintenance mode offline on the s3 datastore, remove this file and bring the store back online. Then check the output again.
 
Chris said:
Okay, so for some reason this does not contain the correct header, the expected output would be 0000000 6ee0 fc58 4d1a 05b4 0000 0000 0000 0000. Please set the maintenance mode offline on the s3 datastore, remove this file and bring the store back online. Then check the output again.
Click to expand...

It seems this fixed the problem, thanks :) I could not find any meaningful result for this error on Google search. Everything seems to working now.

Code: 
root@pbs1:~# hexdump /var/lib/proxmox-backup/s3-statistics/ulakfkm-default-bucket-ulakfkm.shmem | head -n 1
0000000 6ee0 fc58 4d1a 05b4 0000 0000 0000 0000
 
The shared request counters have been introduced very recently only. Could you however share the output of apt-cache policy proxmox-backup-server. What version was running before upgrading to 4.1.7-2?
 
Chris said:
The shared request counters have been introduced very recently only. Could you however share the output of apt-cache policy proxmox-backup-server. What version was running before upgrading to 4.1.7-2?
Click to expand...

Sure, here it is:

Code: 
root@pbs1:~# apt-cache policy proxmox-backup-server
proxmox-backup-server:
  Installed: 4.1.7-2
  Candidate: 4.1.7-2
  Version table:
 *** 4.1.7-2 500
        500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
        100 /var/lib/dpkg/status
     4.1.6-1 500
        500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
     4.1.5-2 500
        500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
     4.1.5-1 500
        500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
     4.1.4-1 500
        500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
     4.1.2-1 500
        500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
     4.1.1-1 500
        500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
     4.1.0-1 500
        500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
     4.0.22-1 500
        500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
     4.0.21-1 500
        500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
     4.0.20-1 500
        500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
     4.0.19-1 500
        500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
     4.0.18-1 500
        500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
     4.0.17-1 500
        500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
     4.0.16-1 500
        500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
     4.0.15-1 500
        500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
     4.0.14-1 500
        500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
     4.0.13-1 500
        500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
     4.0.12-1 500
        500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
     4.0.11-4 500
        500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
     4.0.11-2 500
        500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
     4.0.10-1 500
        500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
     4.0.9-1 500
        500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
     4.0.8-1 500
        500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
     4.0.7-1 500
        500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
     4.0.6-2 500
        500 http://download.proxmox.com/debian/pbs trixie/pbs-no-subscription amd64 Packages
 
I just wanted to post that I encountered the same issue: same symptoms, same diagnosis, same fix. Thanks for posting this here!
 
You must log in or register to reply here.
Top Bottom
Back