[SOLVED] Can't access PBS from one node

[thorro]

I have 2 PVE (pve1, pve2) node cluster and another server with PBS installed, all latest versions.

I defined store1 on PBS node and configured it on pve2, I can access it fine from pve2. But on node pve1, I always get:

pvesm list store1
store1: error fetching datastores - 401 Unauthorized

I've tried running identical commands to set up store1 on both nodes, no luck.

I checked logs on the PBS and for the non-working node I get:

2020-12-05T15:27:27+01:00: authentication failure; rhost=::ffff:1.1.1.1 msg=invalid token name
::ffff:1.1.1.1 - - [05/12/2020:15:27:39 +0100] "GET /api2/json/admin/datastore" 401 42 libwww-perl/6.36

No api token is visible in logs, for working node it is. Store1 shows on pve1, but with a question mark.

pveversion -v
proxmox-ve: 6.3-1 (running kernel: 5.4.78-1-pve)
pve-manager: 6.3-2 (running version: 6.3-2/22f57405)
pve-kernel-5.4: 6.3-2
pve-kernel-helper: 6.3-2
pve-kernel-5.4.78-1-pve: 5.4.78-1
pve-kernel-5.4.41-1-pve: 5.4.41-1
pve-kernel-4.15: 5.4-18
pve-kernel-4.15.18-29-pve: 4.15.18-57
pve-kernel-4.15.18-27-pve: 4.15.18-55
pve-kernel-4.15.18-13-pve: 4.15.18-37
pve-kernel-4.13.13-6-pve: 4.13.13-42
pve-kernel-4.13.13-5-pve: 4.13.13-38
pve-kernel-4.13.13-2-pve: 4.13.13-33
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libproxmox-backup-qemu0: 1.0.2-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-1
libpve-common-perl: 6.3-1
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.3-2
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
openvswitch-switch: 2.12.0-1
proxmox-backup-client: 1.0.5-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-3
pve-cluster: 6.2-1
pve-container: 3.3-1
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-7
pve-xtermjs: 4.7.0-3
pve-zsync: 2.0-4
qemu-server: 6.3-1
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.5-pve1

 

[che]

Hi, how did you setup the PBS storage? Did you follow the docs? https://pbs.proxmox.com/docs/pve-integration.html
Do you have the correct password for accessing the PBS instance on pve1? This should be located in a file on /etc/pve/priv/storage/ on both nodes and synced via the proxmox cluster filesystem pmxcfs.

 

[thorro]

Hey,

yes, all followed and it works perfectly from pve2, there's store.pw on both nodes (as /etc/pve is shared between both). That's why I've no clue why it's not working from pve1.

 

[thorro]

OK, problem solved, but there might be a bug in newest pvesm. Checking the exact version of pvesm's package, there was a minor difference between both nodes, version of package libpve-storage-perl was 6.3-2 on non-working node.

By downgrading it to 6.3-1, problem is solved for now and pvesm list ... works on both nodes now.

apt install libpve-storage-perl=6.3-1

 

[che]

OK, problem solved, but there might be a bug in newest pvesm. Checking the exact version of pvesm's package, there was a minor difference between both nodes, version of package libpve-storage-perl was 6.3-2 on non-working node.

By downgrading it to 6.3-1, problem is solved for now and pvesm list ... works on both nodes now.

apt install libpve-storage-perl=6.3-1

There is indeed an issue, but according to the devs it has already been fixed, just waiting in the testing repo before being pushed out https://forum.proxmox.com/threads/pbs-datastore-in-var-lib-vz.80203/post-354883