[SOLVED] New nodes timeout on storage tabs

[Eliott G.]

Hi there !

We just installed new nodes in our cluster, configured SSL, all good.

But on the new nodes when I try to access infos from the plugged storages it displays a spinner for like a minute and then it says "Communication failure (0)".
It only happens on the storages (even local), I can access panels with server's Summary, Disk, etc.

I can't find error logs anywhere and the AJAX requests hangs forever, never returning anything.

Previous nodes where in version pve-manager/4.4-13/7ea56165
new ones are in pve-manager/4.4-22/2728f613

I'm kind of desperate for any logs I could look for or any idea.

Thanks in advance !

 

[udo]

Hi,
does the new node have different storage than the other cluster-nodes (like zfs + lvm)?

How look the output of following command on the new (and an old one) node?

pvesm status

Udo

 

[Eliott G.]

Hi !
Here is what I got on the old nodes

pvesm status

CEPHOVH01 rbd 1 5754024888 996693940 4757330948 17.82%
CEPHRBX01 rbd 1 70291976760 7995822556 62296154204 11.88%
NASISOGRA nfs 1 104857600 5296128 99561472 5.55%
NASVMRBX01 nfs 1 2197583872 1128894464 1068689408 51.87%
NASVMRBX02 nfs 1 2256390144 179211264 2077178880 8.44%
local dir 1 436037240 225019896 188861264 54.87%

But on the new nodes the command never respond.
Old nodes are ext3 and new nodes are ext4

Thanks !

 

[Eliott G.]

Found out I can list objects from local and NAS

pvesm list NASVMRBX02

NASVMRBX02:2000/vm-2000-disk-1.qcow2 qcow2 214748364800 2000
NASVMRBX02:20001/vm-20001-disk-1.qcow2 qcow2 214748364800 20001
NASVMRBX02:30009/vm-30009-disk-1.qcow2 qcow2 214748364800 30009
NASVMRBX02:3011/vm-3011-disk-1.qcow2 qcow2 214748364800 3011
NASVMRBX02:3012/vm-3012-disk-1.qcow2 qcow2 214748364800 3012
NASVMRBX02:50301/vm-50301-disk-1.qcow2 qcow2 214748364800 50301
NASVMRBX02:5201/vm-5201-disk-1.qcow2 qcow2 214748364800 5201
NASVMRBX02:5212/vm-5212-disk-1.qcow2 qcow2 214748364800 5212

but not from the rbd storages, this might cause the timeout ?

 

[Eliott G.]

Awww... In the web interface I just tried to disable the Ceph storages for thoses nodes only and I've got no more timeouts.
Left to figure out why I can't connect to my rbd storage :/

 

[Eliott G.]

Okay ! That was one Ceph storage we had at OVH that was not correctly added (OVH's APIs all buggy duh).
Just did it again and the storage is available, hence every other device is available !

Thats nevertheless that every storage is considered not available when only one can't respond ?
Maybe that was fixed in Proxmox 5 ?

Thanks for your time !