Newly joined node not working in cluster "unable to create directory '/etc/pve/nodes' - Permission denied"

[guff666]

I've just brought up a new node with ZFS as its store. I updated it and then added it to an existing cluster. I did see an error about "unable to create directory /etc/pve/nodes" but it was fleeting and I didn't get to capture it properly.

The other nodes are all OK, and all nodes report that the cluster is working. BUT, clicking on the node from a brwoser connected to another node creates a connection error, and inspection of the node shows that /etc/pve is only partially populated:

root@pve8:/etc/pve# ls -l
total 1
-rw-r----- 1 root www-data 743 Mar  5 11:37 corosync.conf
lrwxr-xr-x 1 root www-data   0 Jan  1  1970 local -> nodes/pve8
lrwxr-xr-x 1 root www-data   0 Jan  1  1970 lxc -> nodes/pve8/lxc
drwxr-xr-x 2 root www-data   0 Mar  5 12:02 nodes
lrwxr-xr-x 1 root www-data   0 Jan  1  1970 openvz -> nodes/pve8/openvz
drwx------ 2 root www-data   0 Mar  5 11:37 priv
lrwxr-xr-x 1 root www-data   0 Jan  1  1970 qemu-server -> nodes/pve8/qemu-server
drwxr-xr-x 2 root www-data   0 Mar  5 11:44 virtual-guest

Inspection of the journal shows the error at 11:39:55 pvedaemon[4642]: unable to create directory '/etc/pve/nodes' - Permission denied
It then degenerates from there

Extract from journal can be found at https://pastebin.com/PgYPtE7L
Any ideas?

Gareth

 

[guff666]

Update:
Now (15:55) I can now only open the GUI on one node. Login fails on the others.
From the node I can log into, all other nodes are showing with a '?' If I actually click on them, after a long delay, they do show the content, but I occasionally get connection errors.

There's no change to a pvecm status on any of the nodes