It doesn't detect 3-node cluster storage well.

[guilloking]

Hello dear community.

Could you help me with this case, I have a storage unit contracted in the cloud and I mounted it in a proxmox cluster, it contains 5 nodes, of which only 3 correctly detect the storage and the last 2 do not.Could you help me by indicating how to find the solution, I search and can't find how to solve it and have it correctly detect the size.

node 1, 2

node 3


As you can see, despite being the same directory, it does not detect the correct size.

I have proxmox 7.

thx.

 

[bbgeek17]

The most simple explanation is that they do not, in fact, see the same storage. Perhaps the "bad" 2 do not have storage mounted.
You omitted some important information:
a) what is the storage
b) how is it presented to PVE
c) configuration of the storage "cat /etc/pve/storage.cfg"
d) state of the running config on each node, or at least on one "good" and one "bad" one. "df;mount"
e) state of PVE storage daemon from each node "pvesm status"
f) did you examine the logs "journalctl -n 500", and if you did - are there any red flags that jumped out at you?

These are just the first steps.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[guilloking]

Hi dear bbgeek17, thnx for the response.

a) what is the storage
It is a cloud storage contracted by Hetzner.
b) how is it presented to PVE
How directory, configured in the datacenter options storage.

c) configuration of the storage "cat /etc/pve/storage.cfg"
dir: BackupStorage
path /mnt/storagebox
content backup
prune-backups keep-all=1
shared 0

d) state of the running config on each node, or at least on one "good" and one "bad" one. "df;mount"
assembly procedure carried out with cifs "imount -t cifs" in fstab

e) state of PVE storage daemon from each node "pvesm status"
Name Type Status Total Used Available %
Backup cephfs active 1418694656 386711552 1031983104 27.26%
BackupStorage dir active 21474836480 5971021390 15503815090 27.80%
MV rbd active 8059799136 7027814816 1031984320 87.20%
Sources cephfs active 1055309824 23326720 1031983104 2.21%
local dir active 256920004 5078664 238717756 1.98%

f) did you examine the logs "journalctl -n 500", and if you did - are there any red flags that jumped out at you?
I don't see an associated error, the system has been like this for a while, identifying an incorrect storage despite it being the same directory within the datacenter


I don't know if a solution would be to unmount and re-execute mount -a, to re-upload the directory and see if all the nodes take the correct size.

 

[bbgeek17]

d) state of the running config on each node, or at least on one "good" and one "bad" one. "df;mount"
assembly procedure carried out with cifs "imount -t cifs" in fstab

understood about fstab, but why did you skip providing requested data? You are bypassing PVE CIFS storage plugin and mounting the CIFS/SMB manually.

At the very least you did not specify "is-mountpoint" attribute in directory and so you risk writing data to root if your mount fails.
Does the space shown by "bad" nodes match your local root disk information by chance?

I don't see an associated error, the system has been like this for a while, identifying an incorrect storage despite it being the same directory within the datacenter

The directory may be the same , but the mounts behind it are probably not.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[guilloking]

Hi bbgeek17

mmm more data.

d) state of the running config on each node, or at least on one "good" and one "bad" one. "df;mount"
assembly procedure carried out with cifs "imount -t cifs" in fstab

yes, its a storage mount manually.


mount fstab
imount -t cifs -o username=xxx //u334.your-store/backup /mnt/storageboxA
//u334.your-store/backup /mnt/storagebox cifs username=xxx,password=xxx 0 0

df -h
//u334.your-store/backup 20T 5.6T 15T 28% /mnt/storagebox

The mount point information is held by node 1.
The mount point does not appear on the rest of the nodes, since it was uploaded in a directory so that it appears on all nodes in the cluster.

Sorry, more information would you need?
Thx for reponse.

 

[sb-jw]

Is it possible that you have already reached the maximum of 10 simultaneous connections to the StorageBox and the local path may now be displayed?

 

[bbgeek17]

I am not quite sure what you are showing above. These are the basic rules :
a) The share has to be mounted by every node in the cluster
b) That means it has to be in every nodes fstab in your case
c) it has to show up in "mount" output on every node
d) the "df" output on every node must be the same
e) make sure you inform PVE that the "directory" storage you told it to use is in fact "mountpoint" (man pvesm)

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[guilloking]

The size does not match the local unit, I don't know what the reason is, it shouldn't cause any impediment.

It is assumed that in a cluster when mounting a directory this same directory applies to all the nodes, so I do not have to mount the same unit 3 times, in addition that configuration is taken in the first 2 but in the last one the detail appears that It detects but the correct size does not come out.

 

[guilloking]

One question, if I remove the directory and remount it, don't I have to restart the nodes? Does it take the configuration and replicate it to the rest of the nodes?

To see if it replicates the correct size on all nodes or the same error continues.

Can it be done without creating unavailability?

 

[bbgeek17]

It is assumed that in a cluster when mounting a directory this same directory applies to all the nodes

You are using a keyword here "mounting" that is not involved in the dynamic of the "directory" storage type.

All you are doing is telling PVE to use directory. As far as PVE is concerned its a local directory. It does not know anything about "mount", what is behind the directory, if anything.

You made a choice to manually point the directory (from OS perspective) to an SMB share. It is your responsibility to ensure that the directory is properly mounted everywhere. This must be done BEFORE applications start using this directory. The application in this case is PVE.

There is no replication by PVE, all it does is sends a basic request to _LOCAL_ OS - get me space behind this directory.

You shouldnt need to restart nodes to properly mount the folder.

As another member noted, its possible you reached some sort of Cloud Provider limit. I am not familiar with it.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[guilloking]

Yes, assembly because in pm1 the amount is made as indicated in the previous answer.

So you say that despite being a cluster, it must be in all the fstabs of the nodes, and in turn mounted, I ask because I only see the mount point on the first node and from that point in "datacenter-storage" I create a directory with the mount point created and this is replicated to the rest of the nodes, being successful in the second node but in the third it is seen but does not show the correct size.

I don't know if there is a limit in Hetzner services regarding cloud storage.

I will wait non-productive hours to unmount the directory and remount it.

 

[bbgeek17]

So you say that despite being a cluster, it must be in all the fstabs of the nodes, and in turn mounted

Proxmox cluster does not replicate local, unrelated configuration between the nodes.
If you want PVE to be responsible for this storage, you need to replace Directory storage type (in PVE) with CIFS storage type (in PVE). In that case it will be PVE responsibility to connect to the share and ensure its availability. I dont know whether PVE is capable of accessing your CSP share.

I ask because I only see the mount point on the first node

We dont know how you arrived to your current state. Its not worth troubleshooting, to be honest. Likely, you ran commands on purpose or accidentally that mounted the share on "working" nodes.
If you only have one node with correct fstab, the next time you reboot you will have 1 node reporting expected state, and 4 nodes reporting wrong state.

Keep in mind that you likely have some sort of data stored in that directory on all nodes. On "bad" nodes that data is physically on root disk of your node. When you mount the SMB share that data will be hidden from you, and you will be unable to delete it until you unmount the share.
I recommend doing any clean up _before_ you start changing the state.

Good luck.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[guilloking]

This feedback that I have been receiving is very helpful, a question, if I mount the Hetzner cloud storage via cifs/smb it would avoid all this directory inconvenience and so on, the unit would be replicated on all the nodes, right.

I plan to mount the same unit with the credentials under cifs and see how it behaves.

I will inform you.

Regarding the commands executed and so on, I don't know, it is an infrastructure that was given to me and I detected this error, which is new to me.

Thank you,

 

[bbgeek17]

if I mount the Hetzner cloud storage via cifs/smb it would avoid all this directory

If you go to Proxmox GUI (or use CLI) and create storage with type of CIFS/SMB - yes, there will be no "directory" type storage involved. Because you will make this configuration via PVE, it will be part of PVE configuration and so this configuration will be distributed across all nodes in the cluster.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[guilloking]

Good morning everyone.

The solution chosen was to mount the unit in each of the nodes.

//u334.your-store/backup /mnt/storagebox cifs username=xxx,password=xxx 0 0

It now recognizes the storage as correct.

Add it to each of the fstabs so that it does not generate that unavailability again.

Thanks to everyone for your support.