Needed: best practices when updating server that has shared storage in cluster

P

psalet

Member
Jul 17, 2020
15
2
23
66
Hi All,

I am looking for the "best practice" for upgrading a server in a cluster (22 systems) that is sharing it's iscsi mounts to the rest of the cluster.

I am about to take on the task of upgrading the memory on one of my main servers that is the focal point for the connection to the iscsi storage. I am really concerned that it would take down all of the VMs in the cluster as the storage is being shared from that server.

Give me some ideas.
Thanks
--paul
 
If all your storage is in a single iSCSI box, you have a single point of failure, which kinda defeats the whole purpose of a cluster.

You will need to plan for downtime, stop everything on PVE, log out iSCSI from each PVE (not sure if needed, but I would do it), stop the iSCSI storage, update memory, start iSCSI storage, reconnect iSCSI on all PVE, start the VMs. Also, I would run memtest on those sticks before using them on the production system, just in case (although they might break during installation on the production server).

If you have some other storage that you may use, you could do live storage migration so VM's can run from it while you upgrade that ram.
 
I went with the one host sharing the iscsi was to make the migrating be only moving the /etc/pve/qemu-server/conf file to the destination server. So then my question would be "If I mount the iscsi drive on each server this will fix this issue." I just need to keep the iscsi naming the same name on all servers.
I just want to make sure when migrating VMs from one server to another that I am not copying any data files just the conf file.
Hope that made sense.
--paul,
 
Sorry, I don't fully understand how is your cluster configured, which storage(s) are you using and what are your trying to accomplish. I would need details to give you accurate recomendations.
 
Hi VictorSTS

Shown below is an example of one of my Proxmox servers. The connection to storage is thru (2) 10Gb fiber connections bonded and port channeled together to our storage switches. My goal is HA at the very end of the configuration. I agree with you that I should not had done a single connection thru a proxmox server to storage.

Cluster:
PVE 8.1.3, 22 Hosts
Storage:
Remote:
Pure Storage mounted per iscsi using LVM on one PVE host shared to the rest of the servers.
Local Storage per host:
local: 220Gb
local-lvm: 4Tb
Network:
1: 10Gb fiber General purpose VLAN
2: (2) 10Gb fiber bonded, Storage Vlan
3: 1Gb Management

Example host:
1738775242775.png
Example Networking
1738775417370.png
The following is my proposed steps to fix this issue.
1. Migrate all VM's off Proxmox host that I plan to work on.
2. Go to the proxmox host that is sharing storage and stop sharing of the remote storage to the above host.
3. Connect remote storage directly to the above host.
4. Verify storage is working correctly.
5. Migrate VM's back to thi host

Please comment if there is something I missed or I should do make the cluster better
Thanks for all the help, It really is appreciated
--paul
 
Sorry, still don't get the full picture... ;)

psalet said:
Pure Storage mounted per iscsi using LVM on one PVE host shared to the rest of the servers.
Click to expand...
  • Ok, there's an external storage that you connect via iSCSI to... one PVE of the 22 in your cluster? another PVE that does not belong to the cluster?
  • You mention it is "shared to the rest of the servers": who "shares" it? The Purestorage? A single one PVE of the cluster? Yet another PVE that does not belong to the cluster? If any of the later two, how are you sharing that Purestorage Lun to cluster's nodes?
  • The snapshot shows a "TrueNAS" storage, is it related to any of this?
What you usually do to use SAN with PVE is connect the LUN(s) to every node with iSCSI (add here multipath/bonding if required for availability), then create a thick LVM (not lvm-thin, that is not supported with a shared LUN), and all PVE will see the same LVM shared from the SAN directly. There's no dependency from any other PVE to reach the storage.

Your setup looks pretty exotic to me, unless I'm missing something.
 
Sorry for the confusion.
The cluster contains 22 proxmox servers. I have mounted the Pure storage on Proxmox server A. It is a direct connection to the filer. The storage information is then shared thru Proxmox server A to the rest of the cluster (the other 21 servers). The Pure storage is allowing access as it has the required IQN info about rest of the servers in the cluster and is able to mount the storage if necessary. ( it is necessary now lol).

That is why it is so confusing.

We are using thick LVM so that should not be an issue.
Truenas is something I was trying out. Not part of the problem.

History on the build was I tested each server before joining the cluster to see if it could reach the mount the Pure storage (Which it did). I just should have left it that way.

Thanks for all the help
 
Don't reply, my thinking was all messed up. I added the storage to my cluster. WTF was I thinking.
 
  • Like
Reactions: waltar
You must log in or register to reply here.
Top Bottom