Need advice on new proxmox cluster configuration

[Luq]

To whom it may concern:

I am kinda of new in proxmox, we have an existing proxmox cluster(3 hp servers, proxmox 3.4, no shared storage). We are getting 4 new hp servers and 1 netapp San.

1. does it make more sense to create a new proxmox cluster on the 4 new servers with proxmox 4.1 then migrate the existing vms from proxmox 3.4 cluster to new cluster then upgrade 3.4 to 4.1? or stay with proxmox 3.4 just add the 4 new servers to the existing cluster?

2. regarding the san storage:

2.1 we are thinking NFS, should we present one big lun to proxmox cluster then all proxmox servers mount the same lun or each server has its own lun?

2.2 does it make sense to migrate the whole vm to SAN storage or the OS stays on local disk of the servers just move the application partition to the lun? performance wise which would be better, server will be connected to san via 10g ethernet .

2.3 we have application cluster which requires shared storage, should i create a NFS share and add it to the proxmox server then mount it on all the virtual application servers? How do i mount the NFS disk on proxmox server from the virtual servers?

Thank you in advance!

 

[Q-wulf]

To whom it may concern:
[...]

good start

[...]We are getting 4 new hp servers and 1 netapp San.

1. does it make more sense to create a new proxmox cluster on the 4 new servers with proxmox 4.1 then migrate the existing vms from proxmox 3.4 cluster to new cluster then upgrade 3.4 to 4.1? or stay with proxmox 3.4 just add the 4 new servers to the existing cluster?

From an operations perspective it makes sense to setup a new cluster using the 4 new servers. verify that everything is working and familiarise yourself with the product. (as in test your use-cases and do some test-migrations). Once that Cluster is stable and you feel comfortable using it, you migrate the Vm's one by one during your maintenance windows. If a problem arises you keep it on the "old" cluster until you know how to fix said problem.

Then once everything is moved off the old cluster, i'd not even bother with upgrading it. Do a clean install and create a new cluster from it, or add the nodes to your new cluster.

[...]
2.2 does it make sense to migrate the whole vm to SAN storage or the OS stays on local disk of the servers just move the application partition to the lun? performance wise which would be better, server will be connected to san via 10g ethernet .

I am assuming you want to use the HA feature of proxmox 4.x. If so, afaik that requires that all VM-data is on the shared storage.


I'm not sure how many nics you have on your nodes, I assume you have at least 1x10G per node connected to the same network as the netapp from your description.

i'd segregate at a minimum the traffic of

• SAN
• Proxmox-backend (Corosync)
• proxmox frontend (Clients connecting to VM's)

into seperate nics/bonds, unless your comfortable doing QOS on the other end (switches), since it is the easiest way to do.

Q: What kind of network load do you expect ?

 

[Luq]

Hi Q-wulf

Thank you so much for your prompt reply!

Regarding migrate vm from one cluster to another, do i use vzdump?

One thing I heard that NFS can be slow when dealing with a lot of small files?

I will move the vm to the shared NFS disk, the same NFS disk should be mounted on all proxmox servers?

Should I create one big NFS disk or multiple NFS disks?

All servers have 2x1g 2x10g ethernet ports, the servers and netapp will be on the same c segment.

Our applications are internal facing and all web based, about 1,500 users. Traffic wise should not be a big concern.

Thank you again!

good start


From an operations perspective it makes sense to setup a new cluster using the 4 new servers. verify that everything is working and familiarise yourself with the product. (as in test your use-cases and do some test-migrations). Once that Cluster is stable and you feel comfortable using it, you migrate the Vm's one by one during your maintenance windows. If a problem arises you keep it on the "old" cluster until you know how to fix said problem.

Then once everything is moved off the old cluster, i'd not even bother with upgrading it. Do a clean install and create a new cluster from it, or add the nodes to your new cluster.


I am assuming you want to use the HA feature of proxmox 4.x. If so, afaik that requires that all VM-data is on the shared storage.


I'm not sure how many nics you have on your nodes, I assume you have at least 1x10G per node connected to the same network as the netapp from your description.

i'd segregate at a minimum the traffic of

• SAN
• Proxmox-backend (Corosync)
• proxmox frontend (Clients connecting to VM's)

into seperate nics/bonds, unless your comfortable doing QOS on the other end (switches), since it is the easiest way to do.

Q: What kind of network load do you expect ?

 

[Q-wulf]

One thing I heard that NFS can be slow when dealing with a lot of small files?

I will move the vm to the shared NFS disk, the same NFS disk should be mounted on all proxmox servers?

Should I create one big NFS disk or multiple NFS disks?

I do not do the NFS at our shop, so i can not personally answer that with confidence. We use Virtual NAS providing NFS that reside on Ceph storage, so my use-case there is probably completely different from yours. I'd test it (part of what i described above as "feeling comfortable") to make sure your netapp connected via 10G is able to handle your workloads.

1500 users sounds like a lot of bandwith (without knowing what your using the nodes for). If a single 10G Link for client bandwith i'd do the following segregation scheme:

• No Vlan: Clients --> 10G Nic -> vmbr0
• Vlan 1: Corosync + management --> 2x 1G Nic -> Bond1 -> vmbr1
• Vlan 2: SAN --> 10G Nic -> vmbr2

If you have VM's that communicate between each other, then i'd assign em 2 vNics.

• vnic1 --> client traffic -> vmbr0
• vnic2 -> Intervm traffic on vlanX -> vmbr2

I'd look into using openvswitch with proxmox, as it uses less resources compared to native linux and in my experience is easier to manage:
https://pve.proxmox.com/wiki/Open_vSwitch

ps: 1500 Clients on a 10G link on 4 nodes means on average 3,41 Mbit/s per client if spread over those 4 nodes.
pps: if a single 10G link is not enough for clients, i'd stick both 10G links into a bondX --> vmbrX and run San and Clients via that vmbrX

 

[Luq]

Q-wulf

Thank you so much for the in-depth knowledge. I will test it on my side and let you know.

Have a nice holiday.