Hi everyone,
I'm working on a project and one of the needs I've identified is the need to purchase new servers to replace old ones that are in use in a Proxmox virtualization cluster. There is also 1 storage unit in operation in this cluster, but it is new and it will probably not be necessary to purchase a new one.
PROJECT REQUIREMENTS:
- The solution consists of having 3 physical servers running in a Proxmox virtualization cluster;
- The servers must be sized so that 1 server alone supports all the demand required by the Cluster's virtual machines. Therefore, it is expected that if two servers become unavailable, the other will keep the cluster's virtual machines available;
- Each server must have 2 hard drives configured in RAID 1, so that if 1 fails, the other takes over;
- The live migration feature must work, allowing online migration of virtual machines if necessary.
CURRENT DATA
- Total vCPUs of the Cluster (sum of the vCPUs of all hypervisors): 216 vCPUs
- Total vCPUs allocated to the Cluster VMs = 211 vCPUs
- Total CPU Clock of the Cluster (total physical cores x base frequency of the CPUs) = 262.80 GHz
- Total CPU Clock effectively used = 83.21 GHz
- Total RAM memory of the Cluster (sum of the RAM memory of all hypervisors): 608 GB
- Total RAM memory allocated to the Cluster VMs = 401 GB
ISCSI STORAGE with the following configurations:
- Windows Server Operating System;
- 10 8TB HDD disks configured in RAID 10, thus delivering a usable capacity of 40TB.
SERVER CPU
1 - NUMBER OF VCPUS: To meet the CPU demand, I am thinking of suggesting the acquisition of physical servers with the following CPU configuration: 2 CPUs per server Intel® Xeon® Silver 4416+ 2G, 20C/40T, 16GT/s, 37.5M Cache, Turbo, HT (165W) DDR5-4000, 80 vCPUs. This way, the CPUs would deliver a total of 80 GHz per server (2 CPUs x 2GHz per physical core x 20 physical cores per CPU), supporting the entire processing load of the cluster with just 1 server.
Regarding the number of vCPUs, since this CPU has superior performance compared to those currently in use in the environment, it will be necessary to allocate a smaller number of vCPUs to the VMs. However, even so, the maximum number of vCPUs that could be allocated to the VMs would be 35, i would leave 5 for Proxmox OS and its services. However, we have 76 VMs in the environment and we need to have room for growth. As far as I know, the minimum number of vCPUs that can be allocated per VM with Proxmox is 1. Therefore, even if I allocated 1 vCPU per VM, there would still be a lack of vCPUs to be allocated to 41 VMs.
I was reading about CPUunits and CPUlimits, but I still haven't been able to find a solution to this problem, since according to the information in the Proxmox documentation, "It is safe to have a total number of vCPUs allocated to VMs greater than the number of CPU cores on the machine, however, Proxmox VE will prevent you from starting VMs with more virtual CPU cores than are physically available, as this will only reduce performance due to the cost of context switches."
Therefore, I would like to know if you have any suggestions on how to solve this issue of the number of vCPUs.
2 - LIVE MIGRATION: I have been studying the subject and according to the information I have gathered, mainly available on the Proxmox website, for this feature to work on the virtual machines in a cluster, the hosts must have CPUs from the same manufacturer (Intel, AMD) and these CPUs must support the same set of flags, otherwise, when a VM is migrated from one host to another and if the CPUs do not support the same flags, the VM will stop working, which is why there are QEMU models that make it possible to establish a common virtual CPU model for all hosts. The problem is that with the QEMU models, the CPU performance will be leveled by the lowest-performance one, correct?
I'm thinking of taking advantage of 1 physical server that we have in the environment and buying 2 more. The server we already have has the following CPU configuration:
2x Intel® Xeon® Silver 4214 Processor 16.5M Cache, 2.20 GHz
Instruction Set Extensions: Intel® SSE4.2, Intel® AVX, Intel® AVX2, Intel® AVX-512. # of AVX-512 FMA Units 1
And the CPU that I am specifying to insert in the new server to be acquired has the following configuration:
2x Intel® Xeon® Silver 4416+ Processor 37.5M cache, 2.00 GHz.
Instruction Set Extensions: Intel® AMX, Intel® SSE4.2, Intel® AVX, Intel® AVX2, Intel® AVX-512
No. of AVX-512 FMA Units: 2
So I believe that for the live migration feature to work in this case I would have to upgrade the CPU of the server we have so that it supports the same flags as the Intel Xeon Silver 4416+ server or I would have to enable the QEMU model on both servers, this way the performance of the Intel Xeon Silver 4416+ CPU will be leveled by the lower performing one, in this case the Intel Xeon 4214 CPU, correct?
Another solution I was thinking about when reading the documentation on the Proxmox website, but I don't know if it's possible to implement in practice, would be to configure the cluster servers as hosts and set the default QEMU CPU model to x86-64-v4, so that if there is this CPU flags problem, Proxmox would adopt this default, so that the VM would not be interrupted during the live migration. I believe maybe it might be possible to do this, as you can see what is written in the documentation on the Proxmox website:
"If you want to do a live migration of VMs between different hosts, your VM might end up on a new system with a different CPU type or a different microcode version. If the CPU flags passed to the guest are missing, the QEMU process will stop. To remedy this QEMU has also its own virtual CPU types, that Proxmox VE uses by default."
https://www.intel.com/content/www/u...ssor-16-5m-cache-2-20-ghz/specifications.html
https://www.intel.com.br/content/ww...ssor-37-5m-cache-2-00-ghz/specifications.html
Thank you.
I'm working on a project and one of the needs I've identified is the need to purchase new servers to replace old ones that are in use in a Proxmox virtualization cluster. There is also 1 storage unit in operation in this cluster, but it is new and it will probably not be necessary to purchase a new one.
PROJECT REQUIREMENTS:
- The solution consists of having 3 physical servers running in a Proxmox virtualization cluster;
- The servers must be sized so that 1 server alone supports all the demand required by the Cluster's virtual machines. Therefore, it is expected that if two servers become unavailable, the other will keep the cluster's virtual machines available;
- Each server must have 2 hard drives configured in RAID 1, so that if 1 fails, the other takes over;
- The live migration feature must work, allowing online migration of virtual machines if necessary.
CURRENT DATA
- Total vCPUs of the Cluster (sum of the vCPUs of all hypervisors): 216 vCPUs
- Total vCPUs allocated to the Cluster VMs = 211 vCPUs
- Total CPU Clock of the Cluster (total physical cores x base frequency of the CPUs) = 262.80 GHz
- Total CPU Clock effectively used = 83.21 GHz
- Total RAM memory of the Cluster (sum of the RAM memory of all hypervisors): 608 GB
- Total RAM memory allocated to the Cluster VMs = 401 GB
ISCSI STORAGE with the following configurations:
- Windows Server Operating System;
- 10 8TB HDD disks configured in RAID 10, thus delivering a usable capacity of 40TB.
SERVER CPU
1 - NUMBER OF VCPUS: To meet the CPU demand, I am thinking of suggesting the acquisition of physical servers with the following CPU configuration: 2 CPUs per server Intel® Xeon® Silver 4416+ 2G, 20C/40T, 16GT/s, 37.5M Cache, Turbo, HT (165W) DDR5-4000, 80 vCPUs. This way, the CPUs would deliver a total of 80 GHz per server (2 CPUs x 2GHz per physical core x 20 physical cores per CPU), supporting the entire processing load of the cluster with just 1 server.
Regarding the number of vCPUs, since this CPU has superior performance compared to those currently in use in the environment, it will be necessary to allocate a smaller number of vCPUs to the VMs. However, even so, the maximum number of vCPUs that could be allocated to the VMs would be 35, i would leave 5 for Proxmox OS and its services. However, we have 76 VMs in the environment and we need to have room for growth. As far as I know, the minimum number of vCPUs that can be allocated per VM with Proxmox is 1. Therefore, even if I allocated 1 vCPU per VM, there would still be a lack of vCPUs to be allocated to 41 VMs.
I was reading about CPUunits and CPUlimits, but I still haven't been able to find a solution to this problem, since according to the information in the Proxmox documentation, "It is safe to have a total number of vCPUs allocated to VMs greater than the number of CPU cores on the machine, however, Proxmox VE will prevent you from starting VMs with more virtual CPU cores than are physically available, as this will only reduce performance due to the cost of context switches."
Therefore, I would like to know if you have any suggestions on how to solve this issue of the number of vCPUs.
2 - LIVE MIGRATION: I have been studying the subject and according to the information I have gathered, mainly available on the Proxmox website, for this feature to work on the virtual machines in a cluster, the hosts must have CPUs from the same manufacturer (Intel, AMD) and these CPUs must support the same set of flags, otherwise, when a VM is migrated from one host to another and if the CPUs do not support the same flags, the VM will stop working, which is why there are QEMU models that make it possible to establish a common virtual CPU model for all hosts. The problem is that with the QEMU models, the CPU performance will be leveled by the lowest-performance one, correct?
I'm thinking of taking advantage of 1 physical server that we have in the environment and buying 2 more. The server we already have has the following CPU configuration:
2x Intel® Xeon® Silver 4214 Processor 16.5M Cache, 2.20 GHz
Instruction Set Extensions: Intel® SSE4.2, Intel® AVX, Intel® AVX2, Intel® AVX-512. # of AVX-512 FMA Units 1
And the CPU that I am specifying to insert in the new server to be acquired has the following configuration:
2x Intel® Xeon® Silver 4416+ Processor 37.5M cache, 2.00 GHz.
Instruction Set Extensions: Intel® AMX, Intel® SSE4.2, Intel® AVX, Intel® AVX2, Intel® AVX-512
No. of AVX-512 FMA Units: 2
So I believe that for the live migration feature to work in this case I would have to upgrade the CPU of the server we have so that it supports the same flags as the Intel Xeon Silver 4416+ server or I would have to enable the QEMU model on both servers, this way the performance of the Intel Xeon Silver 4416+ CPU will be leveled by the lower performing one, in this case the Intel Xeon 4214 CPU, correct?
Another solution I was thinking about when reading the documentation on the Proxmox website, but I don't know if it's possible to implement in practice, would be to configure the cluster servers as hosts and set the default QEMU CPU model to x86-64-v4, so that if there is this CPU flags problem, Proxmox would adopt this default, so that the VM would not be interrupted during the live migration. I believe maybe it might be possible to do this, as you can see what is written in the documentation on the Proxmox website:
"If you want to do a live migration of VMs between different hosts, your VM might end up on a new system with a different CPU type or a different microcode version. If the CPU flags passed to the guest are missing, the QEMU process will stop. To remedy this QEMU has also its own virtual CPU types, that Proxmox VE uses by default."
https://www.intel.com/content/www/u...ssor-16-5m-cache-2-20-ghz/specifications.html
https://www.intel.com.br/content/ww...ssor-37-5m-cache-2-00-ghz/specifications.html
Thank you.
Last edited: