Intel Xeon vs AMD Epyc

N

ns33

Member
Apr 4, 2024
53
4
13
Does anyone have insight on which is a better choice? Or pitfalls of one vs the other?

We're spec'ing a cluster out for work and seeking dual cpu nodes, min 64 threads per numa nodes. AMD does save a bit a cash but are there any negatives?

We do have a requirements to support nested virtualization as well as Nvidia vGPU
 
  • Like
Reactions: Francois Baillargeon
Hi @ns33,

It really depends on your applications. Neither vendor is a universally "better" choice, and the right answer comes from the workloads you plan to run.

That said, my general preference is single-socket servers for anything network or network-attached-storage related. Keeping everything in one socket removes cross-socket NUMA hops between your NICs, CPU, and memory, which makes latency more consistent and performance easier to predict. With today's core counts, a single socket often covers what used to need two.

We have a slight preference for AMD, but understanding your applications is essential to choosing the right SKUs. Core count, clock speed, cache layout, and memory bandwidth all matter differently depending on what you're running. I'd profile your workloads first and let that drive the platform choice rather than the price difference.

Nested virtualization and NVIDIA vGPU are both workable on either platform. For vGPU, check that your server model and GPU are on NVIDIA's supported list.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: Onslow and ns33
bbgeek17 said:
Hi @ns33,

It really depends on your applications. Neither vendor is a universally "better" choice, and the right answer comes from the workloads you plan to run.

That said, my general preference is single-socket servers for anything network or network-attached-storage related. Keeping everything in one socket removes cross-socket NUMA hops between your NICs, CPU, and memory, which makes latency more consistent and performance easier to predict. With today's core counts, a single socket often covers what used to need two.

We have a slight preference for AMD, but understanding your applications is essential to choosing the right SKUs. Core count, clock speed, cache layout, and memory bandwidth all matter differently depending on what you're running. I'd profile your workloads first and let that drive the platform choice rather than the price difference.

Nested virtualization and NVIDIA vGPU are both workable on either platform. For vGPU, check that your server model and GPU are on NVIDIA's supported list.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...
I second here, NUMA stuff sometimes make slowdowns hard to pin, when you find out a process is basically poking the neighbor through que inter socket interconnect. So go single socket if possible
 
  • Like
Reactions: Onslow and ns33
Thank you for the replies!

I didn't even think about the potential for numa issues in regards to networking/storage. I wrapped my head around it for vGPU; my dev nodes each have two physical gpus and each is installed so each cpu gets a gpu, but never considered this with networking/storage. Especially since we're going with u.2 and 100gb for ceph so pcie will be heavily used

Glad I asked this question!
 
Glad it helped, @ns33.

With U.2 and 100Gb, PCIe placement matters. On dual socket, keep your NVMe and the NIC carrying Ceph traffic on the same socket so I/O isn't crossing the inter-socket link. cat /sys/bus/pci/devices/<addr>/numa_node will tell you where each device lives. Check the board's block diagram before you order.

That said, with Ceph you may not notice much. Ceph adds enough latency per I/O that local vs. remote NUMA, or even NVMe vs. decent SAS SSDs, can get lost in the noise. Good placement still helps throughput and CPU efficiency, just don't expect it to show up in your latency numbers.

Your GPU layout complicates things too. One GPU per socket means one socket is remote from your storage NIC unless you add a second NIC.
This is why I lean single socket. A current EPYC gives you ~128 PCIe lanes, usually enough for U.2 drives, dual 100Gb, and GPUs, all in one NUMA domain. It is worth mapping out your lane budget per node.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: Francois Baillargeon
A few points I would add to the answers above:

Single socket does not automatically mean “one NUMA node”. On EPYC this depends on the NPS setting (NPS1/2/4), and Intel has a similar concept with SNC. Since you specifically require at least 64 threads per NUMA node, I would definitely include the intended NUMA configuration when choosing the CPU. Total core count alone does not tell you how many cores/threads each NUMA domain will actually have.

I would also be careful with the Intel <-> AMD live migration statement. With sufficiently generic CPU models it can work for normal VMs, but since nested virtualization is one of your requirements I would not design the cluster around cross-vendor migration. Intel VMX and AMD SVM are different enough that I would keep all nodes within the same CPU vendor, and preferably the same generation/family.

For your particular setup, PCIe topology may matter more than the Intel-vs-AMD question. With multiple U.2 NVMe drives, 100Gb Ceph networking and two GPUs per node, I would get the block diagram for every server candidate and map:

CPU/NUMA -> PCIe root -> GPU / NIC / NVMe backplane

Also note that a dual-socket system does not simply give you twice the usable PCIe connectivity, because part of the socket I/O is used for the inter-socket links. A modern single-socket EPYC platform can therefore be surprisingly attractive for an I/O-heavy virtualization host.

And regarding Ceph: I would not assume that NUMA placement or NVMe vs SAS SSD latency gets lost in the Ceph overhead. On a fast 100Gb all-flash cluster, CPU locality, PCIe placement and storage media can still be very visible, especially with small I/O and high concurrency.

So before choosing Xeon vs EPYC, I would first decide:
  • required cores/threads per NUMA domain
  • number and placement of GPUs
  • number of U.2/NVMe devices
  • 100Gb NIC placement
  • whether nested-virtualization VMs must live migrate between all nodes
That will probably narrow the CPU/platform choice more than the vendor name itself.
 
  • Like
Reactions: Onslow
You must log in or register to reply here.
Top Bottom
Back