AMD EPYC - Has NUMA ?

Mecanik

Member
Mar 2, 2017
116
2
23
30
Hello,

I`m very confused with NUMA, as I read from docs you have a NUMA capable host if you have multiple CPUs (sockets). However I have installed Proxmox 6.1 (latest) on a AMD EPYC 16 core CPU, and the command to check available numa returns:

Bash:
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 16 17 18 19
node 0 size: 64396 MB
node 0 free: 52367 MB
node 1 cpus: 4 5 6 7 20 21 22 23
node 1 size: 64484 MB
node 1 free: 57565 MB
node 2 cpus: 8 9 10 11 24 25 26 27
node 2 size: 64508 MB
node 2 free: 52654 MB
node 3 cpus: 12 13 14 15 28 29 30 31
node 3 size: 64509 MB
node 3 free: 50373 MB
node distances:
node   0   1   2   3
  0:  10  16  16  16
  1:  16  10  16  16
  2:  16  16  10  16
  3:  16  16  16  10

So this means I have NUMA capability ?

Thanks
 

tburger

Active Member
Oct 13, 2017
778
91
33
38
AMD EPYC processors work a little different than Intel XEONs.

AFAIK the architecure uses multiple, smaller "cpu packages" and combines them oonto a socket. That increases fab output efficiency and drives costs down. AMD recently published a doc (or was it in the earning-report?) that stated this. When trying to create a 32 or even 64 core processor on one DIE, it would likely not be possible to have a decent output of the fab.

So in your case it is not a surprise for me you see 4x 4core packages + HT in a 16-core processor - which ends up in 32 threads.

Try using the search engines. I guess you will find a lot of references about this.
 

Mecanik

Member
Mar 2, 2017
116
2
23
30
AMD EPYC processors work a little different than Intel XEONs.

AFAIK the architecure uses multiple, smaller "cpu packages" and combines them oonto a socket. That increases fab output efficiency and drives costs down. AMD recently published a doc (or was it in the earning-report?) that stated this. When trying to create a 32 or even 64 core processor on one DIE, it would likely not be possible to have a decent output of the fab.

So in your case it is not a surprise for me you see 4x 4core packages + HT in a 16-core processor - which ends up in 32 threads.

Try using the search engines. I guess you will find a lot of references about this.

You are correct, meanwhile I was searching about this. What can I say, long live AMD ?
 

tburger

Active Member
Oct 13, 2017
778
91
33
38
My guess: you can also hide the NUMA architecture from the OS via BIOS settings. Same as you can do on Intel platforms. So check if you find a matching BIOS setting (e.g. NUMA mode enabled, NUMA support, or similar)
 

narrateourale

Member
Dec 21, 2019
96
22
8
34
My guess: you can also hide the NUMA architecture from the OS via BIOS settings. Same as you can do on Intel platforms. So check if you find a matching BIOS setting (e.g. NUMA mode enabled, NUMA support, or similar)
That would have been my next guess and something that I will check on the next boot of that server, if I don't forget until then ;)
 

narrateourale

Member
Dec 21, 2019
96
22
8
34
Btw, which EPYC model do you have installed? I am just reading up on it again so this is from memory and probably not technically 100% correct.

EPYC Naples (1 at the end of the 4 digit product code) has Numa nodes because they had to group the chiplets and not every chiplet has the same fast low latency access to memory and IO.

EPYC Rome (2 at the end of the 4 digit product code) doesn't need Numa / is 1 big Numa node because they managed to have all compute chiplets connected the same way to the memory and IO controller.

In general Numa is something that is needed but not something that is either good or wanted.
 
  • Like
Reactions: Alwin

Mecanik

Member
Mar 2, 2017
116
2
23
30
Btw, which EPYC model do you have installed? I am just reading up on it again so this is from memory and probably not technically 100% correct.

EPYC Naples (1 at the end of the 4 digit product code) has Numa nodes because they had to group the chiplets and not every chiplet has the same fast low latency access to memory and IO.

EPYC Rome (2 at the end of the 4 digit product code) doesn't need Numa / is 1 big Numa node because they managed to have all compute chiplets connected the same way to the memory and IO controller.

In general Numa is something that is needed but not something that is either good or wanted.

I haveAMD Epyc 7371 .
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!