AMD EPYC - Has NUMA ?

[Mecanik]

Hello,

I`m very confused with NUMA, as I read from docs you have a NUMA capable host if you have multiple CPUs (sockets). However I have installed Proxmox 6.1 (latest) on a AMD EPYC 16 core CPU, and the command to check available numa returns:

available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 16 17 18 19
node 0 size: 64396 MB
node 0 free: 52367 MB
node 1 cpus: 4 5 6 7 20 21 22 23
node 1 size: 64484 MB
node 1 free: 57565 MB
node 2 cpus: 8 9 10 11 24 25 26 27
node 2 size: 64508 MB
node 2 free: 52654 MB
node 3 cpus: 12 13 14 15 28 29 30 31
node 3 size: 64509 MB
node 3 free: 50373 MB
node distances:
node   0   1   2   3
  0:  10  16  16  16
  1:  16  10  16  16
  2:  16  16  10  16
  3:  16  16  16  10

So this means I have NUMA capability ?

Thanks

 

[apoc]

AMD EPYC processors work a little different than Intel XEONs.

AFAIK the architecure uses multiple, smaller "cpu packages" and combines them oonto a socket. That increases fab output efficiency and drives costs down. AMD recently published a doc (or was it in the earning-report?) that stated this. When trying to create a 32 or even 64 core processor on one DIE, it would likely not be possible to have a decent output of the fab.

So in your case it is not a surprise for me you see 4x 4core packages + HT in a 16-core processor - which ends up in 32 threads.

Try using the search engines. I guess you will find a lot of references about this.

 

[Mecanik]

AMD EPYC processors work a little different than Intel XEONs.

AFAIK the architecure uses multiple, smaller "cpu packages" and combines them oonto a socket. That increases fab output efficiency and drives costs down. AMD recently published a doc (or was it in the earning-report?) that stated this. When trying to create a 32 or even 64 core processor on one DIE, it would likely not be possible to have a decent output of the fab.

So in your case it is not a surprise for me you see 4x 4core packages + HT in a 16-core processor - which ends up in 32 threads.

Try using the search engines. I guess you will find a lot of references about this.

You are correct, meanwhile I was searching about this. What can I say, long live AMD ?

 

[apoc]

long live AMD

I like my Opteron 6300 series rig as well

 

[narrateourale]

the command to check available numa returns

which command did you use to get that output?

On a AMD EPYC 7302P 16-Core Processor

I get the following:

# dmesg | grep -i numa
[    0.006392] No NUMA configuration found

 

[apoc]

My guess: you can also hide the NUMA architecture from the OS via BIOS settings. Same as you can do on Intel platforms. So check if you find a matching BIOS setting (e.g. NUMA mode enabled, NUMA support, or similar)

 

[narrateourale]

My guess: you can also hide the NUMA architecture from the OS via BIOS settings. Same as you can do on Intel platforms. So check if you find a matching BIOS setting (e.g. NUMA mode enabled, NUMA support, or similar)

That would have been my next guess and something that I will check on the next boot of that server, if I don't forget until then

 

[Mecanik]

which command did you use to get that output?

On a AMD EPYC 7302P 16-Core Processor

I get the following:

# dmesg | grep -i numa
[    0.006392] No NUMA configuration found

numactl -H

 

[narrateourale]

Btw, which EPYC model do you have installed? I am just reading up on it again so this is from memory and probably not technically 100% correct.

EPYC Naples (1 at the end of the 4 digit product code) has Numa nodes because they had to group the chiplets and not every chiplet has the same fast low latency access to memory and IO.

EPYC Rome (2 at the end of the 4 digit product code) doesn't need Numa / is 1 big Numa node because they managed to have all compute chiplets connected the same way to the memory and IO controller.

In general Numa is something that is needed but not something that is either good or wanted.

 

[Mecanik]

Btw, which EPYC model do you have installed? I am just reading up on it again so this is from memory and probably not technically 100% correct.

EPYC Naples (1 at the end of the 4 digit product code) has Numa nodes because they had to group the chiplets and not every chiplet has the same fast low latency access to memory and IO.

EPYC Rome (2 at the end of the 4 digit product code) doesn't need Numa / is 1 big Numa node because they managed to have all compute chiplets connected the same way to the memory and IO controller.

In general Numa is something that is needed but not something that is either good or wanted.

I haveAMD Epyc 7371 .