Missing CPU Cores

T

Thalhammer

Active Member
May 2, 2016
14
0
41
27
Hello,

I have installed Proxmox VE 4.2.2 on my IBM x3850 M2 Server.
This server has a total of 4 CPU's with 6 Cores each.
However only 20 of these are used by PVE.

lscpu reports this:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 20
On-line CPU(s) list: 0-15,20-23
Thread(s) per core: 1
Core(s) per socket: 5
Socket(s): 4
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 29
Model name: Intel(R) Xeon(R) CPU X7460 @ 2.66GHz
Stepping: 1
CPU MHz: 2665.622
BogoMIPS: 5331.31
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 3072K
L3 cache: 16384K
NUMA node0 CPU(s): 0-15,20-23​

Looking in the bios all of them are available.
Thanks in advance,
Thalhammer
 
The problem is that theres no folder for these cores in /sys/devices/system/cpu.
Only the other 20 cores are listed there.
Digging in dmesg brings up this:

[ 0.092140] smpboot: Max logical packages: 4
[ 0.092143] smpboot: APIC(8) Converting physical 1 to logical package 0
[ 0.092144] smpboot: APIC(20) Converting physical 4 to logical package 1
[ 0.092146] smpboot: APIC(10) Converting physical 2 to logical package 2
[ 0.092147] smpboot: APIC(38) Converting physical 7 to logical package 3
[ 0.092151] smpboot: APIC(6c) Package 13 exceeds logical package map
[ 0.092153] smpboot: CPU 16 APICId 6c disabled
[ 0.092154] smpboot: APIC(44) Package 8 exceeds logical package map
[ 0.092155] smpboot: CPU 17 APICId 44 disabled
[ 0.092157] smpboot: APIC(54) Package 10 exceeds logical package map
[ 0.092158] smpboot: CPU 18 APICId 54 disabled
[ 0.092159] smpboot: APIC(7c) Package 15 exceeds logical package map
[ 0.092160] smpboot: CPU 19 APICId 7c disabled
[ 0.093119] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1​

Which shows that linux ideed finds them but disables them for some reason.
I'm quite sure that this is not a hardware problem because it worked with a debian install some time ago, but to make sure I'am going to run a live cd and make sure they are recognised there.

Just installed Debian 8 and everything is detected fine there, so this is definitly a problem of proxmox.
Installing Proxmox on this Debian server installs a 4.4 Kernel from your repository which seems to break things. The 3.16 Kernel installed by debian by default works fine.
Installing Proxmox and booting the 3.16 Kernel of Debian seems to work to.

// EDIT: Current Config:
Debian 8 + proxmox VE + Kernel 4.5 (jessie-backports)
Is working fine and I could not find any errors so far.
 
Last edited:
Hi folks,

is there now another solution , except the debian Installation ?
I have exactly the same problem.
 
What I can see in lscpu output is that your NUMA nodes == 1. Try to enable NUMA in BIOS?
Here's my output for 2 sockets, 6 cores/socket, 2 threads/core:

Code: 
# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                24
On-line CPU(s) list:   0-23
Thread(s) per core:    2
Core(s) per socket:    6
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 44
Model name:            Intel(R) Xeon(R) CPU           X5670  @ 2.93GHz
Stepping:              2
CPU MHz:               1596.000
CPU max MHz:           2927.0000
CPU min MHz:           1596.0000
BogoMIPS:              5852.60
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              12288K
NUMA node0 CPU(s):     0,2,4,6,8,10,12,14,16,18,20,22
NUMA node1 CPU(s):     1,3,5,7,9,11,13,15,17,19,21,23
 
root@ibm-x3850m2:/sys/devices/system/cpu# lscpu

Architecture: x86_64

CPU op-mode(s): 32-bit, 64-bit

Byte Order: Little Endian

CPU(s): 20

On-line CPU(s) list: 0-15,20-23

Thread(s) per core: 1

Core(s) per socket: 5

Socket(s): 4

NUMA node(s): 1

Vendor ID: GenuineIntel

CPU family: 6

Model: 29

Model name: Intel(R) Xeon(R) CPU X7460 @ 2.66GHz

Stepping: 1

CPU MHz: 2665.260

BogoMIPS: 5331.36

Virtualization: VT-x

L1d cache: 32K

L1i cache: 32K

L2 cache: 3072K

L3 cache: 16384K

NUMA node0 CPU(s): 0-15,20-23
 
Oh, dear, because I can equal bake a new kernel. . .
Have you built a new kernel or patched with diff?
 
I didn't do anything. It works on my setup. Just googled for "exceeds logical package map". And considering that you and @Thalhammer present these issues and a patch regarding CPU topology is recent, it makes sense to assume that something is going on in this area.

LE: What concerns me is that your systems present a single NUMA node across all sockets, but mine creates a node for each socket.
 
Here's an older system that I have access to:

Code: 
# cat /proc/cpuinfo | grep "model name" | head -n 1
model name    : Intel(R) Xeon(R) CPU E7-L8867  @ 2.13GHz

Code: 
# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                80
On-line CPU(s) list:   0-79
Thread(s) per core:    2
Core(s) per socket:    10
Socket(s):             4
NUMA node(s):          4
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 47
Stepping:              2
CPU MHz:               2127.979
BogoMIPS:              4255.90
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              30720K
NUMA node0 CPU(s):     0,4,8,12,16,20,24,28,32,36,40,44,48,52,56,60,64,68,72,76
NUMA node1 CPU(s):     1,5,9,13,17,21,25,29,33,37,41,45,49,53,57,61,65,69,73,77
NUMA node2 CPU(s):     2,6,10,14,18,22,26,30,34,38,42,46,50,54,58,62,66,70,74,78
NUMA node3 CPU(s):     3,7,11,15,19,23,27,31,35,39,43,47,51,55,59,63,67,71,75,79
 
What's the output of numastat on your systems?

Code: 
# numastat
                           node0           node1
numa_hit              1475179414       704657639
numa_miss               14347898       119440717
numa_foreign           119440717        14347898
interleave_hit             22634           23857
local_node            1475121851       704619922
other_node              14405461       119478434
 
I'm not 100% sure but as far as I can remember, there is a bios settings to select whether all cores are presented using 1 or more numa nodes.
However I don't run this system any more.
I still have it laying around but I switched to a more modern (and faaaaaarrr more energy efficient) setup using a Xeon E5-2620 v4.

Compiling a new Kernel is not such a big problem, even if you have "only" 20 cores.
 
I have set in BIOS > advanced > CPU Settings > locical to physical, now i have all CPU Cores
Code: 
root@ibm-x3850m2:~# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                24
On-line CPU(s) list:   0-23
Thread(s) per core:    1
Core(s) per socket:    6
Socket(s):             4
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 29
Model name:            Intel(R) Xeon(R) CPU           X7460  @ 2.66GHz
Stepping:              1
CPU MHz:               2660.000
CPU max MHz:           2660.0000
CPU min MHz:           2128.0000
BogoMIPS:              5331.32
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              3072K
L3 cache:              16384K
NUMA node0 CPU(s):     0-23
 
Energy is not the problem, flatrate from sun ;-)
but the old IBM have 256GB RAM and now 24 Cores ;-)
 
Vogel said:
Energy is not the problem, flatrate from sun ;-)
but the old IBM have 256GB RAM and now 24 Cores ;-)
Click to expand...
Me too (kind of) but also sun power isn't endless.
Having 1/2 power consumption results in 2x possible compute power;)
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back