Skip to contentSkip to conversation

RFC: /proc/cpuinfo confusion with AMD processors

DISCUSSION Prarit Bhargava 3 participants 7 messages Jun 30, 2014 – Jul 2, 2014

lore
Share & tools
lore
#0 lore raw

AMD defines a "Package" as the hardware processor itself. Each Package contains multiple Nodes, and each Node has multiple Compute Units. Each Compute Unit can have up to 2 cores that [with the 62xx and 63xx] do not have multiple Threads.

That is, to determine the number of CPUs that Linux sees, multiply

Package * Nodes * Compute Units * Cores

Note that Nodes and Compute Units are not indicated in /proc/cpuinfo directly (although it could be argued that they should be).

The output of /proc/cpuinfo is confusing at this point as ...

processor : 31
vendor_id : AuthenticAMD
cpu family : 21
model : 2
model name : AMD Opteron(tm) Processor 6386 SE
stepping : 0
microcode : 0x6000822
cpu MHz : 2800.000
cache size : 2048 KB
physical id : 1
siblings : 16 <<< this is number of threads per package
core id : 7 <<< this is the core id of this thread relative to node
cpu cores : 8 <<< this is the number of cores per node

which makes deciphering the system topology quite difficult as values are relative to both nodes and the entire package. It is not possible using this information to uniquely identify a processor.

I want to make two changes. The first is to modify /proc/cpuinfo to include both the node information for all x86 processors and also include a "compute unit id" field for AMD systems.

The second change would be to include the same data in /sys which currently does not contain the above information in the topology directory.

Thoughts/concerns?

P.

#1 lore raw
Quoted 14 lines · Prarit Bhargava

AMD defines a "Package" as the hardware processor itself. Each Package contains multiple Nodes, and each Node has multiple Compute Units. Each Compute Unit can have up to 2 cores that [with the 62xx and 63xx] do not have multiple Threads.

That is, to determine the number of CPUs that Linux sees, multiply

Package * Nodes * Compute Units * Cores

Note that Nodes and Compute Units are not indicated in /proc/cpuinfo directly (although it could be argued that they should be).

The output of /proc/cpuinfo is confusing at this point as ...

Quoted 13 lines

processor : 31
vendor_id : AuthenticAMD
cpu family : 21
model : 2
model name : AMD Opteron(tm) Processor 6386 SE
stepping : 0
microcode : 0x6000822
cpu MHz : 2800.000
cache size : 2048 KB
physical id : 1
siblings : 16 <<< this is number of threads per package
core id : 7 <<< this is the core id of this thread relative to node
cpu cores : 8 <<< this is the number of cores per node

siblings / cpu cores = threads per compute unit.

Quoted 3 lines

which makes deciphering the system topology quite difficult as values are relative to both nodes and the entire package. It is not possible using this information to uniquely identify a processor.

To do what with that information? What is the task you're trying to accomplish?

Quoted 1 line

Thoughts/concerns?

BIOS does all kinds of hacks and renumbering to accomodate the brainf*cked design of other OSes so this info you're trying to put in cpuinfo might turn to be completely misleading utterly useless in some cases.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#2 lore raw
Quoted 29 lines · Borislav Petkov
Quoted 14 lines · Prarit Bhargava

AMD defines a "Package" as the hardware processor itself. Each Package contains multiple Nodes, and each Node has multiple Compute Units. Each Compute Unit can have up to 2 cores that [with the 62xx and 63xx] do not have multiple Threads.

That is, to determine the number of CPUs that Linux sees, multiply

Package * Nodes * Compute Units * Cores

Note that Nodes and Compute Units are not indicated in /proc/cpuinfo directly (although it could be argued that they should be).

The output of /proc/cpuinfo is confusing at this point as ...

Quoted 13 lines

processor : 31
vendor_id : AuthenticAMD
cpu family : 21
model : 2
model name : AMD Opteron(tm) Processor 6386 SE
stepping : 0
microcode : 0x6000822
cpu MHz : 2800.000
cache size : 2048 KB
physical id : 1
siblings : 16 <<< this is number of threads per package
core id : 7 <<< this is the core id of this thread relative to node
cpu cores : 8 <<< this is the number of cores per node

Quoted 1 line

siblings / cpu cores = threads per compute unit.

Yes, I get that. But this doesn't uniquely identify *which* processor it is.

Quoted 7 lines
Quoted 4 lines

which makes deciphering the system topology quite difficult as values are relative to both nodes and the entire package. It is not possible using this information to uniquely identify a processor.

To do what with that information? What is the task you're trying to accomplish?

Admins load systems relative to nodes, and look at /proc/cpuinfo as a "known" place to get that info. At the end of the day I'd like to be able to human-read /proc/cpuinfo and get the correct data out of it without jumping through hoops to get processor location information. I can, in theory, use the initial apicid and the apicid to map everything out ... but I shouldn't have to. /proc/cpuinfo provides an easy look up for this data for Intel; why not for AMD?

Additionally the turbostat utility, for example, is broken because of this lack of info. It assumes that /sys/../cpu/cpuX/topology/core_id is *per package* when on AMD systems it is reported as per node.

Quoted 7 lines
Quoted 2 lines

Thoughts/concerns?

BIOS does all kinds of hacks and renumbering to accomodate the brainf*cked design of other OSes so this info you're trying to put in cpuinfo might turn to be completely misleading utterly useless in some cases.

I read that exact same thing somewhere else. The argument you're making is that it might be broken for some other reason. It already is completely misleading and utterly useless in the above case. A simple test patch to arch/x86/kernel/cpu/proc.c shows that including the information above is trivial. Adding it /sys/.../cpu/cpuX/topology is a little bit more difficult.

P.

Quoted 1 line
#3 lore raw
Quoted 2 lines · Prarit Bhargava

Yes, I get that. But this doesn't uniquely identify *which* processor it is.

What do you mean, which processor it is? You want to know which processor on the motherboard, physically? When you look at the mobo and if all is enumerated regularly and you have a, say, two socket system with the processors above, to be able to say that processor 31 is the 16th core on the second node on the motherboard? Something like that?

Quoted 4 lines

Admins load systems relative to nodes, and look at /proc/cpuinfo as a "known" place to get that info. At the end of the day I'd like to be able to human-read /proc/cpuinfo and get the correct data out of it without jumping through hoops

What correct data? All that's there is correct AFAICT. Give an example of what you want to do?

Quoted 4 lines

to get processor location information. I can, in theory, use the initial apicid and the apicid to map everything out ... but I shouldn't have to. /proc/cpuinfo provides an easy look up for this data for Intel; why not for AMD?

What does it show on Intel that's clear there and that's puzzling you on AMD?

Quoted 3 lines

Additionally the turbostat utility, for example, is broken because of this lack of info. It assumes that /sys/../cpu/cpuX/topology/core_id is *per package* when on AMD systems it is reported as per node.

The turbostat utility might need some testing and fixing on AMD. If you can get some resources to do that I think everyone will profit from it.

Quoted 6 lines

I read that exact same thing somewhere else. The argument you're making is that it might be broken for some other reason. It already is completely misleading and utterly useless in the above case. A simple test patch to arch/x86/kernel/cpu/proc.c shows that including the information above is trivial. Adding it /sys/.../cpu/cpuX/topology is a little bit more difficult.

And I'm still saying that that information - albeit being trivial to add - might be lying on some system doing renumbering. So, again: what exactly do you want to be able to figure out from /proc/cpuinfo and what is puzzling you? Give concrete examples please...

Thanks.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#4 lore raw
Quoted 9 lines · Borislav Petkov
Quoted 2 lines · Prarit Bhargava

Yes, I get that. But this doesn't uniquely identify *which* processor it is.

What do you mean, which processor it is? You want to know which processor on the motherboard, physically? When you look at the mobo and if all is enumerated regularly and you have a, say, two socket system with the processors above, to be able to say that processor 31 is the 16th core on the second node on the motherboard? Something like that?

Sorry, yes, exactly that. Requests have come in where an admin is setting up system loads relative to specific nodes and cores. Determining that information is trivial on Intel and a lot more difficult on AMD. (see below)

Quoted 7 lines
Quoted 4 lines

to get processor location information. I can, in theory, use the initial apicid and the apicid to map everything out ... but I shouldn't have to. /proc/cpuinfo provides an easy look up for this data for Intel; why not for AMD?

What does it show on Intel that's clear there and that's puzzling you on AMD?

Intel does not have the concept of multi-node packages (AFAIK). So the Intel existing physical id and core id (both of which are relative to the package) is good enough to determine which core it is. For example from an Intel box

processor : 3
...
physical id : 3
siblings : 2
core id : 1
cpu cores : 2

tells me that this processor 3 is on socket/package (physical id) 3, core 1 out of 2, and the cores are not hyperthreaded. On a system in which an admin would like to (for example) set cpu affinity for a VM or a particular application knowing this information is useful.

On an AMD system,

processor : 31
physical id : 1
siblings : 16
core id : 7
cpu cores : 8

implies (using the logic above), package/socket 1, core 7/8 cores, and multi-threading is on ... which is incorrect. The package in question is 16 thread and 16 core with no multi-threading.

The difference in the result occurs because AMD is (IMO) erroneously reporting per node information. Another view of this could be that on AMD systems we should modify the output to report per package information to make it consistent with Intel (and other arches ... ppc reports per socket as does s390. I haven't checked ARM yet). AMD is the outlier here.

Quoted 7 lines
Quoted 4 lines

Additionally the turbostat utility, for example, is broken because of this lack of info. It assumes that /sys/../cpu/cpuX/topology/core_id is *per package* when on AMD systems it is reported as per node.

The turbostat utility might need some testing and fixing on AMD. If you can get some resources to do that I think everyone will profit from it.

The problem isn't turbostat though -- it is that core_id implies two different things for two different x86 processors. Intel reports that value as per package, while AMD reports it as per node (and there is no way to determine which node it is). The problem is that the values are ambiguous relative to processor vendor and they shouldn't be.

P.

#5 lore raw
Quoted 4 lines · Prarit Bhargava

Sorry, yes, exactly that. Requests have come in where an admin is setting up system loads relative to specific nodes and cores. Determining that information is trivial on Intel and a lot more difficult on AMD. (see below)

Yeah, that might not always work, see below.

Quoted 1 line

Intel does not have the concept of multi-node packages (AFAIK).

Let's establish terminology first: So I think you mean MCM - Multi-Chip Module where you have more than one nodes, right?

https://en.wikipedia.org/wiki/Multi-chip_module

And according to the wiki article, Intel has that too.

HOWEVER(!), it is a whole another question whether that is made visible to the software!

Quoted 21 lines

So the Intel
existing physical id and core id (both of which are relative to the package) is good enough to determine which core it is. For example from an Intel box

processor : 3
...
physical id : 3
siblings : 2
core id : 1
cpu cores : 2

tells me that this processor 3 is on socket/package (physical id) 3, core 1 out of 2, and the cores are not hyperthreaded. On a system in which an admin would like to (for example) set cpu affinity for a VM or a particular application knowing this information is useful.

On an AMD system,

Quoted 7 lines

processor : 31
physical id : 1
siblings : 16
core id : 7
cpu cores : 8

implies (using the logic above), package/socket 1,

Yes, the second socket.

Quoted 1 line

core 7/8 cores, and multi-threading is on ... which is incorrect.

What says that multithreading is on? How do you decide that? I think you're reading too much into it.

Quoted 1 line

The package in question is 16 thread and 16 core with no multi-threading.

No, this is wrong.

So you are on socket 1 ("physical id"), it has 16 siblings, i.e. threads. In this case, 16 threads means 16/2 = 8 compute units. And core id 7 means the id of the the core on this NUMA node.

But just to show you that this information can be misleading, let's look at another AMD box:

processor : 30
cpu family : 21
model : 1
model name : AMD Opteron(TM) Processor 6272
stepping : 2
microcode : 0x600063d
cpu MHz : 1400.000
cache size : 2048 KB
physical id : 0
siblings : 16
core id : 7
cpu cores : 8
apicid : 47
initial apicid : 15

processor : 31
cpu family : 21
model : 1
model name : AMD Opteron(TM) Processor 6272
stepping : 2
microcode : 0x600063d
cpu MHz : 1400.000
cache size : 2048 KB
physical id : 1
siblings : 16
core id : 7
cpu cores : 8
apicid : 79
initial apicid : 47

So those are the last two cores, 30 and 31.

Now look at physical id - 30 is on node 0 and 31 is on node 1. So thread 30 is on physical package 0 and thread 31 is on physical package 1, i.e. on different physical sockets.

And yet, all the info there is correct - it is just not complete for your purposes.

If you want to have the detailed information you're interested in, simply do:

$ grep . /sys/devices/system/cpu/cpu30/topology/*
/sys/devices/system/cpu/cpu30/topology/core_id:7
/sys/devices/system/cpu/cpu30/topology/core_siblings:55555555
/sys/devices/system/cpu/cpu30/topology/core_siblings_list:0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30

All the core siblings, i.e. threads on this node, i.e. package 0.

/sys/devices/system/cpu/cpu30/topology/physical_package_id:0

This is the physical CPU package 0 on the socket in the motherboard.

/sys/devices/system/cpu/cpu30/topology/thread_siblings:50000000 /sys/devices/system/cpu/cpu30/topology/thread_siblings_list:28,30

This is thread 30 in a compute unit together with thread 28.

$ grep . /sys/devices/system/cpu/cpu31/topology/*
/sys/devices/system/cpu/cpu31/topology/core_id:7
/sys/devices/system/cpu/cpu31/topology/core_siblings:aaaaaaaa
/sys/devices/system/cpu/cpu31/topology/core_siblings_list:1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31
/sys/devices/system/cpu/cpu31/topology/physical_package_id:1

This is the physical CPU package 1 on the socket in the motherboard.

/sys/devices/system/cpu/cpu31/topology/thread_siblings:a0000000 /sys/devices/system/cpu/cpu31/topology/thread_siblings_list:29,31

This is thread 31 in a compute unit with thread 29 together.

At the time when we were talking about adding that info to /proc/cpuinfo, the intention was declined by reviewers and it was said that we have the topology hierarchy in sysfs where it all should be. And I agree now, btw...

Quoted 6 lines

The difference in the result occurs because AMD is (IMO) erroneously reporting per node information. Another view of this could be that on AMD systems we should modify the output to report per package information to make it consistent with Intel (and other arches ... ppc reports per socket as does s390. I haven't checked ARM yet). AMD is the outlier here.

... you can't express complicated topologies in /proc/cpuinfo. This is nicely done in sysfs where all tools can access it.

Quoted 4 lines

The problem isn't turbostat though -- it is that core_id implies two different things for two different x86 processors. Intel reports that value as per package, while AMD reports it as per node (and there is no way to determine which node it is).

No, you're making the wrong assumption that a single package can have only one NUMA node. Which is basically not true on AMD. core_id is the id of the core on that NUMA node. It means the same on Intel *and* AMD.

IF you have more than one nodes in a package, the core_id is still correct as it is the numbering within the NUMA node.

Again, /proc/cpuinfo is not the proper place for complicated topologies, at least on x86.

For NUMA we have different tools like numactl which give you that precise information of the system topology and there was this other tool which even generates PDFs of the topology. I'm forgetting the name right now though.

HTH.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#6 lore raw
In reply to Borislav Petkov
Quoted 9 lines · Borislav Petkov
Quoted 2 lines · Prarit Bhargava

Yes, I get that. But this doesn't uniquely identify *which* processor it is.

What do you mean, which processor it is? You want to know which processor on the motherboard, physically? When you look at the mobo and if all is enumerated regularly and you have a, say, two socket system with the processors above, to be able to say that processor 31 is the 16th core on the second node on the motherboard? Something like that?

I'd say so. Some machines support physical cpu hotplug, and it would be useful to know which cpu you are plugging out...

(For example if you want to pull out / replace the cpu that has problems... Or maybe more realistically you want to offline a CPU so that you can replace a noisy CPU fan.)
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html