x86: adapt CPU topology detection for AMD Magny-Cours

Signed-off-by Andreas Herrmann <[email protected]>

---
 arch/x86/include/asm/processor.h |    2 ++
 arch/x86/kernel/cpu/common.c     |    2 ++
 arch/x86/kernel/cpu/proc.c       |    1 +
 arch/x86/kernel/smpboot.c        |    5 ++++-
 4 files changed, 9 insertions(+), 1 deletions(-)

arch/x86/include/asm/processor.h +0 -0

index 0b2fab0..b49d72b 100644

Quoted 8 lines · Andreas Herrmann

Signed-off-by Andreas Herrmann <[email protected]>

---
 arch/x86/include/asm/processor.h |    2 ++
 arch/x86/kernel/cpu/common.c     |    2 ++
 arch/x86/kernel/cpu/proc.c       |    1 +
 arch/x86/kernel/smpboot.c        |    5 ++++-
 4 files changed, 9 insertions(+), 1 deletions(-)

arch/x86/include/asm/processor.h struct cpuinfo_x86 { · line 110

Quoted 10 lines

arch/x86/include/asm/processor.h +2 -0

index 0b2fab0..b49d72b 100644
@@ -106,6 +106,8 @@ struct cpuinfo_x86 {
 	u16			booted_cores;
 	/* Physical processor id: */
 	u16			phys_proc_id;
+	/* Node id in case of multi-node processor: */
+	u16			cpu_node_id;

btw., do you have any plans to propagate this information into the scheduler domains tree?

Another level of domains, to cover the two internal nodes, would do the trick nicely and automatically. This would work even if the BIOS does not provide information and we have to go to lowlevel registers or CPUID to recover it.

This can be done even if there's no SRAT. (there's no SRAT because say due to interleaving there's no real NUMA structure of memory. But there's still CPU scheduling differences worth expressing.)

Hi,

Following patches add support for AMD Magny-Cours CPU.

I slightly change struct cpuinfo where I'd like to introduce cpu_node_id to reflect CPU topology for AMD Magny-Cours CPU which consists of two internal-nodes.

For all cores on the same multi-node CPU (Magny-Cours) /proc/cpuinfo will show:
- same phys_proc_id
- cpu_node_id of the internal node (0 or 1)
- cpu_core_id (e.g. in range of 0 to 5)

I also change identification of core siblings (and thread siblings) which will also be based on cpu_node_id in addition to phys_proc_id.

Furthermore I adapt the L3 cache information to reflect the cache characteristics of one internal node instead of the entire package.

Primarily this changes are needed to correct core sibling information for Magny-Cours. This CPU has two NBs on one physical package -- each internal node has its own processor configuration space (i.e. set of northbridge PCI functions).

Following patches add support for AMD Magny-Cours CPU.

I slightly change struct cpuinfo where I'd like to introduce cpu_node_id to reflect CPU topology for AMD Magny-Cours CPU which consists of two internal-nodes.

Quoted 5 lines · Andreas Herrmann

Following patches add support for AMD Magny-Cours CPU.

I slightly change struct cpuinfo where I'd like to introduce cpu_node_id to reflect CPU topology for AMD Magny-Cours CPU which consists of two internal-nodes.

It's unclear to me why you need this special case versus just using the normal NUMA distances to represent the internal nodes as "nearby nodes" versus sockets as "farer away".

Essentially it should be just two level NUMA which can be already described fine for the scheduler or VM using the existing SRAT mechanism.

Who needs this additional information?

Best example is node interleaving. Usually you won't get a SRAT table on such a system.

/sys/devices/system/node. But on such a configuration you still see (and you want to see) the correct CPU topology information in /sys/devices/system/cpu/cpuX/topology. Based on that you always can figure out which cores are on the same physical package independent of availability and contents of SRAT and even with kernels that are compiled w/o NUMA support.

physical package == two northbridges (two nodes)

and this needs to be represented somehow in the kernel.

Quoted 2 lines

Who needs this additional information?

The kernel needs to know this when accessing processor configuration space, when accessing shared MSRs or for counting northbridge specific events.

Quoted 2 lines

Best example is node interleaving. Usually you won't get a SRAT table on such a system.

That sounds like a BIOS bug. It should supply a suitable SLIT/SRAT even for this case. Or perhaps if the BIOS are really that broken add a suitable quirk that provides distances, but better fix the BIOSes.

Quoted 7 lines

Thus you see just one NUMA node in /sys/devices/system/node. But on such a configuration you still see (and you want to see) the correct CPU topology information in /sys/devices/system/cpu/cpuX/topology. Based on that you always can figure out which cores are on the same physical package independent of availability and contents of SRAT and even with kernels that are compiled w/o NUMA support.

So you're adding a x86 specific mini NUMA for kernels without NUMA (which btw becomes more and more an exotic case -- modern distros are normally unconditionally NUMA) Doesn't seem very useful.

My problem with that is that imho the x86 topology information is already too complicated --

i suspect very few people can make sense of it --
and you're making it even worse, adding another strange special case.

On the other hand NUMA topology is comparatively straight forward and well understood and it's flexible enough to express your case too.

Quoted 3 lines

physical package == two northbridges (two nodes)

and this needs to be represented somehow in the kernel.

It's just two nodes with a very fast interconnect.

Quoted 5 lines

Quoted 1 line

Who needs this additional information?

The kernel needs to know this when accessing processor configuration space, when accessing shared MSRs or for counting northbridge specific events.

You're saying there are MSRs shared between the two in package nodes?

Quoted 3 lines · Andi Kleen

Quoted 2 lines

Best example is node interleaving. Usually you won't get a SRAT table on such a system.

Quoted 3 lines

That sounds like a BIOS bug. It should supply a suitable SLIT/SRAT even for this case. Or perhaps if the BIOS are really that broken add a suitable quirk that provides distances, but better fix the BIOSes.

How do you define SRAT when node interleaving is enabled? (Defining same distances between all nodes, describing only one node, or omitting SRAT entirely? I've observed that the latter is common behavior.)

Quoted 9 lines

Quoted 8 lines

Thus you see just one NUMA node in /sys/devices/system/node. But on such a configuration you still see (and you want to see) the correct CPU topology information in /sys/devices/system/cpu/cpuX/topology. Based on that you always can figure out which cores are on the same physical package independent of availability and contents of SRAT and even with kernels that are compiled w/o NUMA support.

Quoted 3 lines

So you're adding a x86 specific mini NUMA for kernels without NUMA (which btw becomes more and more an exotic case -- modern distros are normally unconditionally NUMA) Doesn't seem very useful.

No, I just tried to give an example why you can't derive CPU topology

Quoted 1 line

and you're making it even worse, adding another strange special case.

It's an abstraction -- I think of it just as another level in the CPU

hierarchy -- where existing CPUs and multi-node CPUs fit in:

physical package --> processor node --> processor core --> thread

I guess the problem is that you are associating node always with NUMA. Would it help to rename cpu_node_id to something else?

or something entirely different?

Quoted 5 lines

On the other hand NUMA topology is comparatively straight forward and well understood and it's flexible enough to express your case too.

Quoted 2 lines

physical package == two northbridges (two nodes)

Quoted 3 lines

Quoted 1 line

and this needs to be represented somehow in the kernel.

It's just two nodes with a very fast interconnect.

In fact, I also thought about representing each internal node as one physical package. But that is even worse as you can't figure out which node is on the same socket.

The best solution is to reflect the correct CPU topology (all levels of the hierarchy) in the kernel. As another use case: for power management you might want to know both which cores are on which internal node _and_ which nodes are on the same physical package.

Quoted 8 lines

Quoted 6 lines

Quoted 2 lines

Who needs this additional information?

The kernel needs to know this when accessing processor configuration space, when accessing shared MSRs or for counting northbridge specific events.

You're saying there are MSRs shared between the two in package nodes?

No. I referred to NB MSRs that are shared between the cores on the same (internal) node.

Quoted 8 lines · Andreas Herrmann

Quoted 6 lines · Andi Kleen

Quoted 2 lines

Best example is node interleaving. Usually you won't get a SRAT table on such a system.

That sounds like a BIOS bug. It should supply a suitable SLIT/SRAT even for this case. Or perhaps if the BIOS are really that broken add a suitable quirk that provides distances, but better fix the BIOSes.

Quoted 4 lines

How do you define SRAT when node interleaving is enabled? (Defining same distances between all nodes, describing only one node, or omitting SRAT entirely? I've observed that the latter is common behavior.)

Either a memory less node with 10 distance (which seems to be envogue recently for some reason) or multiple nodes with 10 distance.

Quoted 8 lines

Quoted 8 lines

Quoted 7 lines

Thus you see just one NUMA node in /sys/devices/system/node. But on such a configuration you still see (and you want to see) the correct CPU topology information in /sys/devices/system/cpu/cpuX/topology. Based on that you always can figure out which cores are on the same physical package independent of availability and contents of SRAT and even with kernels that are compiled w/o NUMA support.

Quoted 5 lines

Quoted 3 lines

So you're adding a x86 specific mini NUMA for kernels without NUMA (which btw becomes more and more an exotic case -- modern distros are normally unconditionally NUMA) Doesn't seem very useful.

No, I just tried to give an example why you can't derive CPU topology

First I must say it's unclear to me if CPU topology is really generally useful to export to the user.

If they want to know how far cores
are away they should look at cache sharing and NUMA distances (especially cache topology gives a very good approximation anyways). For other purposes like power management just having arbitary sets (x is shared with y in a set without hierarchy) seems to work just fine.

Then traditionally there were special cases for SMT and for packages (for error containment etc.) and some hacks for licensing, but these don't really apply in your case or can be already expressed in other ways.

If there is really a good use case for exporting CPU topology I would argue for not adding another adhoc level, but just export a SLIT style arbitary distance table somewhere in sys. That would support to express any possible future hierarchies too. But again I have doubts that's really needed at all.

Quoted 3 lines

Quoted 1 line

and you're making it even worse, adding another strange special case.

It's an abstraction -- I think of it just as another level in the CPU

It's not a general abstraction, just another ad-hoc hack.

Quoted 6 lines

hierarchy -- where existing CPUs and multi-node CPUs fit in:

physical package --> processor node --> processor core --> thread

I guess the problem is that you are associating node always with NUMA. Would it help to rename cpu_node_id to something else?

Nope. It's a general problem, renaming it won't make it better.

Quoted 9 lines

or something entirely different?

Quoted 7 lines

On the other hand NUMA topology is comparatively straight forward and well understood and it's flexible enough to express your case too.

Quoted 3 lines

physical package == two northbridges (two nodes)

and this needs to be represented somehow in the kernel.

Quoted 5 lines

Quoted 1 line

It's just two nodes with a very fast interconnect.

In fact, I also thought about representing each internal node as one physical package. But that is even worse as you can't figure out which node is on the same socket.

That's what the physical id is for.

Quoted 4 lines

Quoted 1 line

You're saying there are MSRs shared between the two in package nodes?

No. I referred to NB MSRs that are shared between the cores on the same (internal) node.

Just check siblings then.

Quoted 12 lines

Quoted 12 lines

Quoted 11 lines

Quoted 7 lines

Thus you see just one NUMA node in /sys/devices/system/node. But on such a configuration you still see (and you want to see) the correct CPU topology information in /sys/devices/system/cpu/cpuX/topology. Based on that you always can figure out which cores are on the same physical package independent of availability and contents of SRAT and even with kernels that are compiled w/o NUMA support.

So you're adding a x86 specific mini NUMA for kernels without NUMA (which btw becomes more and more an exotic case -- modern distros are normally unconditionally NUMA) Doesn't seem very useful.

Quoted 4 lines

Quoted 1 line

No, I just tried to give an example why you can't derive CPU topology

First I must say it's unclear to me if CPU topology is really generally useful to export to the user.

I think it is useful.

(Linux already provides this kind of information. But it should pass only useful/correct information to user space. For Magny-Cours sibling information provided by current Linux kernel is not very useful.)

Quoted 5 lines

If they want to know how far cores
are away they should look at cache sharing and NUMA distances (especially cache topology gives a very good approximation anyways). For other purposes like power management just having arbitary sets (x is shared with y in a set without hierarchy) seems to work just fine.

(a) Are you saying that users have to check NUMA distances when they want to pin tasks on certain CPUs?

(b) Just an example, if SRAT describes

"Either a memory less node with 10 distance (which seems to be
envogue recently for some reason) or multiple nodes with 10 distance."

how would you do (a) to pin tasks say on the same internal node or

on the same physical package? That is not straightforward. But

representing entire CPU topology in sysfs makes this obvious.

Quoted 5 lines

Then traditionally there were special cases for SMT and for packages (for error containment etc.) and some hacks for licensing, but these don't really apply in your case or can be already expressed in other ways.

Yes, it's a new "special case" which can't be expressed in other ways.

SLIT and SRAT are not sufficient.

The kernel

needs to know which cores are on same internal node. The

Quoted 5 lines

Quoted 3 lines

Quoted 1 line

and you're making it even worse, adding another strange special case.

It's an abstraction -- I think of it just as another level in the CPU

It's not a general abstraction, just another ad-hoc hack.

Fine.
But do you have any coNSTructive and usable suggestion how Linux

should handle topology information for multi-node processors?

Quoted 4 lines

Quoted 4 lines

hierarchy -- where existing CPUs and multi-node CPUs fit in:

physical package --> processor node --> processor core --> thread

Quoted 4 lines

Quoted 2 lines

I guess the problem is that you are associating node always with NUMA. Would it help to rename cpu_node_id to something else?

Nope. It's a general problem, renaming it won't make it better.

I don't see "a general problem". There is just a new CPU introducing a topology that slightly differs from what we had so far. Adapting the kernel shouldn't be that problematic.

On Tue, May 05, 2009 at 04:40:27PM +0200, Andreas Herrmann wrote:

I think the general problem with your patchset is that it seems more like a solution in search of a problem, instead of the proper other way around.

Quoted 4 lines

Quoted 2 lines

First I must say it's unclear to me if CPU topology is really generally useful to export to the user.

I think it is useful.

You forgot to state for what?

Quoted 2 lines

(a) Are you saying that users have to check NUMA distances when they want to pin tasks on certain CPUs?

CPU == core ? No you just bind to that CPU. Was that a trick question?

Quoted 1 line

representing entire CPU topology in sysfs makes this obvious.

You didn't do that.

Quoted 1 line

Yes, it's a new "special case" which can't be expressed in other ways.

You seem to assume that it's obviously useful to express. Perhaps I'm dumb, but can you explain for what?

Quoted 3 lines

SLIT and SRAT are not sufficient.

The kernel

Which part of the kernel?

The general problem is that you're trying to stretch an old ad-hoc hack (siblings etc. in /proc/cpuinfo) to a complicated new graph structure and the interface is just not up to that.

As a exercise you can try to write a user space program that uses these fields to query the topology. It will be a mess.

We went through the same with the caches. Initially /proc/cpuinfo tried to express the caches. At some point it just became too messy and cpuinfo now only gives a very simplified "legacy" field and the real information is in /sys. It went into /sys because there was a clear use case. If there's a clear use case for exposing complex CPU topology (so far that's still an open question due to lack of good examples) then also a new proper non hacky interface in /sys would make sense.

Quoted 6 lines

Quoted 4 lines

Quoted 2 lines

First I must say it's unclear to me if CPU topology is really generally useful to export to the user.

I think it is useful.

You forgot to state for what?

You are kidding, aren't you? Isn't this obvious?
Why shouldn't a user be interested in stuff like core_siblings and thread_siblings? Maybe a user want to make scheduling decisions based on that and pin tasks accordingly?

Quoted 4 lines

Quoted 2 lines

(a) Are you saying that users have to check NUMA distances when they want to pin tasks on certain CPUs?

CPU == core ? No you just bind to that CPU. Was that a trick question?

Of course that was no trick question -- at most a stupid typo. (Sometimes in Linux CPU == core.) So, no, sorry I meant "certain cores". (And I meant not pinning to one core but to a set of cores).

Quoted 5 lines

Quoted 4 lines

SLIT and SRAT are not sufficient.

The kernel

Quoted 1 line

Which part of the kernel?

I provided this info in my first reply to you. Here it is again:

"The kernel needs to know this when accessing processor
configuration space, when accessing shared MSRs or for counting northbridge specific events."

To translate this for you. Potential users are
- EDAC ;-)
- other MCA related stuff (e.g. L3 cache index disable)

- performance monitoring
- most probably everything that accesses processor configuration space and shared MSRs

You didn't read all my mails regarding this topic.
The patches fixup sibling information for Magny-Cours. This info is not only exposed to /proc/cpuinfo but also with cpu-topology information in sysfs. I don't see why /sys/devices/system/cpu/cpuX/topology is an old ad-hoc hack.

For all cores on the same multi-node CPU (Magny-Cours) /proc/cpuinfo will show:
- same phys_proc_id
- cpu_node_id of the internal node (0 or 1)
- cpu_core_id (e.g. in range of 0 to 5)

Patches are against tip/master as of today.
Please consider patches 1 and 2 for .30.