Skip to contentSkip to conversation

topology api confusion

DISCUSSION Nathan Lynch 5 participants 7 messages Jul 23, 2005 – Aug 1, 2005

lore
Share & tools
lore
#0 lore raw

We need some clarity on how asm-generic/topology.h is intended to be used. I suspect that it's supposed to be unconditionally included at the end of the architecture's topology.h so that any elements which are undefined by the arch have sensible default definitions. Looking at 2.6.13-rc3, this is what ppc64, ia64, and x86_64 currently do, however i386 does not (i386 pulls in the generic version only when !CONFIG_NUMA).

The #ifndef guards around each element of the topology api cannot serve their apparent intended purpose when the architecture implements a given bit as a function instead of a macro (e.g. cpu_to_node in ppc64):

----

asm-generic/topology.h:

#ifndef cpu_to_node
#define cpu_to_node(cpu) (0)
#endif

----

asm-ppc64/topology.h:

static inline int cpu_to_node(int cpu)

{
        int node;

        node = numa_cpu_lookup_table[cpu];
	....

----

Since ppc64 unconditionally includes asm-generic/topology.h, all uses of cpu_to_node are preprocessed to (0). Similar damage occurs with every other topology function which happens to be a real function instead of a macro. I'm surprised my ppc64 numa systems even boot ;)

If the intent is that the architecture is free to define only a subset of the api and include the generic header for fallback definitions, then we need to do the #ifndef __HAVE_ARCH_FOO thing, no? That is, the code above would look like:

----

asm-generic/topology.h:

#ifndef __HAVE_ARCH_CPU_TO_NODE
#define cpu_to_node(cpu) (0)
#endif

----

asm-ppc64/topology.h:

#define __HAVE_ARCH_CPU_TO_NODE
static inline int cpu_to_node(int cpu)

{
        int node;

        node = numa_cpu_lookup_table[cpu];
	....

----

Thought I'd ask for input first since this would involve a sweep of include/asm-*.

Nathan

#1 lore raw
Quoted 8 lines · Nathan Lynch

We need some clarity on how asm-generic/topology.h is intended to be used. I suspect that it's supposed to be unconditionally included at the end of the architecture's topology.h so that any elements which are undefined by the arch have sensible default definitions. Looking at 2.6.13-rc3, this is what ppc64, ia64, and x86_64 currently do, however i386 does not (i386 pulls in the generic version only when !CONFIG_NUMA).

Quoted 4 lines

The #ifndef guards around each element of the topology api cannot serve their apparent intended purpose when the architecture implements a given bit as a function instead of a macro (e.g. cpu_to_node in ppc64):

When I originally wrote this all up, I had planned it to be used the way i386 does: offer a full implementation of topology if appropriate, else include the generic "sane" default. It has since changed to work more like you described: implement some (or all) of the topology functions, then include the generic one to define any you missed.

Quoted 9 lines

Since ppc64 unconditionally includes asm-generic/topology.h, all uses of cpu_to_node are preprocessed to (0). Similar damage occurs with every other topology function which happens to be a real function instead of a macro. I'm surprised my ppc64 numa systems even boot ;)

If the intent is that the architecture is free to define only a subset of the api and include the generic header for fallback definitions, then we need to do the #ifndef __HAVE_ARCH_FOO thing, no? That is, the code above would look like:

You are correct in noticing that things should (though apparently don't?) go wonky when arches define their topology functions as *functions*, rather than macros. It hasn't really seemed to bite anyone yet, so I've left it alone, though to be honest, it is as surprising to me that it works as it is to you. I've resisted creating #ifdef ARCH_HAVE_XXX all over the place, though maybe it is finally time?

Quoted 2 lines

Thought I'd ask for input first since this would involve a sweep of include/asm-*.

It would indeed... I'd be more than happy to look at any patches you care to generate. As I said, it seems to work, though I'm certain it's all held together by GCC black magic & voodoo, and probably a little duct-tape. A more obviously correct solution would not be a bad thing. :)

-Matt

#2 lore raw
Quoted 14 lines · Matthew Dobson
Quoted 8 lines · Nathan Lynch

We need some clarity on how asm-generic/topology.h is intended to be used. I suspect that it's supposed to be unconditionally included at the end of the architecture's topology.h so that any elements which are undefined by the arch have sensible default definitions. Looking at 2.6.13-rc3, this is what ppc64, ia64, and x86_64 currently do, however i386 does not (i386 pulls in the generic version only when !CONFIG_NUMA).

Quoted 4 lines

The #ifndef guards around each element of the topology api cannot serve their apparent intended purpose when the architecture implements a given bit as a function instead of a macro (e.g. cpu_to_node in ppc64):

Quoted 5 lines

When I originally wrote this all up, I had planned it to be used the way i386 does: offer a full implementation of topology if appropriate, else include the generic "sane" default. It has since changed to work more like you described: implement some (or all) of the topology functions, then include the generic one to define any you missed.

OK.

Quoted 6 lines

You are correct in noticing that things should (though apparently don't?) go wonky when arches define their topology functions as *functions*, rather than macros. It hasn't really seemed to bite anyone yet, so I've left it alone, though to be honest, it is as surprising to me that it works as it is to you. I've resisted creating #ifdef ARCH_HAVE_XXX all over the place, though maybe it is finally time?

Things _do_ go wonky, but likely only on ppc64 -- all cpus show up in all nodes' cpumaps in sysfs. The other architectures which provide overrides and unconditionally include the generic topology.h define only macros iirc. If i386 were to include the generic topology.h it would have similar issues since it uses functions for some things too.

Quoted 7 lines
Quoted 2 lines

Thought I'd ask for input first since this would involve a sweep of include/asm-*.

It would indeed... I'd be more than happy to look at any patches you care to generate. As I said, it seems to work, though I'm certain it's all held together by GCC black magic & voodoo, and probably a little duct-tape. A more obviously correct solution would not be a bad thing. :)

I've got the changes ready, just need to test them a little more.

Thanks.

Nathan

#3 lore raw
In reply to Matthew Dobson
Quoted 35 lines · Matthew Dobson
Quoted 8 lines · Nathan Lynch

We need some clarity on how asm-generic/topology.h is intended to be used. I suspect that it's supposed to be unconditionally included at the end of the architecture's topology.h so that any elements which are undefined by the arch have sensible default definitions. Looking at 2.6.13-rc3, this is what ppc64, ia64, and x86_64 currently do, however i386 does not (i386 pulls in the generic version only when !CONFIG_NUMA).

Quoted 4 lines

The #ifndef guards around each element of the topology api cannot serve their apparent intended purpose when the architecture implements a given bit as a function instead of a macro (e.g. cpu_to_node in ppc64):

When I originally wrote this all up, I had planned it to be used the way i386 does: offer a full implementation of topology if appropriate, else include the generic "sane" default. It has since changed to work more like you described: implement some (or all) of the topology functions, then include the generic one to define any you missed.

Quoted 9 lines

Since ppc64 unconditionally includes asm-generic/topology.h, all uses of cpu_to_node are preprocessed to (0). Similar damage occurs with every other topology function which happens to be a real function instead of a macro. I'm surprised my ppc64 numa systems even boot ;)

If the intent is that the architecture is free to define only a subset of the api and include the generic header for fallback definitions, then we need to do the #ifndef __HAVE_ARCH_FOO thing, no? That is, the code above would look like:

Quoted 6 lines

You are correct in noticing that things should (though apparently don't?) go wonky when arches define their topology functions as *functions*, rather than macros. It hasn't really seemed to bite anyone yet, so I've left it alone, though to be honest, it is as surprising to me that it works as it is to you. I've resisted creating #ifdef ARCH_HAVE_XXX all over the place, though maybe it is finally time?

If I understand the problem, is it amenable to just defining the macros and using another name for a function? In other words, if most arch define xx_generic_add as a function, can you just
#define xx_generic_add xx_local_arch_add
which would satisfy the #ifndef, allow use of a function, etc? Then xx_local_arch_add can be the function. Then the common include would not generate macros for things which exist as function.

Signature · 3 lines
   -bill davidsen ([email protected])
"The secret to procrastination is to put things off until the
 last possible moment - but no longer"  -me
#4 lore raw
In reply to Nathan Lynch

Hi,

Quoted 8 lines

We need some clarity on how asm-generic/topology.h is intended to be used. I suspect that it's supposed to be unconditionally included at the end of the architecture's topology.h so that any elements which are undefined by the arch have sensible default definitions. Looking at 2.6.13-rc3, this is what ppc64, ia64, and x86_64 currently do, however i386 does not (i386 pulls in the generic version only when !CONFIG_NUMA).

Quoted 4 lines

The #ifndef guards around each element of the topology api cannot serve their apparent intended purpose when the architecture implements a given bit as a function instead of a macro (e.g. cpu_to_node in ppc64):

Since it doesnt look like this will be resolved by 2.6.13 and NUMA is currently completely broken on ppc64, how does this patch look?

--

Dont include asm-generic/topology.h unconditionally, we end up overriding all the ppc64 specific functions when NUMA is on.

include/asm-ppc64/topology.h +4 -1
Index: linux-2.6.git-work/include/asm-ppc64/topology.h
===================================================================
@@ -33,6 +33,7 @@
 	return first_cpu(tmp);
 }
 
+#define pcibus_to_node(node)    (-1)
 #define pcibus_to_cpumask(bus)	(cpu_online_map)
 
 #define nr_cpus_node(node)	(nr_cpus_in_node[node])
@@ -59,8 +60,10 @@
 	.nr_balance_failed	= 0,			\
 }
 
-#endif /* CONFIG_NUMA */
+#else
 
 #include <asm-generic/topology.h>
 
+#endif /* CONFIG_NUMA */
+
 #endif /* _ASM_PPC64_TOPOLOGY_H */
#5 lore raw
Quoted 11 lines · Anton Blanchard

Hi,

Quoted 8 lines

We need some clarity on how asm-generic/topology.h is intended to be used. I suspect that it's supposed to be unconditionally included at the end of the architecture's topology.h so that any elements which are undefined by the arch have sensible default definitions. Looking at 2.6.13-rc3, this is what ppc64, ia64, and x86_64 currently do, however i386 does not (i386 pulls in the generic version only when !CONFIG_NUMA).

Quoted 7 lines
Quoted 4 lines

The #ifndef guards around each element of the topology api cannot serve their apparent intended purpose when the architecture implements a given bit as a function instead of a macro (e.g. cpu_to_node in ppc64):

Since it doesnt look like this will be resolved by 2.6.13 and NUMA is currently completely broken on ppc64, how does this patch look?

Yes, this change is the least risk for now, thanks.

Quoted 32 lines

--

Dont include asm-generic/topology.h unconditionally, we end up overriding all the ppc64 specific functions when NUMA is on.

Index: linux-2.6.git-work/include/asm-ppc64/topology.h
===================================================================

include/asm-ppc64/topology.h +4 -1
@@ -33,6 +33,7 @@
 	return first_cpu(tmp);
 }
 
+#define pcibus_to_node(node)    (-1)
 #define pcibus_to_cpumask(bus)	(cpu_online_map)
 
 #define nr_cpus_node(node)	(nr_cpus_in_node[node])
@@ -59,8 +60,10 @@
 	.nr_balance_failed	= 0,			\
 }
 
-#endif /* CONFIG_NUMA */
+#else
 
 #include <asm-generic/topology.h>
 
+#endif /* CONFIG_NUMA */
+
 #endif /* _ASM_PPC64_TOPOLOGY_H */