Skip to contentSkip to conversation

fix zonelist ordering for NUMA

Lse-tech Martin J. Bligh 5 participants 7 messages Feb 26, 2004 – Feb 27, 2004 Generic include/asm header files

lore
Share & tools
lore
#0 lore raw
Quoted 11 lines · --On Wednesday, February 25, 2004 08:54:09 -0800 Jesse Barnes…
Quoted 6 lines · On Wed, Feb 25, 2004 at 02:01:16PM +0900, [email protected] wrote:
Quoted 3 lines

1) make it arch independent
this means having arch code populate a SLIT-like table for use by the generic zonelist building code

I would like to hear the comments from people on other arch. If the same ordering rule can be applicable for others, it's nice.

Martin, does a scheme like this sound ok with you? Arch specific code would populate a node distance table, which would be used to build each pgdat->zonelist in a smarter way than we do currently.

Yeah, looks sensible to me. We probably ought to do this:

+#ifndef node_distance
+#define node_distance(from,to) (1)
+#endif

in the generic fallback topology headers, not in the mm/ .c files. Matt?

Also, I seem to recall those build_zonelists functions are used for both NUMA and UMA ... now they're getting complex enough that it's probably worth making a specific non-NUMA version, if only for the sanity of 99% of the poor souls trying to work out how a UMA machine lays it out ;-)

It looks like it won't change ordering for existing boxes with single layer flat NUMA topologies (round-robin), but we probably ought to check that carefully ;-)

M.

#1 lore raw
Quoted 27 lines · Martin J. Bligh
Quoted 11 lines · --On Wednesday, February 25, 2004 08:54:09 -0800 Jesse Barnes…
Quoted 6 lines · On Wed, Feb 25, 2004 at 02:01:16PM +0900, [email protected] wrote:
Quoted 3 lines

1) make it arch independent
this means having arch code populate a SLIT-like table for use by the generic zonelist building code

I would like to hear the comments from people on other arch. If the same ordering rule can be applicable for others, it's nice.

Martin, does a scheme like this sound ok with you? Arch specific code would populate a node distance table, which would be used to build each pgdat->zonelist in a smarter way than we do currently.

Yeah, looks sensible to me. We probably ought to do this:

(quoted diff) +3 -0
+#ifndef node_distance
+#define node_distance(from,to) (1)
+#endif

in the generic fallback topology headers, not in the mm/ .c files. Matt?

Also, I seem to recall those build_zonelists functions are used for both NUMA and UMA ... now they're getting complex enough that it's probably worth making a specific non-NUMA version, if only for the sanity of 99% of the poor souls trying to work out how a UMA machine lays it out ;-)

Quoted 5 lines

It looks like it won't change ordering for existing boxes with single layer flat NUMA topologies (round-robin), but we probably ought to check that carefully ;-)

M.

Yep... Here's a quickie for i386 and the generic header. All other arches would look pretty similar to the asm/i386/topology.h change.

-Matt

include/asm-generic/topology.h +3 -0
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.3-vanilla/include/asm-generic/topology.h linux-2.6.3-topo_distance/include/asm-generic/topology.h
@@ -44,6 +44,9 @@
 #ifndef pcibus_to_cpumask
 #define pcibus_to_cpumask(bus)	(cpu_online_map)
 #endif
+#ifndef node_distance
+#define node_distance(from, to)	(1)
+#endif
 
 /* Cross-node load balancing interval. */
 #ifndef NODE_BALANCE_RATE
include/asm-i386/topology.h +6 -0
diff -Nurp --exclude-from=/home/mcd/.dontdiff linux-2.6.3-vanilla/include/asm-i386/topology.h linux-2.6.3-topo_distance/include/asm-i386/topology.h
@@ -66,6 +66,12 @@ static inline cpumask_t pcibus_to_cpumas
 	return node_to_cpumask(mp_bus_id_to_node[bus]);
 }
 
+/* Node-to-Node distance */
+static inline int node_distance(int from, int to)
+{
+	return 1;
+}
+
 /* Cross-node load balancing interval. */
 #define NODE_BALANCE_RATE 100
 
#2 lore raw
In reply to Martin J. Bligh
Quoted 8 lines · Martin J. Bligh

Also, I seem to recall those build_zonelists functions are used for both NUMA and UMA ... now they're getting complex enough that it's probably worth making a specific non-NUMA version, if only for the sanity of 99% of the poor souls trying to work out how a UMA machine lays it out ;-)

It looks like it won't change ordering for existing boxes with single layer flat NUMA topologies (round-robin), but we probably ought to check that carefully ;-)

More ifdefs just make the code messier. How about adding a nice comment explaining that UMA will fall right through the loop, and giving the poor souls a temporary reprieve?

-- dave

#3 lore raw
In reply to Martin J. Bligh
Quoted 6 lines · Martin J. Bligh
Quoted 5 lines

Also, I seem to recall those build_zonelists functions are used for both NUMA and UMA ... now they're getting complex enough that it's probably worth making a specific non-NUMA version, if only for the sanity of 99% of the poor souls trying to work out how a UMA machine lays it out ;-)

Quoted 7 lines
Quoted 3 lines

It looks like it won't change ordering for existing boxes with single layer flat NUMA topologies (round-robin), but we probably ought to check that carefully ;-)

More ifdefs just make the code messier. How about adding a nice comment explaining that UMA will fall right through the loop, and giving the poor souls a temporary reprieve?

In this case, I disagree - it's better to just have the ifdef.

M.

#5 lore raw
In reply to Martin J. Bligh
Quoted 7 lines · --On Thursday, February 26, 2004 16:47:10 -0800 Chris Wedgwood <[email protected]>…

Review

Quoted 3 lines · Martin J. Bligh
(quoted diff) +3 -0
+#ifndef node_distance
+#define node_distance(from,to) (1)
+#endif

1 or 0 here?

Dunno ... was from [email protected], but I'd guess 1 makes more sense semantically - we're saying the distance between nodes on a flat NUMA architecture is equal (I think ;-))

M.

#6 lore raw
In reply to Martin J. Bligh

Hi,

Quoted 9 lines
Quoted 5 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+#ifndef node_distance
+#define node_distance(from,to) (1)
+#endif

1 or 0 here?

Dunno ... was from [email protected], but I'd guess 1 makes more sense semantically - we're saying the distance between nodes on a flat NUMA architecture is equal (I think ;-))

Umm. Even on a flat NUMA architecture, I should have distinguish between local and remote. It's NUMA.

#define node_distance(from,to) (from != to)

Best regards.
--
NOMURA, Jun'ichi <[email protected]>